CN117710325A - Method and system for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery - Google Patents
Method and system for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery Download PDFInfo
- Publication number
- CN117710325A CN117710325A CN202311729418.1A CN202311729418A CN117710325A CN 117710325 A CN117710325 A CN 117710325A CN 202311729418 A CN202311729418 A CN 202311729418A CN 117710325 A CN117710325 A CN 117710325A
- Authority
- CN
- China
- Prior art keywords
- image
- endoscopic
- model
- gray
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 210000002990 parathyroid gland Anatomy 0.000 title claims abstract description 67
- 238000000034 method Methods 0.000 title claims abstract description 48
- 210000001685 thyroid gland Anatomy 0.000 title claims abstract description 37
- 238000001356 surgical procedure Methods 0.000 title claims abstract description 34
- 238000012549 training Methods 0.000 claims abstract description 72
- 238000011161 development Methods 0.000 claims abstract description 26
- 238000002372 labelling Methods 0.000 claims abstract description 7
- 238000003062 neural network model Methods 0.000 claims description 40
- 238000001514 detection method Methods 0.000 claims description 27
- 238000012545 processing Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 15
- 238000000605 extraction Methods 0.000 claims description 10
- 230000000849 parathyroid Effects 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000002790 cross-validation Methods 0.000 claims description 7
- 238000012360 testing method Methods 0.000 claims description 7
- 238000007781 pre-processing Methods 0.000 claims description 5
- 238000012795 verification Methods 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 claims description 3
- 238000011478 gradient descent method Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000008439 repair process Effects 0.000 claims description 2
- 238000003384 imaging method Methods 0.000 claims 1
- 238000012800 visualization Methods 0.000 claims 1
- 208000027418 Wounds and injury Diseases 0.000 abstract description 4
- 230000006378 damage Effects 0.000 abstract description 4
- 208000014674 injury Diseases 0.000 abstract description 4
- 238000004590 computer program Methods 0.000 description 7
- 238000013135 deep learning Methods 0.000 description 7
- 230000002980 postoperative effect Effects 0.000 description 4
- 238000012216 screening Methods 0.000 description 4
- 208000024770 Thyroid neoplasm Diseases 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000002674 endoscopic surgery Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- OYPRJOBELJOOCE-UHFFFAOYSA-N Calcium Chemical compound [Ca] OYPRJOBELJOOCE-UHFFFAOYSA-N 0.000 description 2
- 229930003316 Vitamin D Natural products 0.000 description 2
- QYSXJUFSXHHAJI-XFEUOLMDSA-N Vitamin D3 Natural products C1(/[C@@H]2CC[C@@H]([C@]2(CCC1)C)[C@H](C)CCCC(C)C)=C/C=C1\C[C@@H](O)CCC1=C QYSXJUFSXHHAJI-XFEUOLMDSA-N 0.000 description 2
- 229910052791 calcium Inorganic materials 0.000 description 2
- 239000011575 calcium Substances 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 201000002510 thyroid cancer Diseases 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 235000019166 vitamin D Nutrition 0.000 description 2
- 239000011710 vitamin D Substances 0.000 description 2
- 150000003710 vitamin D derivatives Chemical class 0.000 description 2
- 229940046008 vitamin d Drugs 0.000 description 2
- 208000019901 Anxiety disease Diseases 0.000 description 1
- 206010010904 Convulsion Diseases 0.000 description 1
- 206010012289 Dementia Diseases 0.000 description 1
- 206010013700 Drug hypersensitivity Diseases 0.000 description 1
- 208000013038 Hypocalcemia Diseases 0.000 description 1
- 208000000038 Hypoparathyroidism Diseases 0.000 description 1
- 208000000913 Kidney Calculi Diseases 0.000 description 1
- 206010028980 Neoplasm Diseases 0.000 description 1
- 206010029148 Nephrolithiasis Diseases 0.000 description 1
- 208000030714 Recurrent Laryngeal Nerve injury Diseases 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000036506 anxiety Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000004097 bone metabolism Effects 0.000 description 1
- 201000011510 cancer Diseases 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000036461 convulsion Effects 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000008021 deposition Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 201000005311 drug allergy Diseases 0.000 description 1
- 230000004064 dysfunction Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000002682 general surgery Methods 0.000 description 1
- 230000000705 hypocalcaemia Effects 0.000 description 1
- 238000011065 in-situ storage Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000036210 malignancy Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 231100000862 numbness Toxicity 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000009469 supplementation Effects 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 208000013076 thyroid tumor Diseases 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
- 238000011277 treatment modality Methods 0.000 description 1
Abstract
The invention relates to an automatic parathyroid gland identification and tracking method in endoscopic thyroid surgery, which comprises the following steps: acquiring an endoscopic thyroidectomy image in a white light mode; extracting the outline of the endoscopic thyroidectomy image and adjusting the pixel value of the pixel points with unclear development condition in the extracted outline to obtain a final outline image; labeling contours in the corresponding endoscopic thyroidectomy images based on the final contour images to obtain image-enhanced endoscopic thyroidectomy images taking the contour areas as the ROI areas; training a pre-constructed recognition model based on the endoscopic thyroidectomy image in the white light mode and the image-enhanced endoscopic thyroidectomy image to obtain a trained recognition model; and identifying and tracking the parathyroid gland by the real-time endoscopic video through the trained identification model. The invention can automatically and accurately identify the parathyroid glands and track the parathyroid glands in real time, and makes auxiliary contribution for reducing the occurrence rate of parathyroid gland injury.
Description
Technical Field
The invention relates to the technical field of artificial intelligence and medical treatment, in particular to an automatic parathyroid gland identification and tracking method and system in endoscopic thyroid surgery.
Background
Thyroid tumors are the most common malignancy of the head and neck, with surgical treatment being the most prominent treatment modality. Major complications of thyroid cancer surgery include parathyroid injury and recurrent laryngeal nerve injury, and at present, related literature reports at home and abroad that the incidence rates of temporary and permanent parathyroid hypofunction after thyroid surgery are 14% -60% and 4% -11%, respectively. Clinically, patients with hypoparathyroidism after thyroid cancer surgery need to be supplemented with calcium and vitamin D to improve hypocalcemia and its symptoms. The long-term overdose or insufficient supplementation of calcium and/or vitamin D can lead to kidney stones, renal calcareous deposition, limb numbness, convulsion, abnormal bone metabolism, even complications such as depression, anxiety, dementia and the like, and seriously affect the life quality of patients. The parathyroid glands are accurately identified in the operation and reserved in situ, so that the incidence rate of the postoperative parathyroid gland hypofunction can be effectively reduced.
At present, parathyroid gland identification comprises a naked eye identification method, a positive and negative developing method, an autofluorescence technology and other methods, and the methods have the defects of low accuracy, influence on operation of an operation area due to leakage of a developer, drug allergy, requirement of special equipment and the like.
There is therefore a need in the market today for techniques and devices for monitoring and dynamically tracking parathyroid glands in real-time during endoscopic thyroid surgery to reduce the incidence of post-operative parathyroid dysfunction.
Disclosure of Invention
The invention aims to at least solve one of the defects of the prior art and provides a parathyroid gland automatic identification and tracking method and system in endoscopic thyroid surgery.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
specifically, an automatic parathyroid gland recognition and tracking method in endoscopic thyroid surgery is provided, which comprises the following steps:
acquiring an endoscopic thyroidectomy image in a white light mode;
performing contour extraction operation on the endoscopic thyroidectomy image to obtain a contour image of the endoscopic thyroidectomy image under gray scale;
performing image restoration on the contour image to obtain a restored contour image;
judging whether the pixel points in the repaired contour image have the condition of unclear development, if so, carrying out pixel value adjustment on the pixel points with the condition of unclear development to obtain a final contour image;
labeling contours in the corresponding endoscopic thyroidectomy images based on the final contour images to obtain image-enhanced endoscopic thyroidectomy images taking the contour areas as the ROI areas;
training a pre-constructed recognition model based on the endoscopic thyroidectomy image in the white light mode and the image-enhanced endoscopic thyroidectomy image to obtain a trained recognition model;
acquiring a real-time endoscope video in the operation process;
and identifying and tracking the parathyroid gland by the real-time endoscopic video through the trained identification model.
Further, specifically, training the pre-constructed recognition model based on the endoscopic thyroidectomy image in the white light mode and the image-enhanced endoscopic thyroidectomy image to obtain a trained recognition model, comprising,
performing data preprocessing on the endoscopic thyroidectomy image in a white light mode to obtain first training data after marking, and performing data preprocessing on the image-enhanced endoscopic thyroidectomy image to obtain second training data after marking;
training the pre-constructed neural network model through the first training data and the second training data to obtain a trained recognition model;
the method comprises the steps of embedding a video stream processing module in a trained recognition model to obtain a final recognition model, wherein the video stream processing module is used for capturing a video sequence, inputting each frame in the video sequence into the trained recognition model to obtain a target detection result, tracking the parathyroid gland in a current frame according to the target detection result, updating the state of the parathyroid gland according to the position information of the parathyroid gland in the current frame, processing the image of each frame in the video sequence in real time, and continuously updating the position information of the parathyroid gland to realize dynamic tracking of the parathyroid gland.
Further, specifically, judging whether the pixel point in the repaired outline image has a developing unclear condition, including,
for each contour image, calculating an unclear threshold Gray for a certain pixel point in the current contour image TH ，Gray TH The calculation method comprises averaging the neighborhood pixel points of the current pixel point to obtain a neighborhood pixel average value, and taking the neighborhood pixel average value of A times as Gray TH Wherein A is a scaling factor, and is obtained by manual setting;
judging whether the Gray value of the current pixel point is larger than Gray TH If yes, judging that the current pixel point does not have the developing unclear condition, and if not, judging that the current pixel point has the developing unclear condition.
Further, specifically, the pixel value adjustment is performed on the pixel points where the unclear development condition exists, including,
calculating the maximum Gray value Gray of all pixel points in the outline image of the pixel points with unclear development condition max Minimum Gray value Gray min Average Gray value Gray avg ；
Judging Gray value Gray of pixel point with unclear development condition present And [ Gray ] min ，Gray max ]Is characterized by that the relative relation of the above-mentioned components,
if Gray present ∈[Gray min ，Gray avg ]Then there is
If Gray present ∈[Gray avg ，Gray max ]Then there is
Wherein Gray present ， The gray value of the pixel point with the condition of unclear development after adjustment is obtained.
Further, the method further comprises the step of using a hierarchical sampling and cross-validation method when the first training data and the second training data are obtained through division, so that multiple pictures of the same case cannot appear in the training set and the testing set at the same time.
Further, the method further comprises the steps of storing the first training data and the second training data in a file format of an adaptive model before the first training data and the second training data are input into the model for training, converting the corresponding annotation information file into a format required by the model, and carrying out standardized normalization processing on images in the first training data and the second training data.
Further, specifically, the pre-constructed neural network model includes selecting a neural network model suitable for parathyroid recognition as an initial neural network model, respectively constructing a first target detection model for an endoscopic thyrotomy image in a white light mode and a second target detection model for an image after image enhancement in the initial neural network model, fusing characteristic layers of the first target detection model and the second target detection model, removing a final full-connection layer in the initial neural network model, replacing the final full-connection layer with a full-connection layer with the output number of 2 types, and randomly initializing weights of the full-connection layers with the output number of types to finally obtain the pre-constructed neural network model.
Further, specifically, the training process of the pre-constructed neural network model comprises,
in the training process, a random gradient descent method is adopted to optimize the model, in the multi-round training, the model carries out forward propagation calculation through input images to obtain a prediction result, then error between the prediction result and a real label is calculated, and model parameters are updated through reverse propagation to optimize the performance of the model.
Further, specifically, evaluating the performance of the neural network model obtained through training in K-fold cross validation by using the accuracy, wherein the neural network model with the highest accuracy is the optimal neural network model;
and then testing the optimal neural network model, drawing an ROC curve graph of the optimal neural network model according to the condition of the parathyroid gland recognition model, drawing the true positive rate and the false positive rate of doctors participating in verification at the corresponding positions in the ROC curve graph, and if the ROC curve of the optimal neural network model does not completely surround the result points of the doctors, adjusting the model until the ROC curve of the optimal neural network model completely surrounds the result points of the doctors, so as to obtain the trained recognition model.
The invention also provides an automatic parathyroid gland recognition and tracking system in endoscopic thyroid surgery, which comprises the following steps:
the data acquisition module is used for acquiring an endoscopic thyrotomy image in a white light mode;
the contour image extraction module is used for carrying out contour extraction operation on the endoscopic thyroidectomy image to obtain a contour image of the endoscopic thyroidectomy image under gray scale;
the image restoration module is used for carrying out image restoration on the contour image to obtain a restored contour image;
the pixel value adjusting module is used for judging whether the pixel points in the repaired contour image have the condition of unclear development, if so, the pixel values of the pixel points with the condition of unclear development are adjusted to obtain a final contour image;
the influence enhancement image data acquisition module is used for marking contours in the corresponding endoscopic thyroidectomy images based on the final contour image to obtain image enhancement endoscopic thyroidectomy images taking the contour region as the ROI region;
the recognition model training module is used for training the pre-constructed recognition model based on the endoscopic thyroidectomy image in the white light mode and the image-enhanced endoscopic thyroidectomy image to obtain a trained recognition model;
the real-time endoscope video acquisition module is used for acquiring real-time endoscope videos in the operation process;
and the recognition module is used for recognizing and tracking the parathyroid gland of the real-time endoscope video through the trained recognition model.
The beneficial effects of the invention are as follows:
according to the method for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery, provided by the invention, based on an advanced deep learning target detection frame, after an endoscopic thyroid surgery image in a white light mode is obtained, contour extraction is carried out, the unclear positions of pixels in the extracted contour are adjusted, so that parathyroid glands are highlighted, and then an image-enhanced endoscopic thyroid surgery image taking a contour area as an ROI area can be obtained by marking in an original image, so that the attention of the model to parathyroid glands is higher when the model is identified for training, the training convergence process of the model is accelerated on one hand, and the identification accuracy of the model is improved on the other hand. Through this technical scheme, the system can automatic identification and track the position of parathyroid in real time, possesses following advantage:
1. the accuracy is high, and the stability is good: the system adopts a deep learning target detection framework, and can automatically identify and track parathyroid glands in real time by carrying out model training on pictures in a white light mode and an image enhancement mode in endoscopic thyroid surgery, thereby improving the accuracy and stability of parathyroid gland identification.
2. The application range is wide: the system is suitable for different types of endoscope systems and operation, has higher universality, and can be widely applied to a plurality of clinical fields such as general surgery in hospitals, head and neck surgery and the like.
3. Improving the postoperative quality of life of the patient: by reducing the incidence of parathyroid injury, the system can improve the safety and accuracy of the operation and improve the postoperative life quality of patients.
Drawings
The above and other features of the present disclosure will become more apparent from the detailed description of the embodiments illustrated in the accompanying drawings, in which like reference numerals designate like or similar output voltages, it is apparent that the accompanying drawings in which the following description is given only by way of example of the present disclosure, and that other drawings may be obtained by those skilled in the art without undue effort, in which:
FIG. 1 is a flow chart of the method for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery according to the present invention;
fig. 2 is a schematic structural diagram of a neural network model related to an automatic parathyroid recognition and tracking method in endoscopic thyroid surgery without adding a video stream processing module.
Detailed Description
The conception, specific structure, and technical effects produced by the present invention will be clearly and completely described below with reference to the embodiments and the drawings to fully understand the objects, aspects, and effects of the present invention. It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The same reference numbers will be used throughout the drawings to refer to the same or like parts.
Referring to fig. 1, embodiment 1 of the present invention provides an automatic parathyroid gland recognition and tracking method in endoscopic thyroid surgery, comprising the following steps:
step 110, obtaining an endoscopic thyroidectomy image in a white light mode;
step 120, performing contour extraction operation on the endoscopic thyroidectomy image to obtain a contour image of the endoscopic thyroidectomy image under gray scale;
step 130, performing image restoration on the contour image to obtain a restored contour image;
step 140, judging whether the pixel points in the repaired contour image have the condition of unclear development, if so, carrying out pixel value adjustment on the pixel points with the condition of unclear development to obtain a final contour image;
step 150, labeling the contour in the corresponding endoscopic thyroidectomy image based on the final contour image to obtain an image-enhanced endoscopic thyroidectomy image taking the contour region as the ROI region;
step 160, training a pre-constructed recognition model based on the endoscopic thyroidectomy image in the white light mode and the image-enhanced endoscopic thyroidectomy image to obtain a trained recognition model;
step 170, acquiring real-time endoscope video in the operation process;
and 180, identifying and tracking the parathyroid gland of the real-time endoscope video through the trained identification model.
In this embodiment 1, based on the advanced deep learning target detection framework, after an endoscopic thyroid operation image in a white light mode is obtained, the outline is extracted, the position where pixels in the extracted outline are unclear is adjusted, the parathyroid gland is highlighted, and then an image enhanced endoscopic thyroid operation image taking the outline area as an ROI area can be obtained by labeling in the original image, so that the attention of the model to the parathyroid gland is higher when the recognition model is trained, on one hand, the training convergence process of the model is accelerated, and on the other hand, the recognition accuracy of the model is improved.
In particular, in practice, the present invention includes three processes,
1. data acquisition and preprocessing
In order to establish an operation auxiliary system for automatically identifying and tracking parathyroid glands in endoscopic thyroid operations in real time, images of a white light mode and an image enhancement mode in endoscopic thyroid operations and corresponding parathyroid gland position information are required to be collected and preprocessed for training and testing of models.
(1) Collecting thyroid endoscope operation pictures: and collecting thyroid operation pictures in a white light mode and an image enhancement mode in a targeted manner according to task targets. The white light mode can provide natural and real image information, and is helpful for the model to learn visual characteristics in the real world; the image enhancement mode highlights specific characteristics of the target through image enhancement processing, so that the model is facilitated to capture key characteristics of the target more effectively;
(2) Data labeling and enhancement: dividing the collected thyroid endoscope operation pictures into positive pictures and negative pictures according to whether parathyroid glands are contained or not, and marking nerves in the positive pictures. In addition, by applying a data enhancement technology and through operations such as image rotation, translation, scaling and the like, the diversity of a data set is increased, and the generalization capability of a model is improved;
(3) Image quality screening: and (3) quality screening is carried out on all pictures, and pictures with poor quality such as blurring, serious noise and the like are removed, so that the accuracy and the reliability of a data set are ensured. An image quality evaluation algorithm is introduced to automatically screen the images, so that the screening efficiency is improved;
(4) Data partitioning and cross-validation: firstly, dividing the marked white light mode and the marked image of the image enhancement mode into two pieces of training data. In the specific division process, a hierarchical sampling and cross-validation method is used to ensure that multiple pictures of the same case cannot appear in the training set and the testing set at the same time. Namely, all pictures contained in the case are distributed to a training set or a test set so as to avoid data leakage and deviation of an evaluation result;
(5) Data format conversion and standardization: and storing the marked data set in a proper file format, and converting the corresponding marked information file into a format required by the model. The image is preprocessed, including operations such as image standardization and normalization, so that model training difficulty is reduced, and convergence speed is improved.
2. Model training and optimization
The deep learning target detection framework is adopted, and the images in the white light mode and the image enhancement mode are used for training, so that the model has high-efficiency parathyroid recognition capability. The specific process comprises the following steps:
(1) Loading a pre-training model: loading a pre-trained neural network model suitable for the task, such as YOLO V7 (only YOLO V7 is taken as an example, and other suitable target detection frames can be selected in practical application);
(2) Model construction and fusion: with reference to fig. 2, target detection models for the white light mode picture and the image enhancement mode picture are respectively constructed, and then feature layers of the white light mode picture and the image enhancement mode picture are fused. And removing the last full-connection layer in the neural network model, replacing the last full-connection layer with the output number of 2 types and randomly initializing the weight of the full-connection layer with the output number of 2 types and types. Finally obtaining a deep learning neural network model for thyroid endoscope image target detection;
(3) Model training and optimizing: and training the divided training sets respectively, and optimizing the model by adopting a random gradient descent method in the training process. In the multi-round training, the model carries out forward propagation calculation through input images to obtain a prediction result, then calculates an error between the prediction result and a real label, and updates model parameters through backward propagation to optimize the performance of the model. The image enhancement mode pictures in the training data are used for increasing the diversity of training set samples and improving the generalization capability of the model;
(4) Model verification and selection: the performance of the neural network model obtained through training in K-fold cross validation is evaluated by using the accuracy, and the neural network model with the highest accuracy is the optimal neural network model;
(5) Model evaluation and comparison: and drawing an ROC curve graph of the optimal neural network model according to the condition of the parathyroid gland recognition model, and drawing the true positive rate and the false positive rate of the doctor participating in verification at the corresponding positions in the ROC curve graph. If the ROC curve of the optimal neural network model surrounds the result points of doctors, the optimal neural network model can reach or exceed the performance of human experts, and the ROC curve has the capability of identifying parathyroid glands in actual endoscopic surgery;
(6) Dynamic tracking algorithm: and embedding a video stream processing module in the model, inputting each frame in the video sequence into a deep learning network, extracting high-level abstract feature representation, and performing forward propagation calculation on the image to obtain a target detection result, wherein the target detection result comprises positive or negative judgment of parathyroid glands. And tracking the target object in the current frame according to the target detection result, and updating the state of the target object according to the position information of the target object in the current frame, thereby improving the detection accuracy of the next frame. Processing the image of each frame in real time in the video sequence, and continuously updating the position information of the target object to realize the dynamic tracking of the target object;
(7) Model integration and application: the optimized model and the dynamic tracking algorithm are embedded into an endoscopic surgery auxiliary system together, so that a real-time parathyroid gland identification and tracking function is provided for doctors, and the safety and accuracy of surgery are improved.
3. System implementation and application
(1) Model deployment: embedding the trained deep learning model into an endoscopic surgery auxiliary system to realize the automatic recognition and real-time tracking functions of parathyroid glands;
(2) Data acquisition and processing: when parathyroid glands enter an endoscope visual field, a video stream processing module of the system processes real-time video, and each frame in a video sequence is input into a deep learning network for detection;
(3) Target detection and optimization: inputting an image or video stream to be detected into a model, and outputting detected target position and category information by the model; meanwhile, post-processing is carried out on the target detection result output by the model according to actual demands, such as removing repeated detection, screening targets with low confidence and the like;
(4) Target tracking and display: the system automatically recognizes the position of the parathyroid gland, marks the position of the parathyroid gland on a screen, and tracks the movement of the parathyroid gland in real time, so that an operator can adjust surgical instruments in time, the parathyroid gland is protected, and the occurrence rate of parathyroid gland injury is reduced.
As a preferred embodiment of the present invention, specifically, determining whether there is a developing unclear condition of the pixel point in the contour image after repair includes,
for each contour image, calculating an unclear threshold Gray for a certain pixel point in the current contour image TH ，Gray TH The calculation method comprises the steps of averaging the neighborhood pixel points of the current pixel point to obtain a neighborhood pixel average value, wherein the neighborhood refers to the 8 neighborhood of the pixel, and the neighborhood pixel average value which is A times is used as Gray TH Wherein A is a scaling factor, and is obtained by manual setting;
judging whether the Gray value of the current pixel point is larger than Gray TH If yes, judging that the current pixel point does not have the developing unclear condition, and if not, judging that the current pixel point has the developing unclear condition.
In addition, the situation of unclear development possibly existing in the actual image is considered to influence the training result of the subsequent recognition model, so that the situation of unclear development is found through the mode, and the subsequent processing and adjustment are facilitated.
As a preferred embodiment of the present invention, specifically, pixel value adjustment is performed for a pixel where there is a developing unclear condition, including,
calculating the maximum Gray value Gray of all pixel points in the outline image of the pixel points with unclear development condition max Minimum Gray value Gray min Average Gray value Gray avg ；
Judging Gray value Gray of pixel point with unclear development condition present And [ Gray ] min ，Gray max ]Is characterized by that the relative relation of the above-mentioned components,
if Gray present ∈[Gray min ，Gray avg ]ThenHas the following components
If Gray present ∈[Gray avg ，Gray max ]Then there is
Wherein Gray present ， The gray value of the pixel point with the condition of unclear development after adjustment is obtained.
In the preferred embodiment, the gray value is adjusted in the condition of unclear development in the above manner, so that the contour image can be as clear as possible, and the contour regions of parathyroid glands can be more clearly distinguished to form the respective ROI regions, thereby facilitating the training of the subsequent model.
The invention also provides an automatic parathyroid gland recognition and tracking system in endoscopic thyroid surgery, which comprises the following steps:
the data acquisition module is used for acquiring an endoscopic thyrotomy image in a white light mode;
the contour image extraction module is used for carrying out contour extraction operation on the endoscopic thyroidectomy image to obtain a contour image of the endoscopic thyroidectomy image under gray scale;
the image restoration module is used for carrying out image restoration on the contour image to obtain a restored contour image;
the pixel value adjusting module is used for judging whether the pixel points in the repaired contour image have the condition of unclear development, if so, the pixel values of the pixel points with the condition of unclear development are adjusted to obtain a final contour image;
the image enhancement image data acquisition module is used for marking the outline in the corresponding endoscopic thyroidectomy image based on the final outline image to obtain an image enhancement endoscopic thyroidectomy image taking the outline area as the ROI area;
the recognition model training module is used for training the pre-constructed recognition model based on the endoscopic thyroidectomy image in the white light mode and the image-enhanced endoscopic thyroidectomy image to obtain a trained recognition model;
the real-time endoscope video acquisition module is used for acquiring real-time endoscope videos in the operation process;
and the recognition module is used for recognizing and tracking the parathyroid gland of the real-time endoscope video through the trained recognition model.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed over a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules.
The integrated modules, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on this understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth.
While the present invention has been described in considerable detail and with particularity with respect to several described embodiments, it is not intended to be limited to any such detail or embodiments or any particular embodiment, but is to be construed as providing broad interpretation of such claims by reference to the appended claims in view of the prior art so as to effectively encompass the intended scope of the invention. Furthermore, the foregoing description of the invention has been presented in its embodiments contemplated by the inventors for the purpose of providing a useful description, and for the purposes of providing a non-essential modification of the invention that may not be presently contemplated, may represent an equivalent modification of the invention.
The present invention is not limited to the above embodiments, but is merely preferred embodiments of the present invention, and the present invention should be construed as being limited to the above embodiments as long as the technical effects of the present invention are achieved by the same means. Various modifications and variations are possible in the technical solution and/or in the embodiments within the scope of the invention.
Claims (10)
1. The automatic parathyroid gland identification and tracking method in endoscopic thyroid surgery is characterized by comprising the following steps of:
acquiring an endoscopic thyroidectomy image in a white light mode;
performing contour extraction operation on the endoscopic thyroidectomy image to obtain a contour image of the endoscopic thyroidectomy image under gray scale;
performing image restoration on the contour image to obtain a restored contour image;
judging whether the pixel points in the repaired contour image have the condition of unclear development, if so, carrying out pixel value adjustment on the pixel points with the condition of unclear development to obtain a final contour image;
labeling contours in the corresponding endoscopic thyroidectomy images based on the final contour images to obtain image-enhanced endoscopic thyroidectomy images taking the contour areas as the ROI areas;
training a pre-constructed recognition model based on the endoscopic thyroidectomy image in the white light mode and the image-enhanced endoscopic thyroidectomy image to obtain a trained recognition model;
acquiring a real-time endoscope video in the operation process;
and identifying and tracking the parathyroid gland by the real-time endoscopic video through the trained identification model.
2. The method for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery according to claim 1, wherein the training of the pre-constructed identification model based on the endoscopic thyroid surgery image in white light mode and the image-enhanced endoscopic thyroid surgery image to obtain a trained identification model comprises,
performing data preprocessing on the endoscopic thyroidectomy image in a white light mode to obtain first training data after marking, and performing data preprocessing on the image-enhanced endoscopic thyroidectomy image to obtain second training data after marking;
training the pre-constructed neural network model through the first training data and the second training data to obtain a trained recognition model;
the method comprises the steps of embedding a video stream processing module in a trained recognition model to obtain a final recognition model, wherein the video stream processing module is used for capturing a video sequence, inputting each frame in the video sequence into the trained recognition model to obtain a target detection result, tracking the parathyroid gland of the current frame according to the target detection result, updating the state of the parathyroid gland according to the position information of the parathyroid gland in the current frame, processing the image of each frame in the video sequence in real time, and continuously updating the position information of the parathyroid gland to realize dynamic tracking of the parathyroid gland.
3. The method for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery according to claim 1, wherein specifically determining whether there is an unclear imaging condition of the pixels in the contour image after repair comprises,
for each contour image, calculating an unclear threshold Gray for a certain pixel point in the current contour image TH ，Gray TH The calculation method comprises averaging the neighborhood pixel points of the current pixel point to obtain a neighborhood pixel average value, and taking the neighborhood pixel average value of A times as Gray TH Wherein A is a scaling factor, and is obtained by manual setting;
judging whether the Gray value of the current pixel point is larger than Gray TH If yes, judging that the current pixel point does not have the developing unclear condition, and if not, judging that the current pixel point has the developing unclear condition.
4. The method for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery according to claim 3, wherein the adjusting of the pixel values of the pixels with unclear visualization comprises,
calculating the maximum Gray value Gray of all pixel points in the outline image of the pixel points with unclear development condition max Minimum Gray value Gray min Average Gray value Gray avg ；
Judging Gray value Gray of pixel point with unclear development condition present And [ Gray ] min ，Gray max ]Is characterized by that the relative relation of the above-mentioned components,
if Gray present ∈[Gray min ，Gray avg ]Then there is
If Gray present ∈[Gray avg ，Gray max ]Then there is
Wherein Gray present ， The gray value of the pixel point with the condition of unclear development after adjustment is obtained.
5. The method for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery according to claim 2, further comprising, when the first training data and the second training data are obtained by dividing, using a hierarchical sampling and cross-validation method, ensuring that multiple pictures of the same case do not appear in the training set and the test set at the same time.
6. The method according to claim 2, further comprising, before the first training data and the second training data are input into the model for training, storing the first training data and the second training data in a file format adapted to the model, converting the corresponding labeling information file into a format required by the model, and performing standardized normalization processing on the images in the first training data and the second training data.
7. The method for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery according to claim 1, wherein the pre-built neural network model comprises the steps of selecting a neural network model suitable for parathyroid gland identification as an initial neural network model, respectively constructing a first target detection model aiming at an endoscopic thyroid surgery image in a white light mode and a second target detection model aiming at an image after image enhancement in the initial neural network model, fusing characteristic layers of the first target detection model and the second target detection model, removing a final full-connection layer in the initial neural network model, replacing the final full-connection layer with a full-connection layer with an output number of 2 types, randomly initializing weights of the full-connection layer with the output number of the types, and finally obtaining the pre-built neural network model.
8. The method for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery according to claim 1, wherein training the pre-constructed neural network model comprises,
in the training process, a random gradient descent method is adopted to optimize the model, in the multi-round training, the model carries out forward propagation calculation through input images to obtain a prediction result, then error between the prediction result and a real label is calculated, and model parameters are updated through reverse propagation to optimize the performance of the model.
9. The method for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery according to claim 8, wherein the neural network model with the highest accuracy is the optimal neural network model, specifically, the performance of the neural network model obtained by training in K-fold cross validation is evaluated through the accuracy;
and then testing the optimal neural network model, drawing an ROC curve graph of the optimal neural network model according to the condition of the parathyroid gland recognition model, drawing the true positive rate and the false positive rate of doctors participating in verification at the corresponding positions in the ROC curve graph, and if the ROC curve of the optimal neural network model does not completely surround the result points of the doctors, adjusting the model until the ROC curve of the optimal neural network model completely surrounds the result points of the doctors, so as to obtain the trained recognition model.
10. An automatic parathyroid recognition and tracking system in endoscopic thyroid surgery is characterized by comprising the following steps:
the data acquisition module is used for acquiring an endoscopic thyrotomy image in a white light mode;
the contour image extraction module is used for carrying out contour extraction operation on the endoscopic thyroidectomy image to obtain a contour image of the endoscopic thyroidectomy image under gray scale;
the image restoration module is used for carrying out image restoration on the contour image to obtain a restored contour image;
the pixel value adjusting module is used for judging whether the pixel points in the repaired contour image have the condition of unclear development, if so, the pixel values of the pixel points with the condition of unclear development are adjusted to obtain a final contour image;
the influence enhancement image data acquisition module is used for marking contours in the corresponding endoscopic thyroidectomy images based on the final contour image to obtain image enhancement endoscopic thyroidectomy images taking the contour region as the ROI region;
the recognition model training module is used for training the pre-constructed recognition model based on the endoscopic thyroidectomy image in the white light mode and the image-enhanced endoscopic thyroidectomy image to obtain a trained recognition model;
the real-time endoscope video acquisition module is used for acquiring real-time endoscope videos in the operation process;
and the recognition module is used for recognizing and tracking the parathyroid gland of the real-time endoscope video through the trained recognition model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311729418.1A CN117710325A (en) | 2023-12-15 | 2023-12-15 | Method and system for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311729418.1A CN117710325A (en) | 2023-12-15 | 2023-12-15 | Method and system for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117710325A true CN117710325A (en) | 2024-03-15 |
Family
ID=90147485
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311729418.1A Pending CN117710325A (en) | 2023-12-15 | 2023-12-15 | Method and system for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117710325A (en) |
-
2023
- 2023-12-15 CN CN202311729418.1A patent/CN117710325A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11151721B2 (en) | System and method for automatic detection, localization, and semantic segmentation of anatomical objects | |
| CN110033456B (en) | Medical image processing method, device, equipment and system | |
| CN110930416B (en) | MRI image prostate segmentation method based on U-shaped network | |
| US11195277B2 (en) | Systems and methods for generating normative imaging data for medical image processing using deep learning | |
| EP1465109A2 (en) | Method for automated analysis of digital chest radiographs | |
| JP2007236939A (en) | Method and device for breast border detection | |
| CN112634283A (en) | Hip joint segmentation model establishment method using small sample image training and application thereof | |
| WO2019184851A1 (en) | Image processing method and apparatus, and training method for neural network model | |
| CN111507932B (en) | High-specificity diabetic retinopathy characteristic detection method and storage device | |
| CN104766316A (en) | Novel lip segmentation algorithm for traditional Chinese medical inspection diagnosis | |
| KR102163108B1 (en) | Method and system for detecting in real time an object of interest in image | |
| CN111062953A (en) | Method for identifying parathyroid hyperplasia in ultrasonic image | |
| Goutham et al. | Automatic localization of landmarks in cephalometric images via modified U-Net | |
| CN111612749B (en) | Focus detection method and device based on lung image | |
| JP2004188202A (en) | Automatic analysis method of digital radiograph of chest part | |
| CN112488165A (en) | Infrared pedestrian identification method and system based on deep learning model | |
| CN117710325A (en) | Method and system for automatically identifying and tracking parathyroid glands in endoscopic thyroid surgery | |
| CN115880554A (en) | Knowledge distillation and interpretable multi-modal medical image fusion model and method | |
| CN112885464B (en) | Internal nasal disease real-time auxiliary diagnosis and treatment system based on Att-Res2-CE-Net | |
| CN114972881A (en) | Image segmentation data labeling method and device | |
| CN113989588A (en) | Self-learning-based intelligent evaluation system and method for pentagonal drawing test | |
| CN117710323A (en) | Parathyroid gland identification and blood transport judgment method and system in endoscopic thyroid surgery | |
| CN113362280A (en) | Dynamic target tracking method based on medical radiography | |
| CN109902598A (en) | A kind of Preprocessing Technique for complex background | |
| CN117710324A (en) | Method and system for identifying and tracking recurrent laryngeal nerves in endoscopic thyroid surgery |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |