CN112862754A - System and method for prompting missing detection of retained image based on intelligent identification - Google Patents
System and method for prompting missing detection of retained image based on intelligent identification Download PDFInfo
- Publication number
- CN112862754A CN112862754A CN202110008261.8A CN202110008261A CN112862754A CN 112862754 A CN112862754 A CN 112862754A CN 202110008261 A CN202110008261 A CN 202110008261A CN 112862754 A CN112862754 A CN 112862754A
- Authority
- CN
- China
- Prior art keywords
- image
- endoscope
- digestive tract
- doctor
- retained
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2132—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on discrimination criteria, e.g. discriminant analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30028—Colon; Small intestine
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30092—Stomach; Gastric
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
Abstract
The invention discloses a system and a method for prompting missing detection of retained images based on intelligent identification, wherein the system comprises a retained image starting switch, a video stream processing unit, an arithmetic unit and a display unit; when the system does not receive the retained image signal, acquiring a stomach image from the endoscope in real time and judging the position of the endoscope, thereby realizing the positioning of the position of the endoscope; when the system receives the image retention signal, acquiring a retained image at a corresponding moment, and carrying out image position identification and quality scoring after the retained image is preprocessed; if the score is lower than the preset score, sending a prompt message to prompt a doctor to re-retain the image at the image retaining position; if a plurality of retained graphs exist in the same retained graph position, outputting the retained graph with the highest score; the doctor who outputs with the system stays the picture and keeps the picture with preset standard and compare, if there is the omission of keeping the picture, then show the position and the quantity of omitting to send out prompt message in order to remind the doctor to keep the picture again, avoid appearing lou examining, supplementary doctor makes more comprehensive inspection.
Description
Technical Field
The invention relates to the technical field of image recognition, in particular to a system and a method for prompting missing detection of a retained image based on intelligent recognition.
Background
The latest global cancer burden report shows that three of the first five cancers are digestive tract cancers (liver cancer, stomach cancer and colorectal cancer); four of the first five types of cancer mortality are digestive tract cancers (liver cancer, stomach cancer, esophageal cancer, colorectal cancer).
Gastric cancer, a major cancer of the upper digestive tract, is also the third major tumor worldwide, the fifth most common malignant tumor. The survival rate of the gastric cancer in the late 5 years is 5% -25%, and the survival rate in the early stage reaches 90%. Therefore, early discovery is a key strategy to improve patient survival.
Endoscopy, which is a common means for gastrointestinal tract examination, requires a high level of skill and judgment accuracy for the operating physician. Because doctors need to judge images in real time according to their own experience and knowledge, the resulting efforts are high, and missed diagnosis is easy to occur.
Disclosure of Invention
Aiming at the problem of high missing diagnosis rate of the doctor in the prior art, the invention provides a system and a method for prompting missing diagnosis of the retained picture based on intelligent identification.
In order to achieve the purpose, the invention provides the following technical scheme:
a retention chart missing detection prompting system based on intelligent identification comprises a retention chart starting switch, a video stream processing unit, an operation unit and a display unit;
the first output end of the image-remaining starting switch is connected with the input end of the endoscope, the second output end of the image-remaining starting switch is connected with the first input end of the video stream processing unit, the output end of the endoscope is connected with the second input end of the video stream processing unit, the output end of the video stream processing unit is connected with the input end of the operation unit, the output end of the operation unit is connected with the input end of the display unit, and the output end of the display unit outputs images and prompt information.
Preferably, the retained image starting switch is a two-way foot switch and is used for receiving a stepping signal of a doctor and transmitting the stepping signal to the video stream processing unit so as to obtain a retained image from the endoscope.
Preferably, the operation unit comprises an image preprocessing module and a deep learning module, and is used for identifying the position, the quantity and the quality of the retained images, and if the quality of the retained images is unqualified or the quantity of the retained images is insufficient, the display unit sends alarm information.
Preferably, the deep learning module is provided with an image quality evaluation model, an endoscope position identification model and an image position identification model;
the endoscope position identification model is used for identifying whether the position of the endoscope is an upper digestive tract or a lower digestive tract;
the image position identification model is used for identifying the position of each image left in the doctor operation process and comprises an upper digestive tract image position identification model and a lower digestive tract image position identification model;
the image quality evaluation model is used for evaluating the quality of the image kept by the doctor and grading.
Preferably, the evaluation unit is bidirectionally connected with the arithmetic unit and is used for quantitatively evaluating the surgical process.
The invention also provides a method for prompting missing detection of retained images based on intelligent identification, which specifically comprises the following steps:
s1: firstly, training a system to complete system initialization;
s2: when the system does not receive the retained image signal, acquiring a stomach image from the endoscope in real time and judging the position of the endoscope, thereby realizing the positioning of the position of the endoscope;
s3: or when the system receives the image retention signal, acquiring the image retention at the corresponding moment, and carrying out image position identification and quality scoring after preprocessing the image retention; if the score is lower than the preset score, sending a prompt message to prompt a doctor to re-retain the image at the image retaining position; if a plurality of retained graphs exist in the same retained graph position, outputting the retained graph with the highest score;
s4: and comparing the doctor image-keeping information output by the system with preset standard image-keeping information, if image-keeping omission exists, displaying the positions and the number of the omitted image-keeping, and sending prompt information to remind the doctor to keep the image again, so as to avoid missing detection.
Preferably, the S1 includes:
s1-1: acquiring a large amount of endoscope data, carrying out manual annotation, inputting the annotated endoscope data into a system for repeated training, wherein the annotated data comprises 27 image retention positions of an upper digestive tract, 33 image retention positions of a lower digestive tract and image quality scores;
s1-2: training a secondary classification model by adopting an upper classification data set and a lower classification data set of an endoscope position identification model, and initializing by using a pre-training weight of ImageNet; the image quantity proportion of the training set, the test set and the verification set is 6: 2: 2, and the ratio of the two classes in the 3 data sets is 1: 1;
s1-3: the training data set of the image quality evaluation model comprises two categories of visual field unclear and visual field clear in the endoscopic examination process, wherein the image quantity proportion of the training set, the testing set and the verification set is 6: 2: 2, and the ratio of the two classes in the 3 data sets is about 1: 1;
s1-4: the image position recognition model comprises an upper digestive tract image position recognition model and a lower digestive tract image position recognition model, wherein the image quantity proportion of the training set, the testing set and the verification set is 6: 2: 2; the upper digestive tract image position identification model adopts the structure of effcientnet-B6, and the lower digestive tract image position identification model adopts the structure of effcientnet-B7.
Preferably, the preprocessing in S3 includes image cropping, image scaling, image normalization and image normalization;
image cropping and image zooming: the useful image area is left as the middle organized area, so the surrounding frame is cut, and the cut image is zoomed to 528 × 528;
image normalization: calculating the average image and the standard deviation image of the whole endoscope image database in the following calculation mode:
in the formula (1), XiRepresenting the ith image, which may be a (528, 3) image matrix; n represents the number of endoscope images in the database; mean represents the mean graph; std represents a standard deviation graph;
image X for any input systemiAll need to be standardized to obtain corresponding standardized images Xj：
Xj＝(Xi-mean)/std。
Preferably, in S4, the mapping comparison includes upper digestive tract image position identification and lower digestive tract image position identification, the upper digestive tract image position identification completes the intelligent identification of 27 anatomical positions of the upper digestive tract division, and the lower digestive tract image position identification completes the intelligent identification of 33 anatomical positions of the lower digestive tract division; when the number of the remained pictures of the doctor is inconsistent with the preset standard number of the remained pictures, the prompt message is sent to remind the doctor to remain the pictures again, and missing detection is avoided.
Preferably, also comprises
S5: the system scores doctor operation according to a preset image retention position and image retention quality, and the specific calculation formula is as follows:
in formula (2), S represents a score; m represents a predetermined number of retentates, 27 in the upper and 33 in the lower digestive tract; qkShowing the medical doctor in the k position during the operation, and Q when the k position has the stayk1, otherwise 0; y iskThe figure-left quality of the k-th position is a numerical value between 0 and 1, and the result is output by the image quality evaluation model; wkIndicating the assigned score for the k-th position.
In summary, due to the adoption of the technical scheme, compared with the prior art, the invention at least has the following beneficial effects:
according to the invention, the position and quality of the retained image of the doctor are intelligently identified, the position of the retained image of the doctor is compared with the specified retained image position, the omitted retained image position and quantity are displayed and output, the doctor is assisted to make more comprehensive examination, and the missed diagnosis is prevented. The system can be matched with any endoscope equipment for use, and can be simultaneously suitable for the upper digestive tract and the lower digestive tract. The system scores the doctor operation process according to the omission condition of the retained images and the comprehensive consideration of the quality of the retained images at all positions so as to improve the enthusiasm and the accuracy of the doctor.
Description of the drawings:
fig. 1 is a schematic diagram of a system for prompting missing detection and leaving of a picture based on intelligent recognition according to an exemplary embodiment of the present invention.
Fig. 2 is a schematic diagram of an arithmetic unit according to an exemplary embodiment of the present invention.
Fig. 3 is a schematic diagram of a graph leaving and missing detection prompting method based on intelligent recognition according to an exemplary embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to examples and embodiments. It should be understood that the scope of the above-described subject matter is not limited to the following examples, and any techniques implemented based on the disclosure of the present invention are within the scope of the present invention.
In the description of the present invention, it is to be understood that the terms "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used merely for convenience of description and for simplicity of description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed in a particular orientation, and be operated, and thus, are not to be construed as limiting the present invention.
As shown in fig. 1, the present invention provides a missing detection prompting system based on intelligent identification, which comprises a graph-keeping start switch, a video stream processing unit, an arithmetic unit and a display unit; the first output end of the image-remaining starting switch is connected with the input end of gastrointestinal endoscope equipment (endoscope), the second output end of the image-remaining starting switch is connected with the first input end of the video stream processing unit, the output end of the gastrointestinal endoscope equipment is connected with the second input end of the video stream processing unit, the output end of the video stream processing unit is connected with the input end of the operation unit, the output end of the operation unit is connected with the input end of the display unit, and the output end of the display unit outputs images and prompt information.
In this embodiment, the mapping start switch is a two-way foot switch for receiving the trampling signal of the doctor and transmitting the trampling signal to the gastrointestinal endoscope device and the video stream processing unit respectively.
A video stream processing unit for acquiring a stomach image (i.e. a retained image) from an endoscope; and the operation unit is used for identifying and judging the stomach image and displaying the result on the display unit, and if the quality of the retained image is judged to be unqualified or the quantity of the retained image is not enough, the display unit sends alarm information to remind a doctor of paying attention, so that the missed detection is avoided.
As shown in fig. 2, the operation unit includes an image preprocessing module and a depth learning module, which are connected in sequence, and the depth learning module is provided with an image quality evaluation model, an endoscope position recognition model, and an image position recognition model.
And the image preprocessing module is used for performing preliminary processing on the stomach image information, so that training is required to improve the processing precision.
In this embodiment, the training method of the image preprocessing module includes:
a large amount of endoscope data are collected, all endoscope data need to be manually marked, the marked data comprise 27 anatomical positions of an upper digestive tract, 33 anatomical positions of a lower digestive tract and qualified and unqualified image quality, and the image quality is mainly evaluated according to the cleanness and the definition of an image visual field area. The participated marking personnel carry out cross validation marking by a plurality of digestive tract endoscopy physicians with years of working experience so as to ensure the accuracy and reliability of marking data.
During image preprocessing module training, scope image database can carry out data augmentation operation, include: image brightness and contrast conversion; image scaling transformation; image rotation transformation; image mirror transformation; local distortion transformation, etc.
The image preprocessing module carries out primary processing on the image information and comprises the following steps: image cropping, image scaling, image normalization, and image normalization.
Image cropping and image zooming: since the useful image area in the endoscopic image is only the central organized area, the surrounding frame is cropped, and the pixels of the cropped image are scaled to 528 × 528.
Image normalization: calculating the average image and the standard deviation image of the whole endoscope image database in the following calculation mode:
in the formula (1), XiRepresenting the ith image, which may be a (528, 3) image matrix; n represents the number of endoscope images in the database; mean represents the mean graph (mean value of graph), Std represents the standard deviation graph (standard deviation value of graph).
Image X for any input systemiAll need to be standardized to obtain corresponding standardized images Xj：
Xj＝(Xi-mean)/std
Image normalization: is to standardize the image XjTo between 0 and 1.
In this embodiment, the deep learning module is provided with an image quality evaluation model, an endoscope position recognition model and an image position recognition model, and is used for performing quality evaluation and position recognition on acquired stomach image information, so that the definition and integrity of the stomach image are ensured.
In this embodiment, the endoscope position identification model is used to identify the position of the endoscope. If the operation doctor does not set the type of the current operation, the endoscope position identification model works in the initial stage of the operation, identifies the position of the endoscope so as to determine the type of the current operation, and activates the corresponding image position identification model.
The endoscope position recognition model adopts two types of data of an upper digestion channel and a lower digestion channel to train a two-classification model, the basic model adopts Effcientnet-B0, and the pretrained weight of ImageNet is used for initialization. The image quantity proportion of the training set, the test set and the verification set is 6: 2: 2, and the ratio of the two classes in the 3 data sets is about 1: 1. through experimental verification, the endoscope position identification model has a single task and a very prominent classification result, and can achieve more than 98% of accuracy on a test set and a verification set. The high accuracy of the endoscope position identification model provides guarantee for activating the correct image position model in the next stage.
In this embodiment, the image quality evaluation model is used to evaluate the cleanliness and clarity of the retained images of the doctor and determine whether the retained images are qualified, and simultaneously displays result records and prompts the doctor to operate on a display interface.
In the operation procedure, when images caused by operations such as camera movement, washing, dyeing and the like in an endoscope are unclear or the visual field is blocked, the image retention significance of a doctor is not large, so that the evaluation of the image quality of each image retention of the doctor is necessary for the evaluation of the whole operation process.
The training data for the image quality assessment model is derived from two categories, view ambiguity and view clarity during endoscopy. The image quantity proportion of the training set, the test set and the verification set is 6: 2: 2, and the ratio of the two classes in the 3 data sets is about 1: 1; the base model was initialized with effcientnet-B3 modified class 2 model using pretrained weights for ImageNet. Through experimental verification, the accuracy of the quality evaluation model on a test set and a verification set can reach more than 97%, and the missed diagnosis rate and the misdiagnosis rate can be controlled within 2% on average.
In this embodiment, the image position recognition model is used to intelligently recognize the anatomical position of each mapping in the doctor's operation process.
The image position recognition model comprises an upper digestive tract image position recognition model and a lower digestive tract image position recognition model. The input end of the image position recognition model is connected with the output end of the image preprocessing module, and the output end of the image position recognition model is connected with the input end of the display unit, so that the recognition result is output, displayed and recorded in real time.
The upper digestive tract image position recognition model completes intelligent recognition of 27 anatomical positions of the upper digestive tract division, and the lower digestive tract image position recognition model completes intelligent recognition of 33 anatomical positions of the lower digestive tract division. The modeling data of the image position recognition model is divided into a training set, a test set and a verification set, and the image quantity proportion is 6: 2: 2, and keeping the number of images in each anatomical position category balanced. The upper digestive tract image position recognition model adopts an effcientnet-B6 framework, and the lower digestive tract image position recognition model adopts an effcientnet-B7 framework, and is initialized by using the pre-training weight of ImageNet. Through experimental verification, the accuracy of the upper digestive tract image position identification model and the accuracy of the lower digestive tract image position identification model can reach more than 98% on an independent test set.
In order to improve the accuracy and reliability of the operation, the device also comprises a scoring unit which is bidirectionally connected with the operation unit and is used for quantitatively scoring whether the operation process is standardized. The scoring unit is used for comprehensively evaluating and calculating according to a preset figure retaining position and figure retaining quality, and the specific calculation formula is as follows:
in formula (2), S represents a score; m represents a predetermined number of retentates, 27 in the upper and 33 in the lower digestive tract; qkShowing the medical doctor in the k position during the operation, and Q when the k position has the stayk1, otherwise 0; y iskThe figure-left quality of the k-th position is a numerical value between 0 and 1, and the result is output by the image quality evaluation model; wkThe assigned score of the kth position is shown, the score of the whole operation is calculated by percentage, the total score of the upper/lower alimentary tracts is 100, and the score is calculated according to the cancer at each anatomical positionThe distribution of the disease condition of the position, for example, 27 positions of the digestive tract, each position is divided into 3.7 points on average, 33 positions of the lower digestive tract are divided into 3.03 points on average, the score of the anatomical position with high incidence of cancer (such as esophagus, stomach, colorectal and the like) is higher than the average score, the score of the position which is not easy to suffer from early cancer is lower than the average score, and the specific distribution score can be statistically analyzed according to the diagnosis result of the cancer in the hospital (5 years before operation).
Based on the missing detection prompting system based on intelligent identification, the invention also provides a missing detection prompting method based on intelligent identification, as shown in fig. 3, which specifically comprises the following steps:
s1: firstly, training the system to complete the system initialization.
In this embodiment, because the retained image of the doctor needs to be compared with the preset standard retained image, the operation unit needs to be trained first, so that the operation unit can perform intelligent recognition on the retained image of the doctor. The operation unit comprises an image preprocessing module and a deep learning module which are sequentially connected, and the deep learning module is provided with an image quality evaluation model, an endoscope position identification model and an image position identification model.
S1-1: the training method of the image preprocessing module comprises the following steps:
a large amount of endoscope data are collected, all endoscope data need to be manually marked, the marked data comprise 27 anatomical positions of an upper digestive tract, 33 anatomical positions of a lower digestive tract and qualified and unqualified image quality, and the image quality is mainly evaluated according to the cleanness and the definition of an image visual field area. The participated marking personnel carry out cross validation marking by a plurality of digestive tract endoscopy physicians with years of working experience so as to ensure the accuracy and reliability of marking data.
During image preprocessing module training, scope image database can carry out data augmentation operation, include: image brightness and contrast conversion; image scaling transformation; image rotation transformation; image mirror transformation; local distortion transformation, etc.
The image preprocessing module carries out primary processing on the image information and comprises the following steps: image cropping, image scaling, image normalization, and image normalization.
Image cropping and image zooming: since the useful image area in the endoscopic image is only the central organized area, the surrounding frame is cropped, and the pixels of the cropped image are scaled to 528 × 528.
Image normalization: calculating the average image and the standard deviation image of the whole endoscope image database in the following calculation mode:
in the formula (3), XiRepresenting the ith image, which may be a (528, 3) image matrix; n represents the number of endoscope images in the database; mean represents the mean graph (mean value of graph), Std represents the standard deviation graph (standard deviation value of graph).
Image X for any input systemiAll need to be standardized to obtain corresponding standardized images Xj：
Xj＝(Xi-mean)/std
Image normalization: is to standardize the image XjTo between 0 and 1.
S1-2: the endoscope position recognition model adopts two types of data of an upper digestion channel and a lower digestion channel to train a two-classification model, the basic model adopts Effcientnet-B0, and the pretrained weight of ImageNet is used for initialization. The image quantity proportion of the training set, the test set and the verification set is 6: 2: 2, and the ratio of the two classes in the 3 data sets is about 1: 1.
s1-3: the training data for the image quality assessment model is derived from two categories, view ambiguity and view clarity during endoscopy. The image quantity proportion of the training set, the test set and the verification set is 6: 2: 2, and the ratio of the two classes in the 3 data sets is about 1: 1; the base model was initialized with effcientnet-B3 modified class 2 model using pretrained weights for ImageNet.
S1-4: the image position recognition model comprises an upper digestive tract image position recognition model and a lower digestive tract image position recognition model. The modeling data is divided into a training set, a test set and a verification set, and the image quantity proportion is 6: 2: 2, and keeping the number of images in each anatomical position category balanced. The upper digestive tract image position recognition model adopts an effcientnet-B6 framework, and the lower digestive tract image position recognition model adopts an effcientnet-B7 framework, and is initialized by using the pre-training weight of ImageNet.
S2: when the system does not receive the retained image signal, the stomach image is obtained from the endoscope in real time, and the position of the endoscope is judged, so that the accurate positioning of the position of the endoscope is realized.
In the operation process, before a first foot switch signal (namely, a doctor leaves a picture for the first time), a stomach image acquired in real time enters an endoscope position distinguishing model after passing through an image preprocessing unit, and the current endoscope position is intelligently recognized to be an upper digestive tract or a lower digestive tract, so that the corresponding upper digestive tract image position recognizing model and the lower digestive tract image position recognizing model are activated and used. The endoscope position judgment is based on the average value of all stomach image classification results before the first foot switch signal. Since the upper tract requires 27 mapping positions and the lower tract 33, the number of subsequent identifications and scoring criteria are different.
S3: when the system receives the image retention signal, acquiring an image retention image at a corresponding moment, and carrying out image position identification and quality scoring after the image retention image is preprocessed; if the score is lower than the preset score, sending a prompt message to prompt a doctor to re-retain the image at the image retaining position; if multiple remaining maps exist in the same remaining map position, the remaining map with the highest score is output.
The image position recognition comprises an upper digestive tract and a lower digestive tract, wherein the image positions of the upper digestive tract are 27, and the image positions of the lower digestive tract are 33.
The image quality evaluation model is used for finishing image quality scoring, training data come from two categories of unclear vision and clear vision in the endoscopic examination process, and scoring of the training data is manually marked and tested and verified to ensure accuracy.
S4: and comparing the doctor retained image output by the system with a preset labeled retained image, if the retained image is missed, displaying the position and the quantity of the missed retained image by the display unit, and sending prompt information to remind the doctor to retain the image again so as to avoid missing detection.
The image position recognition model comprises an upper digestive tract image position recognition model and a lower digestive tract image position recognition model. The upper digestive tract image position recognition model completes intelligent recognition of 27 anatomical positions of the upper digestive tract division, and the lower digestive tract image position recognition model completes intelligent recognition of 33 anatomical positions of the lower digestive tract division. When the number of the retained images of the doctor is inconsistent with the preset standard number of the retained images, prompt information is sent to remind the doctor to retain the images again, and missing detection is avoided.
S5: the system scores physician actions based on pre-specified mapping locations and mapping quality.
In order to improve the accuracy and reliability of the doctor operation, the system can comprehensively score the whole operation so as to help the doctor to find the self deficiency. The system carries out comprehensive evaluation calculation according to the preset figure retaining position and figure retaining quality, and the specific calculation formula is as follows:
in formula (4), S represents a score; m represents a predetermined number of retentates, 27 in the upper and 33 in the lower digestive tract; qkShowing the medical doctor in the k position during the operation, and Q when the k position has the stayk1, otherwise 0; y iskThe figure-left quality of the k-th position is a numerical value between 0 and 1, and the result is output by the image quality evaluation model; wkThe assigned score of the kth position is shown, the score of the whole operation is in a percentage system, the total score of the upper/lower digestive tracts is 100, the score is assigned according to the disease condition of the cancer at each anatomical position, for example, 27 positions of the digestive tract are provided, each position is averagely divided into 3.7 points, 33 positions of the lower digestive tract are provided, each position is averagely divided into 3.03 points, the score of the anatomical position with high cancer (such as esophagus, stomach, colorectal and the like) is higher than the average score, and the patient is not easy to suffer from early cancerThe position score of (2) is lower than the average score, and the specific distribution score can be statistically analyzed according to the hospital cancer diagnosis result (5 years before operation).
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.
Claims (10)
1. A retention chart missing detection prompting system based on intelligent identification is characterized by comprising a retention chart starting switch, a video stream processing unit, an arithmetic unit and a display unit;
the first output end of the image-remaining starting switch is connected with the input end of the endoscope, the second output end of the image-remaining starting switch is connected with the first input end of the video stream processing unit, the output end of the endoscope is connected with the second input end of the video stream processing unit, the output end of the video stream processing unit is connected with the input end of the operation unit, the output end of the operation unit is connected with the input end of the display unit, and the output end of the display unit outputs images and prompt information.
2. The system for prompting missing remained images based on intelligent recognition of claim 1, wherein the remained image starting switch is a two-way foot switch for receiving a stepping signal of a doctor and transmitting the stepping signal to the video stream processing unit so as to obtain the remained image from the endoscope.
3. The system for prompting missing detection of remained images based on intelligent identification as claimed in claim 1, wherein the arithmetic unit comprises an image preprocessing module and a deep learning module, and is used for identifying the position, quantity and quality of the remained images, and if the quality of the remained images is not qualified or the quantity is not enough, the display unit sends alarm information.
4. The system for prompting missing remained images based on intelligent identification as claimed in claim 3, wherein the deep learning module is provided with an image quality evaluation model, an endoscope position identification model and an image position identification model;
the endoscope position identification model is used for identifying whether the position of the endoscope is an upper digestive tract or a lower digestive tract;
the image position identification model is used for identifying the position of each image left in the doctor operation process and comprises an upper digestive tract image position identification model and a lower digestive tract image position identification model;
the image quality evaluation model is used for evaluating the quality of the image kept by the doctor and grading.
5. The system for prompting missing remained images based on intelligent recognition of claim 1, further comprising a scoring unit bidirectionally connected with the arithmetic unit for quantitatively scoring the surgical procedure.
6. An intelligent recognition-based graph retention and omission prompting method based on any one of the systems of claims 1-5, which is characterized by specifically comprising the following steps:
s1: firstly, training a system to complete system initialization;
s2: when the system does not receive the retained image signal, acquiring a stomach image from the endoscope in real time and judging the position of the endoscope, thereby realizing the positioning of the position of the endoscope;
s3: or when the system receives the image retention signal, acquiring the image retention at the corresponding moment, and carrying out image position identification and quality scoring after preprocessing the image retention; if the score is lower than the preset score, sending a prompt message to prompt a doctor to re-retain the image at the image retaining position; if a plurality of retained graphs exist in the same retained graph position, outputting the retained graph with the highest score;
s4: and comparing the doctor image-keeping information output by the system with preset standard image-keeping information, if image-keeping omission exists, displaying the positions and the number of the omitted image-keeping, and sending prompt information to remind the doctor to keep the image again, so as to avoid missing detection.
7. The method according to claim 6, wherein the S1 includes:
s1-1: acquiring a large amount of endoscope data, carrying out manual annotation, inputting the annotated endoscope data into a system for repeated training, wherein the annotated data comprises 27 image retention positions of an upper digestive tract, 33 image retention positions of a lower digestive tract and image quality scores;
s1-2: training a secondary classification model by adopting an upper classification data set and a lower classification data set of an endoscope position identification model, and initializing by using a pre-training weight of ImageNet; the image quantity proportion of the training set, the test set and the verification set is 6: 2: 2, and the ratio of the two classes in the 3 data sets is 1: 1;
s1-3: the training data set of the image quality evaluation model comprises two categories of visual field unclear and visual field clear in the endoscopic examination process, wherein the image quantity proportion of the training set, the testing set and the verification set is 6: 2: 2, and the ratio of the two classes in the 3 data sets is about 1: 1;
s1-4: the image position recognition model comprises an upper digestive tract image position recognition model and a lower digestive tract image position recognition model, wherein the image quantity proportion of the training set, the testing set and the verification set is 6: 2: 2; the upper digestive tract image position identification model adopts the structure of effcientnet-B6, and the lower digestive tract image position identification model adopts the structure of effcientnet-B7.
8. The method according to claim 6, wherein the preprocessing in S3 includes image cropping, image scaling, image normalization and image normalization;
image cropping and image zooming: the useful image area is left as the middle organized area, so the surrounding frame is cut, and the cut image is zoomed to 528 × 528;
image normalization: calculating the average image and the standard deviation image of the whole endoscope image database in the following calculation mode:
in the formula (1), XiRepresenting the ith image, which may be a (528, 3) image matrix; n represents the number of endoscope images in the database; mean represents the mean graph; std represents a standard deviation graph;
image X for any input systemiAll need to be standardized to obtain corresponding standardized images Xj：
Xj＝(Xi-mean)/std。
9. The method for prompting missing image detection based on intelligent recognition as claimed in claim 6, wherein in S4, the image-to-image comparison includes upper and lower digestive tract image position recognition, the upper digestive tract image position recognition completes the intelligent recognition of 27 anatomical positions of the upper digestive tract division, and the lower digestive tract image position recognition completes the intelligent recognition of 33 anatomical positions of the lower digestive tract division; when the number of the remained pictures of the doctor is inconsistent with the preset standard number of the remained pictures, the prompt message is sent to remind the doctor to remain the pictures again, and missing detection is avoided.
10. The method as claimed in claim 6, further comprising a step of prompting missing inspection of retained images based on intelligent recognition
S5: the system scores doctor operation according to a preset image retention position and image retention quality, and the specific calculation formula is as follows:
in formula (2), S represents a score; m represents a predetermined number of retentates, 27 in the upper and 33 in the lower digestive tract; qkShowing the medical doctor in the k position during the operation, and Q when the k position has the stayk1, otherwise 0; y iskThe figure-left quality of the k-th position is a numerical value between 0 and 1, and the result is output by the image quality evaluation model; wkRepresents the k < th >The assigned score of the location.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110008261.8A CN112862754A (en) | 2021-01-05 | 2021-01-05 | System and method for prompting missing detection of retained image based on intelligent identification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110008261.8A CN112862754A (en) | 2021-01-05 | 2021-01-05 | System and method for prompting missing detection of retained image based on intelligent identification |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112862754A true CN112862754A (en) | 2021-05-28 |
Family
ID=76003850
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110008261.8A Pending CN112862754A (en) | 2021-01-05 | 2021-01-05 | System and method for prompting missing detection of retained image based on intelligent identification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112862754A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115546206A (en) * | 2022-11-23 | 2022-12-30 | 武汉楚精灵医疗科技有限公司 | Biopsy image retention method and device for gastrointestinal endoscope image and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107967946A (en) * | 2017-12-21 | 2018-04-27 | 武汉大学 | Operating gastroscope real-time auxiliary system and method based on deep learning |
| CN109146884A (en) * | 2018-11-16 | 2019-01-04 | 青岛美迪康数字工程有限公司 | Endoscopy monitoring method and device |
| CN109272483A (en) * | 2018-08-01 | 2019-01-25 | 安翰光电技术（武汉）有限公司 | Capsule endoscope diagosis and the system and control method of quality control |
| CN109460717A (en) * | 2018-10-22 | 2019-03-12 | 青岛美迪康数字工程有限公司 | Alimentary canal Laser scanning confocal microscope lesion image-recognizing method and device |
| CN109614995A (en) * | 2018-11-28 | 2019-04-12 | 武汉大学人民医院（湖北省人民医院） | The system and method for pancreatic duct and pancreas structure is identified under a kind of endoscopic ultrasonography |
| CN109859827A (en) * | 2018-12-14 | 2019-06-07 | 上海珍灵医疗科技有限公司 | Gastrointestinal Endoscopes operation horizontal points-scoring system and method in real time |
| CN110136808A (en) * | 2019-05-23 | 2019-08-16 | 安翰科技(武汉)股份有限公司 | A kind of filming apparatus auxiliary display system |
| CN110613417A (en) * | 2019-09-24 | 2019-12-27 | 浙江同花顺智能科技有限公司 | Method, equipment and storage medium for outputting upper digestion endoscope operation information |
-
2021
- 2021-01-05 CN CN202110008261.8A patent/CN112862754A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107967946A (en) * | 2017-12-21 | 2018-04-27 | 武汉大学 | Operating gastroscope real-time auxiliary system and method based on deep learning |
| CN109272483A (en) * | 2018-08-01 | 2019-01-25 | 安翰光电技术（武汉）有限公司 | Capsule endoscope diagosis and the system and control method of quality control |
| CN109460717A (en) * | 2018-10-22 | 2019-03-12 | 青岛美迪康数字工程有限公司 | Alimentary canal Laser scanning confocal microscope lesion image-recognizing method and device |
| CN109146884A (en) * | 2018-11-16 | 2019-01-04 | 青岛美迪康数字工程有限公司 | Endoscopy monitoring method and device |
| CN109614995A (en) * | 2018-11-28 | 2019-04-12 | 武汉大学人民医院（湖北省人民医院） | The system and method for pancreatic duct and pancreas structure is identified under a kind of endoscopic ultrasonography |
| CN109859827A (en) * | 2018-12-14 | 2019-06-07 | 上海珍灵医疗科技有限公司 | Gastrointestinal Endoscopes operation horizontal points-scoring system and method in real time |
| CN110136808A (en) * | 2019-05-23 | 2019-08-16 | 安翰科技(武汉)股份有限公司 | A kind of filming apparatus auxiliary display system |
| CN110613417A (en) * | 2019-09-24 | 2019-12-27 | 浙江同花顺智能科技有限公司 | Method, equipment and storage medium for outputting upper digestion endoscope operation information |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115546206A (en) * | 2022-11-23 | 2022-12-30 | 武汉楚精灵医疗科技有限公司 | Biopsy image retention method and device for gastrointestinal endoscope image and storage medium |
| CN115546206B (en) * | 2022-11-23 | 2023-03-14 | 武汉楚精灵医疗科技有限公司 | Biopsy image retention method and device for gastrointestinal endoscope image and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107967946B (en) | Gastroscope operation real-time auxiliary system and method based on deep learning | |
| US7319781B2 (en) | Method and system for multiple passes diagnostic alignment for in vivo images | |
| JP2023171651A (en) | Systems and methods for generating and displaying study information of in vivo image stream | |
| US20220172828A1 (en) | Endoscopic image display method, apparatus, computer device, and storage medium | |
| US8423123B2 (en) | System and method for in-vivo feature detection | |
| CN100377685C (en) | Medical treatment system, endoscope system, endoscope insert operation program, and endoscope device | |
| CN107920722A (en) | Rebuild for the image captured from capsule cameras by object detection | |
| JP5771757B2 (en) | Endoscope system and method for operating endoscope system | |
| CN104114077A (en) | Image processing apparatus and image processing method | |
| US9002083B2 (en) | System, method, and software for optical device recognition association | |
| CN111986196B (en) | Automatic monitoring method and system for retention of gastrointestinal capsule endoscope | |
| CN110867233B (en) | System and method for generating electronic laryngoscope medical test reports | |
| WO2024012080A1 (en) | Endoscope auxiliary examination system, method, apparatus, and storage medium | |
| CN112801958A (en) | Ultrasonic endoscope, artificial intelligence auxiliary identification method, system, terminal and medium | |
| CN109859827A (en) | Gastrointestinal Endoscopes operation horizontal points-scoring system and method in real time | |
| CN111544037B (en) | Ultrasonic positioning method and system based on binocular vision | |
| CN112862754A (en) | System and method for prompting missing detection of retained image based on intelligent identification | |
| CN116797889B (en) | Updating method and device of medical image recognition model and computer equipment | |
| CN116563216B (en) | Endoscope ultrasonic scanning control optimization system and method based on standard site intelligent recognition | |
| WO2021024301A1 (en) | Computer program, processor for endoscope, and information processing method | |
| CN111275754A (en) | Face acne mark proportion calculation method based on deep learning | |
| EP4307311A1 (en) | Information processing system and information processing method | |
| CN115035086A (en) | Intelligent tuberculosis skin test screening and analyzing method and device based on deep learning | |
| Li et al. | Computer aided detection of bleeding in capsule endoscopy images | |
| CN112861909A (en) | Digestive tract mucosa staining detection system and method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |