CN113017702A - Method and system for identifying extension length of small probe of ultrasonic endoscope and storage medium - Google Patents
Method and system for identifying extension length of small probe of ultrasonic endoscope and storage medium Download PDFInfo
- Publication number
- CN113017702A CN113017702A CN202110244849.3A CN202110244849A CN113017702A CN 113017702 A CN113017702 A CN 113017702A CN 202110244849 A CN202110244849 A CN 202110244849A CN 113017702 A CN113017702 A CN 113017702A
- Authority
- CN
- China
- Prior art keywords
- small probe
- image
- eus
- probe
- small
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/42—Details of probe positioning or probe attachment to the patient
- A61B8/4245—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
- A61B8/4254—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient using sensors mounted on the probe
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/44—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device
- A61B8/4444—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device related to the probe
Landscapes
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Endoscopes (AREA)
- Image Processing (AREA)
Abstract
The invention provides a method, a system and a storage medium for identifying the extension length of a small probe of an ultrasonic endoscope, wherein the identification method comprises the following steps: in the optical endoscope image, whether the small probe is too long or not is judged by detecting whether an EUS small probe sheath tube color area exists or not, and if not, the next step is carried out; if the color part of the sheath tube of the EUS small probe does not appear in the optical endoscope image, the area of the EUS small probe is segmented from the image, then the extension length value of the current EUS small probe on the optical endoscope image is calculated for the segmentation result image, and if the extension length value is larger than a set value, a warning is given. By adopting the technical scheme of the invention, the protruding length of the EUS small probe can be displayed on the display screen of the image host of the optical endoscope in real time, and meanwhile, when the EUS small probe is too long, the protruding length of the EUS small probe gives warning to an operating doctor in advance, so that the top end of the small probe is prevented from touching the outer wall of soft tissue, the operation of the doctor can be better supported, and the operation is more reliable.
Description
Technical Field
The invention relates to the technical field of optics and ultrasonic endoscopes, in particular to a method, a system and a storage medium for identifying the extension length of a small probe of an ultrasonic endoscope, and particularly relates to a method, a system and a storage medium for identifying the extension length of a small probe of an ultrasonic endoscope (EUS small probe) under an optical endoscope image.
Background
An ultrasonic Endoscope (EUS) is characterized in that a long miniature high-frequency ultrasonic small probe is inserted into a body cavity through a biopsy channel of an optical endoscope (called an endoscope for short), the gastrointestinal mucosa lesion is observed on an endoscope image, and the ultrasonic small probe scans in real time to obtain the histological characteristics of the hierarchical structure of the gastrointestinal tract and the ultrasonic images of surrounding adjacent organs, so that the diagnosis level of the endoscope and the ultrasonic is further improved. It combines the advantages of optics and ultrasound and provides a comprehensive and multi-level structure of tissue and organ imaging for doctors. An ultrasonic endoscope small probe (EUS small probe) is imaged by the optical endoscope and displayed on an image host display screen of the optical endoscope. When a doctor operates an endoscope at present, the doctor needs to observe the imaging of the endoscope at the same time, and also needs to pay attention to the stretching condition of the small probe of the EUS, on one hand, the small probe needs to be stretched out for a certain length to be positioned near a suspicious lesion, on the other hand, the small probe needs to be controlled not to be stretched out for too long, and the probe is prevented from touching a human body forwards. The EUS small probe extension length prompt and the EUS small probe extension overlength prompt can assist a doctor to operate the EUS small probe better, and the burden of the doctor is relieved. However, the conventional endoscope image cannot be measured at present, and the length of the EUS small probe extending out of the channel outlet of the endoscope instrument cannot be acquired in real time.
Disclosure of Invention
Aiming at the technical problems, the invention discloses a method, a system and a storage medium for identifying the extension length of a small probe of an ultrasonic endoscope, which can display the extension length of the small probe of the EUS in real time and give a warning when the extension length is too long.
In contrast, the technical scheme adopted by the invention is as follows:
a method for identifying the extension length of a small probe of an ultrasonic endoscope comprises the following steps:
step S1, in the optical endoscope (hereinafter referred to as "endoscope") image, judging whether the small probe is too long by detecting whether there is a color area of the sheath tube of the small probe, otherwise, entering the next step;
and step S2, if the color part of the sheath tube of the small probe does not appear in the endoscopic image, segmenting the area of the EUS small probe from the image, then calculating the extension length value of the current EUS small probe on the endoscopic image according to the segmentation result image, and giving an alarm if the extension length value is larger than a set value.
As a further improvement of the present invention, step S1 includes converting the endoscope RGB image into a YUV color space, then retrieving a blue region in the image based on the YUV color space, where the blue region is a region where the blue region at the rear end of the sheath of the small probe appears on the endoscope image, continuously detecting whether the area of several frames of blue regions exceeds a preset area threshold blunareth, and if so, determining that the probe is too long, and giving a warning.
As a further improvement of the invention, when the color part of the sheath tube of the small probe does not appear in the endoscope image, when the extension length value Probenen of the current small probe is larger than or equal to thresh, a warning is given, wherein the thresh is the physical length-sigma of the transparent end of the sheath tube at the front end of the small probe, and sigma is the set relative offset. Furthermore, the physical length of the transparent end of the sheath tube at the front end of the small probe is 35 mm; further, σ is set to 3.
As a further improvement of the invention, an EUS small probe target semantic segmentation module of an optical endoscope image is adopted to segment the area of the EUS small probe from the image, and an EUS small probe extension length estimation module is adopted to calculate the extension length value of the current small probe on the optical endoscope image for the segmentation result image;
the semantic segmentation module of the EUS small probe target of the optical endoscope image adopts a Convolutional Neural Network (CNN) model to perform semantic segmentation of the small probe target, and the area of the EUS small probe is segmented from the image;
the EUS small probe extension length estimation module is used for further carrying out binarization on a segmentation result image output by an EUS small probe target semantic segmentation module to obtain a binary mask (mask) containing an EUS small probe region, a completely communicated small probe region is obtained through morphological operation, through communicated block analysis, a top (foremost) midpoint P1(xp1, yp1) and a bottom (tail) midpoint P2(xp2, yp2) of the mask of the small probe region are obtained, and the pixel Euclidean distance pixelliDist is calculated according to the top and bottom position points and the following formula;
wherein, pixelit is the length of the pixel extending out of the EUS small probe on the current endoscopic image, namely the extending length value of the current small probe.
As a further improvement of the present invention, the step of performing a prediction process by the Convolutional Neural Network (CNN) model includes:
preparing a sample data set: acquiring internal scene images containing different extending lengths of the small probe as data samples of an EUS small probe target semantic segmentation module of the optical endoscope image, wherein the actual extending physical length of the small probe on each sample image is measured and recorded by a caliper in advance, and each image has a unique physical extending length value Probelen corresponding to the actual extending length physical value;
data preprocessing and labeling: making a compromise on the speed and accuracy of segmentation, resampling all endoscopic image sample data to the size of 128 × 3(RGB three channels), labeling the image samples, labeling the area belonging to a small probe part, making labeled data, dividing the labeled data into two parts, wherein the sizes of the two parts are M, N respectively, performing data enhancement processing on M samples to obtain M1 samples serving as a training set, and taking N samples serving as a test set; training the data sample by using a semantic segmentation network with a coding and decoding structure, such as a SegNet model, a Unet model and the like;
adjusting the training parameters and the data sample composition according to the loss values and the accuracy of the required training set and the required test set to obtain an optimal network model and a weight file;
and when the process is executed on line, loading the trained semantic segmentation model and the weight file, inputting an endoscope image (128 x 3), outputting a segmentation result image (128 x 128) in real time from end to end, and automatically marking the probability of the target area of the EUS small probe by the model in the result image.
The invention also discloses that for each endoscopic image in the offline data sample, the pixel length pixelDist and the actual ProbeLen which extend out from the EUS small probe correspond to each endoscopic image, the training set has M groups of data pairs (pixelDist, ProbeLen), and the following linear relation is obtained through least square linear fitting:
ProbeLen＝a*pixelDist+b (2)
wherein a and b are parameters obtained by fitting;
in the online execution process, the actual physical length of the EUS small probe extending out of the current endoscopic image is obtained in real time through the calculation in the formula (2) and the numerical value prompt is displayed.
The invention also discloses a system for identifying the extension length of the small probe of the ultrasonic endoscope, which comprises the following components:
the small probe extending overlength judging module is used for judging whether the small probe extends overlength or not by detecting whether a small probe sheath tube color area exists or not in an endoscope image;
and if the color part of the sheath tube of the small probe does not appear in the endoscopic image, calculating the extension length value of the current small probe on the endoscopic image through the optical endoscopic image EUS small probe target semantic segmentation module and the EUS small probe extension length estimation module, and giving a warning if the extension length value is greater than a set value.
As a further improvement of the invention, the module for judging the protrusion and overlength of the small probe based on the color characteristics converts an endoscope RGB image into a YUV color space, then searches a blue area in the image based on the YUV color space, wherein the blue area is the area of the blue area at the rear end of the small probe sheath tube on the endoscope image, continuously detects whether the areas of a plurality of frames of blue areas exceed a preset area threshold BlunaAreaTh, and if so, considers that the probe is too long and gives a warning;
the semantic segmentation module of the EUS small probe target of the optical endoscope image adopts a Convolutional Neural Network (CNN) model to perform semantic segmentation of the small probe target, and the area of the EUS small probe is segmented from the image;
the EUS small probe extension length estimation module is used for further carrying out binarization on a segmentation result image output by an EUS small probe target semantic segmentation module to obtain a binary mask (mask) containing an EUS small probe region, a completely communicated small probe region is obtained through morphological operation, a communicated block analysis is carried out to obtain a top (most front) midpoint P1(xp1, yp1) and a bottom (tail) midpoint P2(xp2, yp2) of the mask of the small probe region, and the pixel Euclidean distance pixelliDist is calculated according to the top and bottom position points and the following formula;
wherein, pixelit is the length of the pixel extending out of the EUS small probe on the current endoscopic image, namely the extending length value of the current small probe.
As a further improvement of the invention, the module for judging the small probe extending-out overlength based on the color characteristics judges whether the small probe extending-out is overlength, and when the color part of the small probe sheath tube does not appear in an endoscope image, when the extending-out length value Probenen of the current small probe is more than or equal to thresh, wherein the thresh is the physical length-sigma of the transparent end of the small probe front end sheath tube, a warning is given. Where σ is the set relative offset.
As a further improvement of the present invention, the step of performing a prediction process by the Convolutional Neural Network (CNN) model includes:
preparing a sample data set: acquiring endoscope images containing different extending lengths of EUS small probes as data samples of an optical endoscope image EUS small probe target semantic segmentation module, wherein the actual extending physical length of the small probes on each sample image is measured and recorded by a caliper in advance, and each image has a unique physical extending length value Probelen corresponding to the actual extending length physical value;
data preprocessing and labeling: making a compromise on the speed and accuracy of segmentation, resampling all endoscopic image sample data to a set size, labeling image samples, labeling an area belonging to a small probe part, making labeled data, dividing the labeled data into two parts with the sizes of M, N respectively, performing data enhancement processing on M samples to obtain M1 samples serving as a training set, and taking N samples serving as a test set; training the data sample by using a semantic segmentation network with a coding and decoding structure, such as a SegNet model, a Unet model and the like; preferably, the predetermined size is 128 × 3(RGB three channels).
Adjusting the training parameters and the data sample composition according to the loss values and the accuracy of the required training set and the required test set to obtain an optimal network model and a weight file;
when the process is executed on line, the trained semantic segmentation model and the weight file are loaded, an endoscope image (such as 128 x 3) is input, a segmentation result image (such as 128 x 128) is output in real time from end to end, and the model in the result image automatically marks the probability of the target area of the EUS small probe.
As a further improvement of the invention, for each endoscopic image in an offline data sample, corresponding to the pixel length pixesdt of the small probe head of EUS and the actual Probelen, the training set has M groups of data pairs (pixesdt, Probelen), and the following linear relationship is obtained by least square linear fitting:
ProbeLen＝a*pixelDist+b (2)
wherein a and b are parameters obtained by fitting;
in the online execution process, the actual physical length of the EUS small probe extending out of the current endoscopic image is obtained in real time through the calculation in the formula (2) and the numerical value prompt is displayed.
The invention also discloses a computer readable storage medium, wherein the computer readable storage medium stores instructions, and when the instructions are run on the optical endoscope equipment, the optical endoscope equipment is enabled to execute the method for identifying the extension length of the small probe of the ultrasonic endoscope.
The invention also discloses an optical endoscope, which comprises: the system comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the processor executes the computer program, the method for identifying the extension length of the small probe of the ultrasonic endoscope is realized.
Compared with the prior art, the invention has the beneficial effects that:
by adopting the technical scheme of the invention, the extension length of the ultrasonic small probe (EUS) can be displayed on the display screen of the image host of the optical endoscope in real time, and meanwhile, when the EUS small probe is too long, the alarm is given to an operating doctor in advance, so that the top end of the small probe is prevented from touching the outer wall of soft tissue, the operation of the doctor can be better supported, and the operation is more reliable.
Drawings
Fig. 1 is a basic flow diagram of the determination process for the determination of the small probe protrusion length of the EUS according to the embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a small EUS probe in an optical endoscope according to an embodiment of the present invention.
Fig. 3 is a flow chart of a color-based EUS small probe overhang excess determination process according to an embodiment of the present invention.
Fig. 4 is a flowchart of the EUS small probe protrusion length determination process based on the estimated length according to an embodiment of the present invention.
FIG. 5 is a flow diagram of an EUS small probe target semantic segmentation module according to an embodiment of the present invention.
The reference numerals include:
1-ultrasonic imaging unit, 2-blue part of sheath tube (rear end of sheath tube) of EUS small probe, 3-end surface of insertion part of optical endoscope body, 4-insertion part of optical endoscope body; a is the distance between the front end of the spring seat and the foremost end of the transparent part of the outer sheath tube; b-transparent part of outer sheath tube.
Detailed Description
Preferred embodiments of the present invention are described in further detail below.
A method for identifying the extension length of a small probe of an ultrasonic endoscope is shown in figure 1 and comprises the following steps:
1. small probe extending-out overlong discrimination processing based on color characteristics
As shown in fig. 2, the sheath tube of the EUS small probe comprises a sheath tube transparent part B with a certain length, namely the front end of the sheath tube, the rear end of the sheath tube transparent part B is an EUS small probe sheath tube blue part 2, namely a blue EUS colored sheath tube, wherein the distance A between the front end of the spring seat and the foremost end of the transparent part of the sheath tube is 2 +/-1 mm, the length of the sheath tube transparent part B is 34 +/-2 mm, the transparent end of the sheath tube of the EUS small probe comprises an ultrasonic imaging unit 1 (transducer), the end surface 3 of the optical endoscope body insertion part 4 of the optical endoscope body insertion part has a clamp channel outlet which passes through the EUS small probe, and when the blue EUS sheath tube appears in an image, the EUS probe is shown to extend out for more than 35 mm. Based on the practical theory, the method for judging whether the probe is too long according to the steps shown in fig. 3 specifically comprises the following steps:
firstly, converting an endoscope RGB image into a YUV color space, then searching a blue area in the image based on the YUV color space, wherein the blue area is an area where the blue area at the rear end of a small probe sheath tube appears on the endoscope image, continuously detecting whether the area of a plurality of frames of blue areas exceeds a preset area threshold BlunaAreaTh, and if so, determining that the probe is too long to stretch out and giving a warning; otherwise, entering an EUS small probe extending overlong judging processing flow based on the estimated length.
2. Small probe extending-out overlength discrimination processing based on estimated length
When the blue part of the sheath tube does not appear in the endoscopic image, the extended length value of the current small probe on the endoscopic image is calculated through the endoscopic image EUS small probe target semantic segmentation module and the EUS small probe extended length estimation module, and if the length value Probelen is more than or equal to thresh, wherein the thresh is 35-sigma, a warning is given. Where 35 denotes the physical length of the transparent end of the outer sheath at the distal end of the small probe, and σ denotes the relative offset amount, where σ is 3. The discrimination process is shown in fig. 4 and mainly comprises semantic segmentation performed by an EUS small probe target semantic segmentation module, extension estimation performed by an EUS small probe extension estimation module, and discrimination of the extension length of a small probe.
(1) Semantic segmentation by EUS small probe target semantic segmentation module
And (3) carrying out small probe target semantic segmentation by adopting a Convolutional Neural Network (CNN) model in EUS small probe target segmentation on the endoscopic image, and segmenting the area of the EUS small probe from the image. The model-based training prediction process of the present invention, as shown in fig. 5, comprises the following steps:
(a) preparing a sample data set: acquiring internal scene images containing different extension lengths of the small probe as data samples of an EUS (endoscopic image) small probe target semantic segmentation module, wherein the actual extension physical length of the small probe is measured and recorded by a caliper in advance on each sample image, and each image has a unique physical extension length value ProbeLen corresponding to the actual extension length physical value ProbeLen (unit mm);
(b) data preprocessing and labeling: the invention makes a compromise on the speed and accuracy of segmentation, and resamples all endoscopic image sample data into the size of 128 × 3(RGB three channels) and then labels the sample data. Marking image samples, manually marking areas belonging to a small probe part, making labeled data, dividing the labeled data into two parts, wherein the sizes of the two parts are M and N respectively, and because the data volume of medical images is limited, the method performs rotation, scaling, overturning and shifting on the sample data in a certain proportion, performs data enhancement on the M samples to obtain M1 samples serving as training sets, and uses the N samples as test sets;
(c) training a semantic segmentation model: training data samples by using the existing widely used semantic segmentation network with a coding and decoding structure, such as SegNet, Unet and other models;
adjusting the training parameters and the data sample composition according to the loss values and the accuracy of the required training set and the required test set to obtain an optimal network model and a weight file;
(d) the semantic segmentation model executes: when the online execution process is carried out, the trained semantic segmentation model and the weight file are loaded, an endoscope image (128 x 3) is input, a segmentation result image (128 x 128) is output in real time from end to end, and the model in the result image automatically marks the probability of the target area of the EUS small probe.
(2) EUS small probe extension length estimation module estimates extension length
(a) Performing binarization on a segmentation result image output by a semantic segmentation network in a EUS small probe target semantic segmentation module on an endoscopic image to obtain a binary mask (mask) containing an EUS small probe region, obtaining a completely communicated small probe region through morphological operation, obtaining a top (foremost) midpoint P1(xp1, yp1) and a bottom (tail) midpoint P2(xp2, yp2) of the small probe region mask through communicated block analysis, and calculating a pixel Euclidean distance pixelDist according to the following formula (1) according to the top and bottom position points;
wherein, pixelist is the pixel length of the EUS small probe extending out on the current endoscope image.
(b) Thus, for each endoscopic image in the offline sample data, the pixel length pixelDist and the actual Probelen of the small probe extension of the EUS are obtained. The training set has M groups of data pairs (pixedit, ProbeLen). The problem that the protruding length of the transparent end of the EUS small probe on an endoscopic image does not exceed 35mm (blue sheath tubes can appear when the protruding length exceeds 35mm, and the protruding length is directly too long in the prompt) is considered, and the problem that the front end is bent after protruding is not considered. The endoscope imaging principle is the same as that of a common camera, the position coordinate and the size (Probe len) of a small probe real object in a three-dimensional space and the position coordinate and the size (PixelDist) on an image are determined by the external reference and the internal reference of an endoscope (camera) together, in the process of operation, in the process of moving and extending the probe through an endoscope instrument channel, the extending position is fixed in the visual field of the endoscope, only the extending change of the probe exists, the moving probe of the endoscope can move along with the movement, and therefore, in the process of moving the small probe, the endoscope (camera) can be considered to be relatively static, and the internal reference and the external reference of the endoscope are not changed. The physical dimensions of the small probe in the three-dimensional space, ProbeLen and pixellit, are only a simple linear relationship, and a group of linear relationships can be obtained by least squares linear fitting according to M groups of data pairs (pixellit, ProbeLen) in the training set:
ProbeLen＝a*pixelDist+b (2)，
wherein a and b are the fitted parameters.
(c) In the online execution process, the pixel length of the EUS small probe extending on the current endoscope image is obtained in real time through the step (a), and the actual physical length (unit mm) of the EUS small probe extending on the current endoscope image is obtained in real time through a calculation formula in the step (b) and displayed with a numerical prompt.
(3) Discrimination of small probe extending too long
The length value of the current small probe on the endoscope image is calculated through the EUS small probe target semantic segmentation module and the EUS small probe extension length estimation module, and if the length value Probelen is larger than or equal to thresh which is 35-sigma, warning is given. Where 35 denotes the physical length of the transparent end of the outer sheath at the distal end of the small probe, and σ denotes the relative offset amount, where σ is 3.
The invention also discloses a system for identifying the extension length of the small probe of the ultrasonic endoscope, which comprises the following components:
the small probe extending overlength judging module is used for judging whether the small probe extends overlength or not by detecting whether a small probe sheath tube color area exists or not in an endoscope image;
and if the color part of the sheath tube of the small probe does not appear in the endoscopic image, calculating the extension length value of the current small probe on the endoscopic image through the endoscopic image EUS small probe target semantic segmentation module and the EUS small probe extension length estimation module, and giving a warning if the extension length value is greater than a set value.
Further, the module for judging the protrusion and overlength of the small probe based on the color features converts an endoscope RGB image into a YUV color space, then searches a blue area in the image based on the YUV color space, wherein the blue area is the area of the blue area at the rear end of the small probe sheath tube on the endoscope image, continuously detects whether the areas of a plurality of frames of blue areas exceed a preset area threshold BlunaAreaTh, and if so, considers that the probe is too long and gives a warning;
the endoscope image EUS small probe target semantic segmentation module performs small probe target semantic segmentation by adopting a Convolutional Neural Network (CNN) model, and segments the area of the EUS small probe from the image;
the EUS small probe extending length estimation module is used for further carrying out binarization on a segmentation result image output by an EUS small probe target semantic segmentation module of an endoscopic image to obtain a binary mask (mask) containing an EUS small probe region, a completely communicated small probe region is obtained through morphological operation, a top (foremost) midpoint P1(xp1, yp1) and a bottom (tail) midpoint P2(xp2, yp2) of the mask of the small probe region are obtained through communicated block analysis, and the pixel Euclidean distance pixellist is calculated according to the top and bottom position points and the following formula;
wherein, pixelit is the length of the pixel extending out of the EUS small probe on the current endoscopic image, namely the extending length value of the current small probe.
The judgment module for the small probe extending-out overlength based on the color characteristics judges whether the small probe extends out excessively or not, and gives a warning when the extending-out length value Probelen of the current small probe is not less than thresh if the color part of the small probe outer sheath tube does not appear in an endoscope image. Wherein thresh is the physical length-sigma of the transparent end of the sheath tube at the front end of the small probe, and sigma is a set relative offset.
Further, the step of performing a prediction process by the Convolutional Neural Network (CNN) model includes:
preparing a sample data set: acquiring internal scene images containing different extension lengths of the small probe as data samples of an EUS (endoscopic image) small probe target semantic segmentation module, wherein on each sample image, the actual extension physical length of the small probe is measured and recorded by a caliper in advance, and each image has a unique actual extension length physical value Probelen corresponding to the actual extension length physical value;
data preprocessing and labeling: making a compromise on the speed and accuracy of segmentation, resampling all endoscopic image sample data to a set size, labeling the image samples, labeling an area belonging to a small probe part, making labeled data, dividing the labeled data into two parts with the sizes of M, N respectively, performing data enhancement processing on M samples to obtain M1 samples serving as a training set, and taking N samples serving as a test set; training the data sample by using a semantic segmentation network with a coding and decoding structure, such as a SegNet model, a Unet model and the like;
adjusting the training parameters and the data sample composition according to the loss values and the accuracy of the required training set and the required test set to obtain an optimal network model and a weight file;
and when the process is executed on line, loading the trained semantic segmentation model and the weight file, inputting an endoscope image, outputting a segmentation result image in an end-to-end real-time manner, and automatically marking the probability of the target area of the EUS small probe by the model in the result image.
For each endoscopic image in the offline data sample, the pixel length pixesst extended by the EUS small probe and the actual ProbeLen correspond to each endoscopic image, the training set has M groups of data pairs (pixesst, ProbeLen), and the following linear relationship is obtained through least square linear fitting:
ProbeLen＝a*pixelDist+b (2)
wherein a and b are parameters obtained by fitting;
in the online execution process, the actual physical length of the EUS small probe extending out of the current endoscopic image is obtained in real time through the calculation in the formula (2) and the numerical value prompt is displayed.
The invention also discloses a computer readable storage medium, wherein the computer readable storage medium stores instructions, and when the instructions are run on an optical endoscope device, the endoscope device is caused to execute the identification method of the extension length of the ultrasonic small probe.
The invention also discloses an optical endoscope, which comprises: the system comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the processor executes the computer program, the method for identifying the extension length of the small probe of the ultrasonic endoscope is realized.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (10)
1. A method for identifying the extension length of a small probe of an ultrasonic endoscope is characterized by comprising the following steps:
step S1, judging whether the small probe is too long or not by detecting whether a color area of the sheath tube of the small probe exists in the optical endoscope image, and if not, entering the next step;
and step S2, if the color part of the sheath tube of the small probe does not appear in the optical endoscope image, segmenting the area of the EUS small probe from the image, then calculating the extension length value of the current small probe on the optical endoscope image according to the segmentation result image, and giving an alarm if the extension length value is larger than a set value.
2. The method for identifying the extension length of a small ultrasonic endoscope probe according to claim 1, characterized in that: step S1 includes converting the RGB image of the optical endoscope into a YUV color space, then retrieving a blue region in the image based on the YUV color space, where the blue region is a region where the blue region at the rear end of the sheath of the small probe appears on the image of the optical endoscope, continuously detecting whether the area of the blue region of a plurality of frames exceeds a preset area threshold blunareth, and if so, determining that the probe is too long, and giving a warning.
3. The method for identifying the extension length of a small ultrasonic endoscope probe according to claim 2, characterized in that: and if the color part of the sheath tube of the small probe does not appear in the image of the optical endoscope, giving an alarm when the extension length value Probenen of the current small probe is larger than or equal to thresh, wherein thresh is the physical length-sigma of the transparent end of the sheath tube at the front end of the small probe, and sigma is the set relative offset.
4. The method for identifying the extension length of a small probe of an ultrasonic endoscope according to any one of claims 1-3, characterized in that: an EUS small probe target semantic segmentation module of an optical endoscope image is adopted to segment the area of the EUS small probe from the image, and an EUS small probe extension length estimation module is adopted to calculate the extension length value of the current small probe on the optical endoscope image for the segmentation result image;
the semantic segmentation module of the small probe target of the optical endoscope image EUS adopts a convolutional neural network model to perform semantic segmentation of the small probe target, and the region of the small probe is segmented from the image;
the EUS small probe extension length estimation module is used for further carrying out binarization on a segmentation result image output by an EUS small probe target semantic segmentation module of an endoscopic image to obtain a binary mask containing an EUS small probe region, obtaining a completely communicated small probe region through morphological operation, obtaining a top midpoint P1(xp1, yp1) and a bottom midpoint P2(xp2, yp2) of the small probe region mask through communicated block analysis, and calculating the pixel Euclidean distance pixelDist according to the following formula;
wherein, pixelit is the length of the pixel extending out of the EUS small probe on the current optical endoscope image, namely the extending length value of the current small probe.
5. The method for identifying the extension length of a small ultrasonic endoscope probe according to claim 4, characterized in that: the step of the convolutional neural network model for the prediction process comprises the following steps:
preparing a sample data set: acquiring optical endoscope images containing different extending lengths of the small probe as data samples of an EUS small probe target semantic segmentation module, wherein the actual extending physical length of the small probe is measured and recorded by a caliper in advance on each sample image, and each image has a unique physical extending length value Probelen corresponding to the actual extending length physical value;
data preprocessing and labeling: making a compromise on the speed and accuracy of segmentation, resampling all optical endoscope image sample data to a set size, labeling the image samples, labeling an area belonging to a small probe part, making labeled data, dividing the labeled data into two parts with the sizes of M, N respectively, performing data enhancement processing on M samples to obtain M1 samples serving as a training set, and taking N samples serving as a test set; training the data sample by using a semantic segmentation network model;
adjusting the training parameters and the data sample composition according to the loss values and the accuracy of the required training set and the required test set to obtain an optimal network model and a weight file;
when the process is executed on line, loading the trained semantic segmentation model and the weight file, inputting an optical endoscope image, outputting a segmentation result image in real time from end to end, and automatically marking the probability of the target area of the EUS small probe by the model in the result image;
for each optical endoscope image in the offline data sample, the pixel length pixelDist of the EUS small probe extending and the actual ProbeLen correspond to each other, the training set has M groups of data pairs (pixelDist, ProbeLen), and the following linear relationship is obtained through least square linear fitting:
ProbeLen＝a*pixelDist+b (2)
wherein a and b are parameters obtained by fitting;
in the online execution process, the actual physical length of the EUS small probe extending out of the current optical endoscope image is obtained in real time through the calculation in the formula (2), and the numerical value prompt is displayed.
6. A system for identifying the extension length of a small probe of an ultrasonic endoscope is characterized by comprising:
the small probe extending overlength judging module is used for judging whether the small probe extends overlength or not by detecting whether a small probe sheath tube color area exists or not in an optical endoscope image;
and if the color part of the sheath tube of the small probe does not appear in the optical endoscope image, calculating the extension length value of the current small probe on the optical endoscope image through the endoscope image EUS small probe target semantic segmentation module and the EUS small probe extension length estimation module, and giving a warning if the extension length value is greater than a set value.
7. The system for identifying the extension length of a small ultrasound endoscope probe according to claim 6, comprising: the module for judging the protrusion and the overlength of the small probe based on the color characteristics converts an RGB image of the optical endoscope into a YUV color space, then searches a blue area in the image based on the YUV color space, wherein the blue area is the area of the blue area at the rear end of the sheath tube of the small probe on the image of the optical endoscope, continuously detects whether the areas of a plurality of frames of blue areas exceed a preset area threshold BlunaAreaTh, and if so, considers that the probe is too long and gives a warning;
the semantic segmentation module of the EUS small probe target of the optical endoscope image adopts a convolutional neural network model to perform semantic segmentation of the small probe target, and the region of the EUS small probe is segmented from the image;
the EUS small probe extension length estimation module is used for further carrying out binarization on a segmentation result image output by the EUS small probe target semantic segmentation module to obtain a binary mask containing an EUS small probe region, a completely communicated small probe region is obtained through morphological operation, a top midpoint P1(xp1, yp1) and a bottom midpoint P2(xp2, yp2) of the small probe region mask are obtained through communicated block analysis, and the pixel Euclidean distance pixelDist is calculated according to the following formula;
wherein, pixelit is the length of the pixel extending out of the EUS small probe on the current optical endoscope image, namely the extending length value of the current small probe.
8. The system for identifying the extension length of a small ultrasound endoscope probe according to claim 7, comprising: the judging module for the small probe extending-out overlength based on the color characteristics judges whether the small probe extends out excessively, and gives a warning when the extending-out length value Probenen of the current small probe is not less than thresh if the color part of the small probe sheath tube does not appear in the optical endoscope image, wherein the thresh is the physical length-sigma of the transparent end of the small probe front end sheath tube; wherein σ is the set relative offset;
the step of the convolutional neural network model for the prediction process comprises the following steps:
preparing a sample data set: acquiring optical endoscope images containing different extending lengths of the small probe as data samples of an optical endoscope image EUS small probe target semantic segmentation module, wherein the actual extending physical length of the small probe on each sample image is measured and recorded by a caliper in advance, and each image has a unique actual extending length physical value Probelen corresponding to the actual extending length physical value;
data preprocessing and labeling: making a compromise on the speed and accuracy of segmentation, resampling all optical endoscope image sample data to a set size, labeling the image samples, labeling an area belonging to a small probe part, making labeled data, dividing the labeled data into two parts with the sizes of M, N respectively, performing data enhancement processing on M samples to obtain M1 samples serving as a training set, and taking N samples serving as a test set; training the data sample by using a semantic segmentation network model;
adjusting the training parameters and the data sample composition according to the loss values and the accuracy of the required training set and the required test set to obtain an optimal network model and a weight file;
when the process is executed on line, loading the trained semantic segmentation model and the weight file, inputting an optical endoscope image, outputting a segmentation result image in real time from end to end, and automatically marking the probability of the target area of the EUS small probe by the model in the result image;
for each optical endoscope image in the offline data sample, the pixel length pixelDist of the EUS small probe extending and the actual ProbeLen correspond to each other, the training set has M groups of data pairs (pixelDist, ProbeLen), and the following linear relationship is obtained through least square linear fitting:
ProbeLen＝a*pixelDist+b (2)
wherein a and b are parameters obtained by fitting;
in the online execution process, the actual physical length of the EUS small probe extending out of the current optical endoscope image is obtained in real time through the calculation in the formula (2), and the numerical value prompt is displayed.
9. A computer-readable storage medium having stored therein instructions that, when run on an optical endoscope apparatus, cause the optical endoscope apparatus to perform the method for identifying an extension length of a small probe of an ultrasonic endoscope according to any one of claims 1-5.
10. An optical endoscope, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the method for identifying an extension length of a small probe of an ultrasonic endoscope according to any one of claims 1-5 when executing the computer program.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110244849.3A CN113017702B (en) | 2021-03-05 | 2021-03-05 | Method and system for identifying extension length of small probe of ultrasonic endoscope and storage medium |
| PCT/CN2022/079166 WO2022184154A1 (en) | 2021-03-05 | 2022-03-04 | Method and system for recognizing extension length of miniature endoscopic ultrasonography probe, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110244849.3A CN113017702B (en) | 2021-03-05 | 2021-03-05 | Method and system for identifying extension length of small probe of ultrasonic endoscope and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113017702A true CN113017702A (en) | 2021-06-25 |
| CN113017702B CN113017702B (en) | 2022-07-08 |
Family
ID=76467976
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110244849.3A Active CN113017702B (en) | 2021-03-05 | 2021-03-05 | Method and system for identifying extension length of small probe of ultrasonic endoscope and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113017702B (en) |
| WO (1) | WO2022184154A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114052789A (en) * | 2021-11-10 | 2022-02-18 | 深圳英美达医疗技术有限公司 | Probe identification and parameter configuration device and method |
| WO2022184154A1 (en) * | 2021-03-05 | 2022-09-09 | 深圳英美达医疗技术有限公司 | Method and system for recognizing extension length of miniature endoscopic ultrasonography probe, and storage medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117243700B (en) * | 2023-11-20 | 2024-03-08 | 北京云力境安科技有限公司 | Method and related device for detecting endoscope conveying length |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2099581A (en) * | 1981-05-28 | 1982-12-08 | Stanford Res Inst Int | Endoscopic probes, systems and methods |
| EP0284055A2 (en) * | 1987-03-26 | 1988-09-28 | Washington Research Foundation | Endoscopically deliverable ultrasound imaging system |
| US20040092821A1 (en) * | 2000-11-24 | 2004-05-13 | Steffen Hering | Ultrasonic probe with positioning device for examination devices and operation devices |
| WO2007093994A2 (en) * | 2006-02-16 | 2007-08-23 | Vision - Sciences Inc. | Endoscope with imaging capsule |
| CN101828931A (en) * | 2010-02-12 | 2010-09-15 | 广州市番禺区胆囊病研究所 | Miniature ultrasonic electronic hysteroscope system |
| JP2011055938A (en) * | 2009-09-08 | 2011-03-24 | Hoya Corp | Endoscope apparatus |
| CN109145713A (en) * | 2018-07-02 | 2019-01-04 | 南京师范大学 | A kind of Small object semantic segmentation method of combining target detection |
| CN112004496A (en) * | 2018-04-25 | 2020-11-27 | 直观外科手术操作公司 | Systems and methods relating to elongated devices |
| CN212522659U (en) * | 2020-04-09 | 2021-02-12 | 上海微创电生理医疗科技股份有限公司 | Medical intervention catheter part, medical intervention catheter and medical intervention system |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105243664B (en) * | 2015-10-08 | 2018-07-17 | 东南大学 | A kind of wheeled mobile robot fast-moving target tracking method of view-based access control model |
| CN107966497A (en) * | 2017-10-30 | 2018-04-27 | 航天材料及工艺研究所 | A kind of manual ultrasonic visualization non-destructive testing device and method based on camera positioning |
| CN108734211B (en) * | 2018-05-17 | 2019-12-24 | 腾讯科技（深圳）有限公司 | Image processing method and device |
| CN109961448A (en) * | 2019-04-10 | 2019-07-02 | 杭州智团信息技术有限公司 | Sketch the contours method and system in lesion tissue region |
| US10758309B1 (en) * | 2019-07-15 | 2020-09-01 | Digital Surgery Limited | Methods and systems for using computer-vision to enhance surgical tool control during surgeries |
| CN111899252B (en) * | 2020-08-06 | 2023-10-27 | 腾讯科技（深圳）有限公司 | Pathological image processing method and device based on artificial intelligence |
| CN113017702B (en) * | 2021-03-05 | 2022-07-08 | 深圳英美达医疗技术有限公司 | Method and system for identifying extension length of small probe of ultrasonic endoscope and storage medium |
-
2021
- 2021-03-05 CN CN202110244849.3A patent/CN113017702B/en active Active
-
2022
- 2022-03-04 WO PCT/CN2022/079166 patent/WO2022184154A1/en unknown
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2099581A (en) * | 1981-05-28 | 1982-12-08 | Stanford Res Inst Int | Endoscopic probes, systems and methods |
| EP0284055A2 (en) * | 1987-03-26 | 1988-09-28 | Washington Research Foundation | Endoscopically deliverable ultrasound imaging system |
| US20040092821A1 (en) * | 2000-11-24 | 2004-05-13 | Steffen Hering | Ultrasonic probe with positioning device for examination devices and operation devices |
| WO2007093994A2 (en) * | 2006-02-16 | 2007-08-23 | Vision - Sciences Inc. | Endoscope with imaging capsule |
| JP2011055938A (en) * | 2009-09-08 | 2011-03-24 | Hoya Corp | Endoscope apparatus |
| CN101828931A (en) * | 2010-02-12 | 2010-09-15 | 广州市番禺区胆囊病研究所 | Miniature ultrasonic electronic hysteroscope system |
| CN112004496A (en) * | 2018-04-25 | 2020-11-27 | 直观外科手术操作公司 | Systems and methods relating to elongated devices |
| CN109145713A (en) * | 2018-07-02 | 2019-01-04 | 南京师范大学 | A kind of Small object semantic segmentation method of combining target detection |
| CN212522659U (en) * | 2020-04-09 | 2021-02-12 | 上海微创电生理医疗科技股份有限公司 | Medical intervention catheter part, medical intervention catheter and medical intervention system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022184154A1 (en) * | 2021-03-05 | 2022-09-09 | 深圳英美达医疗技术有限公司 | Method and system for recognizing extension length of miniature endoscopic ultrasonography probe, and storage medium |
| CN114052789A (en) * | 2021-11-10 | 2022-02-18 | 深圳英美达医疗技术有限公司 | Probe identification and parameter configuration device and method |
| CN114052789B (en) * | 2021-11-10 | 2023-12-15 | 深圳英美达医疗技术有限公司 | Probe identification and parameter configuration device and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113017702B (en) | 2022-07-08 |
| WO2022184154A1 (en) | 2022-09-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113017702B (en) | Method and system for identifying extension length of small probe of ultrasonic endoscope and storage medium | |
| US7822461B2 (en) | System and method for endoscopic path planning | |
| US7630529B2 (en) | Methods for digital bowel subtraction and polyp detection | |
| US20110032347A1 (en) | Endoscopy system with motion sensors | |
| US7830378B2 (en) | Medical image processing apparatus and medical image processing method | |
| JP4902735B2 (en) | Medical image processing apparatus and medical image processing method | |
| JP5276225B2 (en) | Medical image processing apparatus and method of operating medical image processing apparatus | |
| JP5851160B2 (en) | Image processing apparatus, operation method of image processing apparatus, and image processing program | |
| CN111091559A (en) | Depth learning-based auxiliary diagnosis system for small intestine sub-scope lymphoma | |
| JP2006519631A (en) | Execution system and execution method for virtual endoscopy | |
| WO2005101314A2 (en) | Method and apparatus for processing images in a bowel subtraction system | |
| CN111105873B (en) | Auxiliary diagnosis and measurement method and system in endoscopic retrograde cholangiopancreatography | |
| WO2023143014A1 (en) | Endoscope-assisted inspection method and device based on artificial intelligence | |
| US20220409030A1 (en) | Processing device, endoscope system, and method for processing captured image | |
| JPWO2012153568A1 (en) | Medical image processing device | |
| CN111311626A (en) | Skull fracture automatic detection method based on CT image and electronic medium | |
| CN116958147B (en) | Target area determining method, device and equipment based on depth image characteristics | |
| JP4981335B2 (en) | Medical image processing apparatus and medical image processing method | |
| JPWO2021171464A5 (en) | ||
| JPWO2019088008A1 (en) | Image processing equipment, image processing methods, programs, and endoscopic systems | |
| CN114730478A (en) | System and method for processing electronic medical images to determine enhanced electronic medical images | |
| EP4191527A1 (en) | An endoscope image processing device | |
| CN116807361B (en) | CT image display method, electronic equipment and device | |
| JP2007236630A (en) | Medical image processor and medical image processing method | |
| CN115797348B (en) | Endoscopic target structure evaluation system, method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |