CN110991561B - Method and system for identifying images of endoscope in lower digestive tract - Google Patents
Method and system for identifying images of endoscope in lower digestive tract Download PDFInfo
- Publication number
- CN110991561B CN110991561B CN201911327593.1A CN201911327593A CN110991561B CN 110991561 B CN110991561 B CN 110991561B CN 201911327593 A CN201911327593 A CN 201911327593A CN 110991561 B CN110991561 B CN 110991561B
- Authority
- CN
- China
- Prior art keywords
- category
- image
- endoscope
- digestive tract
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
Abstract
The present disclosure provides a lower gastrointestinal endoscope image recognition method and system. Acquiring a lower digestive tract part image, marking a category to be identified and an auxiliary category for identifying an interference image, and dividing the categories into a training set and a test set; training and testing a lower digestive tract part recognition model A and a lower digestive tract part recognition model B by utilizing a training set and a testing set; in the process of endoscope entering operation, if the type of the current lower gastrointestinal endoscope image is identified by the lower gastrointestinal part identification model A as a cecum in a first-class type, and the probability that N continuous non-similar images are cecum exceeds a preset threshold value, starting endoscope withdrawing operation; n is a positive integer greater than or equal to 3; and in the process of endoscope withdrawal operation, real-time detection is carried out on an endoscope image of the alimentary tract based on the lower alimentary tract part identification model A and the lower alimentary tract part identification model B, images belonging to auxiliary categories are excluded, and a primary category and a secondary category to which the images belong are output.
Description
Technical Field
The disclosure belongs to the field of digestive tract endoscope image identification, and particularly relates to a lower digestive tract endoscope image identification method and system.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
Lower gastrointestinal endoscopy often requires determining where in the colon and rectum a lesion is located. The inventor finds that most of the current lower gastrointestinal endoscopy is human eye observation, and the lesion part cannot be accurately judged. For example, multiple polyps in the lower alimentary tract are found by inspection with the human eye, but cannot be accurately determined whether they are located in the ascending colon, the transverse colon, or the descending or sigmoid colon;
in addition, some parts may be missed during lower gastrointestinal endoscopy, for example, the image quality of a certain part is poor due to the slippery body or the poor filling of the intestinal lumen, so that the lower gastrointestinal endoscopy quality is affected, and the possibility of missed diagnosis of lesions is caused.
Disclosure of Invention
In order to solve the above problems, the present disclosure provides a method and a system for recognizing an image of a lower gastrointestinal endoscope, which can accurately determine a cecum at an endoscope entry end point of the lower gastrointestinal endoscope and recognize a part to which an image acquired in a lower gastrointestinal endoscope withdrawal process belongs in real time.
In order to achieve the purpose, the following technical scheme is adopted in the disclosure:
a first aspect of the present disclosure provides a lower alimentary canal endoscope image recognition method, including:
acquiring a lower digestive tract part image, marking a class to be identified and an auxiliary class for identifying an interference image, and dividing the class into a training set and a testing set; the category to be identified comprises a primary category and a secondary category, and the secondary category belongs to a subcategory of the primary category;
training and testing a lower digestive tract part recognition model A and a lower digestive tract part recognition model B by utilizing a training set and a testing set; the lower digestive tract part identification model A is used for identifying a primary class and an auxiliary class, and the lower digestive tract part identification model B is used for identifying a secondary class;
in the process of endoscope entering operation, if the trained lower gastrointestinal tract part recognition model A recognizes that the type of the current lower gastrointestinal tract endoscope image is a cecum in a first-class category, and the probability that the cecum exists in N continuous non-similar images exceeds a preset threshold value, starting endoscope withdrawing operation; n is a positive integer greater than or equal to 3;
in the process of endoscope withdrawal operation, real-time detection is carried out on an endoscope image of the alimentary tract based on a lower alimentary tract part identification model A and a lower alimentary tract part identification model B, images belonging to auxiliary categories are excluded, and if the probability that the continuous N non-similar images are all in the same category exceeds a preset threshold value, a primary category and a secondary category to which the images belong are output.
A second aspect of the present disclosure provides a lower gastrointestinal endoscope image recognition system, comprising:
the image marking module is used for acquiring the lower digestive tract part image, marking the category to be identified and the auxiliary category for identifying the interference image, and dividing the category into a training set and a testing set; the category to be identified comprises a primary category and a secondary category, and the secondary category belongs to a subcategory of the primary category;
a model training module for training and testing a lower digestive tract part recognition model A and a lower digestive tract part recognition model B by using a training set and a test set; the lower digestive tract part identification model A is used for identifying a primary class and an auxiliary class, and the lower digestive tract part identification model B is used for identifying a secondary class;
a withdrawal operation starting judgment module, which is used for starting withdrawal operation if the trained lower gastrointestinal tract part recognition model A recognizes that the type of the current lower gastrointestinal tract endoscope image is a cecum in a first-class type and the probability that the cecum exists in the continuous N non-similar images exceeds a preset threshold value in the process of the retraction operation; n is a positive integer greater than or equal to 3;
and the image real-time identification module is used for detecting the endoscope image of the alimentary canal in real time based on the lower alimentary canal part identification model A and the lower alimentary canal part identification model B in the process of endoscope withdrawal operation, eliminating the images belonging to the auxiliary category, and outputting the primary category and the secondary category to which the images belong if the probability that the N continuous non-similar images are all in the same category exceeds a preset threshold value.
A third aspect of the present disclosure provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps in the lower gastrointestinal endoscope image recognition method as described above.
A fourth aspect of the present disclosure provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the lower alimentary canal endoscope image recognition method as described above when executing the program.
The beneficial effects of this disclosure are:
(1) the method comprises the steps of obtaining a lower digestive tract part image and marking a category to be identified and an auxiliary category for identifying an interference image; the category to be identified comprises a primary category and a secondary category, and the secondary category belongs to a subcategory of the primary category; the lower digestive tract part identification model A is trained by using the lower digestive tract part images marked with the auxiliary categories and the lower digestive tract part images marked with the primary categories, so that the influence of blurred images, defocused images, reflection artifacts, residual sundries, effusion and the like on image identification is reduced, and the accuracy of lower digestive tract part identification is improved; training a lower digestive tract part recognition model B by marking a lower digestive tract part image of a secondary category, and using a two-stage recognition model to improve the recognition accuracy;
(2) the method also utilizes the probability that the continuous N non-similar images are in the same category and exceeds a preset threshold value as the final category of the images, so as to determine the position reached by the current endoscope head, and improve the accuracy of the position judgment of the endoscope head;
(3) this openly can each anatomical site under intelligent recognition among the alimentary canal endoscopy to with its sign, be convenient for confirm its place colorectal part after discovering pathological change under the endoscope, can discern the interference image and get rid of through alimentary canal position recognition model A down moreover, do not carry out one-level classification and secondary classification, for example: the endoscope inspection quality of the lower digestive tract is guaranteed due to the fact that the endoscope body slips or is poor in filling, the detection rate of pathological changes is improved, the anatomical part where the pathological changes are located can be identified in real time to be marked in time, follow-up treatment and follow-up visit of a patient are facilitated, and the computer-assisted endoscope inspection quality is guaranteed.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure and are not to limit the disclosure.
FIG. 1 is a flow chart of a method for identifying images of a lower alimentary canal endoscope provided by an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a lower gastrointestinal endoscope image recognition system provided in an embodiment of the present disclosure.
Detailed Description
The present disclosure is further described with reference to the following drawings and examples.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
Example 1
Fig. 1 is a flowchart of a lower ablation tract endoscope image recognition method according to the embodiment.
The following describes in detail the specific implementation process of the lower alimentary canal endoscope image recognition method according to this embodiment with reference to fig. 1:
as shown in fig. 1, a method for recognizing an image of a lower alimentary canal endoscope of the present embodiment at least includes:
step S101: acquiring a lower digestive tract part image, marking a class to be identified and an auxiliary class for identifying an interference image, and dividing the class into a training set and a testing set; the category to be identified comprises a first-level category and a second-level category, and the second-level category belongs to a sub-category of the first-level category.
In a specific implementation, the first class includes ileocecal valve, colon, sigmoid colon, and rectum; the secondary classes belong to the sub-classes of the colon, including ascending, transverse and descending; the auxiliary category is a preset interference image category and is used for eliminating interference images;
the auxiliary categories comprise appendix opening, effusion, lens distance smaller than a preset value from the intestinal wall, lens shielding, intestinal cavity contraction, incomplete intestinal cavity and fuzziness.
For all cases during the following examination of the digestive tract, specific classifications are shown, for example, in table 1:
TABLE 1 image Classification and selection principles
The intestinal cavity is poor in inspiration or inflation and good in inflation, and can be judged according to the air volume in the intestinal cavity.
In the embodiment, the category to be detected is the category to be identified for the lower digestive tract examination, and since the background of the lower digestive tract image is single and is easily interfered by special factors, the addition of the auxiliary category is beneficial to eliminating the interference, and the effective image is more accurately screened.
The first class is a primary judgment, and the second class is a secondary judgment by using a fine-grained classification network because the similarity of ascending, transverse and descending colon is too high so as to improve the identification accuracy.
Step S102: training and testing a lower digestive tract part recognition model A and a lower digestive tract part recognition model B by utilizing a training set and a testing set; the lower digestive tract part recognition model A is used for recognizing the primary category and the auxiliary category, and the lower digestive tract part recognition model B is used for recognizing the secondary category.
In specific implementation, the lower digestive tract part identification model A adopts an image classification model provided by a deep learning framework Keras application module;
keras is a highly modular, written in pure Python and backend with Tensorflow, Theano, and CNTK. Keras was generated to support rapid experiments. Keras understands a model as a working graph of sequences or data of one layer, and fully configurable modules can be freely combined together with minimum cost and are also easy to expand.
And (2) training the local area in a characteristic supervision mode by using a multi-branch structure and simultaneously utilizing local information and global information of the lower digestive tract endoscope image in the training set through a fine-grained classification network DFL-CNN (learning a cognitive Filter Bank with a CNN), so as to obtain a lower digestive tract part identification model B.
For the fine-grained classification network DFL-CNN: global information is also crucial for fine-grained classification. One branch is needed to decode the global information. I.e. the normal conv + fc layer. And then selecting a proper higher-layer convolution, and separating out another branch to strengthen mid-level capability and pay attention to local information. The method can accurately locate the key area with resolution and extract effective features from the detected key area for classification.
Step S103: in the process of endoscope entering operation, if the trained lower digestive tract part recognition model A recognizes that the type of the current lower digestive tract endoscope image is a cecum in a first-class category and the probability that the continuous N non-similar images are cecum exceeds a preset threshold (such as 0.8 or 0.9), starting endoscope withdrawing operation; n is a positive integer greater than or equal to 3.
Because the lower digestive tract identification difficulty is high, the part of the single image cannot be accurately judged, and the identification is carried out by adopting a continuous multi-frame non-similar image combined judgment mode.
The specific logic is as follows:
the images collected in the checking process are judged in real time through a lower digestive tract part identification model A, after auxiliary class images are eliminated, similarity comparison is carried out on the continuously identified class images to be detected, and if N continuous non-similar images belong to the same class to be detected, a lens is considered to arrive at the part.
And the similarity calculation logic generates a hash sequence through a mean hash algorithm and calculates a Hamming distance, and when the Hamming distance is greater than a set threshold value, the image is judged to be a non-similar image. The correlation algorithm is as follows:
(a) mean value hash algorithm
Zooming: the picture is scaled to 8 x 8, the structure is preserved, and the details are removed.
Graying: and converting into a 256-step gray scale map.
And (3) averaging: the average of all pixels of the gray map is calculated.
And (3) comparison: the pixel value greater than the average is noted as 1 and conversely as 0 for a total of 64 bits.
Generating a hash: and combining the 1 and 0 generated in the steps in sequence.
(b) Hamming distance calculation
The Hamming Distance/Hamming Distance is used for calculating the similarity of two vectors; that is, by comparing whether each bit of the vector is the same or not, if different, the hamming distance is added by 1, so as to obtain the hamming distance. The higher the vector similarity, the smaller the corresponding hamming distance. For example, positions 10001001 and 10110001 differ by 3.
The lower gastrointestinal examination is divided into two stages of endoscope entering and endoscope withdrawing, the endoscope entering stage does not need to be identified, and the critical point for distinguishing the endoscope entering and the endoscope withdrawing is whether the lens reaches the cecum or not, so the identification accuracy of the cecum is particularly important.
Combining the lower digestive tract part identification model A, and simultaneously reducing the error rate of the blind returning valve identification as much as possible by increasing the threshold, for example, the threshold is set to be 0.9, which means that the output probability of the current frame image model exceeds 90% before the blind returning valve is considered to be reached.
Step S104: in the process of endoscope withdrawal operation, real-time detection is carried out on an endoscope image of the alimentary tract based on a lower alimentary tract part identification model A and a lower alimentary tract part identification model B, images belonging to auxiliary categories are excluded, and if the probability that the continuous N non-similar images are all in the same category exceeds a preset threshold value, a primary category and a secondary category to which the images belong are output.
In a specific implementation, when the lower gastrointestinal tract endoscope image is identified by using the lower gastrointestinal tract part identification model a and the category to which the lower gastrointestinal tract endoscope image belongs is output as a colon, after the auxiliary category image is excluded, if the probability that the N consecutive non-similar images simultaneously belong to the colon exceeds a first preset threshold (for example, 0.9), the lower gastrointestinal tract endoscope image is secondarily judged by using the lower gastrointestinal tract part identification model B, and when the probability that the N consecutive non-similar images output as the same secondary category exceeds a second preset threshold (for example, 0.8), the lens is judged to have reached the secondary category part.
It should be noted that, in other embodiments, the values of the first preset threshold and the second preset threshold may be selectively set by a person skilled in the art according to actual situations.
The embodiment acquires the lower digestive tract part image and marks out the category to be identified and the auxiliary category for identifying the interference image; the category to be identified comprises a primary category and a secondary category, and the secondary category belongs to a subcategory of the primary category; the lower digestive tract part identification model A is trained by using the lower digestive tract part images marked with the auxiliary categories and the lower digestive tract part images marked with the primary categories, so that the influence of blurred images, defocused images, reflection artifacts, residual sundries, effusion and the like on image identification is reduced, and the accuracy of lower digestive tract part identification is improved; training a lower digestive tract part recognition model B by marking a lower digestive tract part image of a secondary category, and using a two-stage recognition model to improve the recognition accuracy;
in the embodiment, the probability that the N continuous non-similar images are in the same category exceeds a preset threshold value is used as the final category of the image, so that the position where the current endoscope head reaches is determined, and the accuracy of judging the position of the endoscope head is improved;
this embodiment can each anatomical site under intelligent recognition among the alimentary canal endoscopy to with its sign, discover under the endoscope that it is convenient for confirm its colorectal part of place after the pathological change, can discern the interference image and get rid of through alimentary canal position recognition model A down moreover, do not carry out one-level classification and secondary classification, for example: the endoscope inspection quality of the lower digestive tract is guaranteed due to the fact that the endoscope body slips or is poor in filling, the detection rate of pathological changes is improved, the anatomical part where the pathological changes are located can be identified in real time to be marked in time, follow-up treatment and follow-up visit of a patient are facilitated, and the computer-assisted endoscope inspection quality is guaranteed.
Example 2
Fig. 2 is a schematic structural diagram of a lower gastrointestinal endoscope image recognition system according to the present embodiment.
The following describes the structural components of the lower gastrointestinal endoscope image recognition system of the present embodiment in detail with reference to fig. 2:
as shown in fig. 2, the lower gastrointestinal endoscope image recognition system of the present embodiment at least includes:
(1) the image marking module is used for acquiring the lower digestive tract part image, marking the category to be identified and the auxiliary category for identifying the interference image, and dividing the category into a training set and a testing set; the category to be identified comprises a first-level category and a second-level category, and the second-level category belongs to a sub-category of the first-level category.
In a specific implementation, the first class includes ileocecal valve, colon, sigmoid colon, and rectum; the secondary classes belong to the sub-classes of the colon, including ascending, transverse and descending; the auxiliary category is a preset interference image category and is used for eliminating interference images;
the auxiliary categories comprise appendix opening, effusion, lens distance smaller than a preset value from the intestinal wall, lens shielding, intestinal cavity contraction, incomplete intestinal cavity and fuzziness.
For all cases during the following examination of the digestive tract, specific classifications are shown, for example, in table 1:
TABLE 1 image Classification and selection principles
The intestinal cavity is poor in inspiration or inflation and good in inflation, and can be judged according to the air volume in the intestinal cavity.
In the embodiment, the category to be detected is the category to be identified for the lower digestive tract examination, and since the background of the lower digestive tract image is single and is easily interfered by special factors, the addition of the auxiliary category is beneficial to eliminating the interference, and the effective image is more accurately screened.
The first class is a primary judgment, and the second class is a secondary judgment by using a fine-grained classification network because the similarity of ascending, transverse and descending colon is too high so as to improve the identification accuracy.
(2) A model training module for training and testing a lower digestive tract part recognition model A and a lower digestive tract part recognition model B by using a training set and a test set; the lower digestive tract part recognition model A is used for recognizing the primary category and the auxiliary category, and the lower digestive tract part recognition model B is used for recognizing the secondary category.
In specific implementation, the lower digestive tract part identification model A adopts an image classification model provided by a deep learning framework Keras application module;
keras is a highly modular, written in pure Python and backend with Tensorflow, Theano, and CNTK. Keras was generated to support rapid experiments. Keras understands a model as a working graph of sequences or data of one layer, and fully configurable modules can be freely combined together with minimum cost and are also easy to expand.
And (2) training the local area in a characteristic supervision mode by using a multi-branch structure and simultaneously utilizing local information and global information of the lower digestive tract endoscope image in the training set through a fine-grained classification network DFL-CNN (learning a cognitive Filter Bank with a CNN), so as to obtain a lower digestive tract part identification model B.
For the fine-grained classification network DFL-CNN: global information is also crucial for fine-grained classification. One branch is needed to decode the global information. I.e. the normal conv + fc layer. And then selecting a proper higher-layer convolution, and separating out another branch to strengthen mid-level capability and pay attention to local information. The method can accurately locate the key area with resolution and extract effective features from the detected key area for classification.
(3) A withdrawal operation starting judgment module, configured to, in a process of a retraction operation, start a withdrawal operation if the trained lower gastrointestinal tract part recognition model a recognizes that the type of the current lower gastrointestinal tract endoscopic image is a cecum in a first-class category and a probability that all of N consecutive non-similar images are cecum exceeds a preset threshold (e.g., 0.8 or 0.9); n is a positive integer greater than or equal to 3.
Because the lower digestive tract identification difficulty is high, the part of the single image cannot be accurately judged, and the identification is carried out by adopting a continuous multi-frame non-similar image combined judgment mode.
The specific logic is as follows:
the images collected in the checking process are judged in real time through a lower digestive tract part identification model A, after auxiliary class images are eliminated, similarity comparison is carried out on the continuously identified class images to be detected, and if N continuous non-similar images belong to the same class to be detected, a lens is considered to arrive at the part.
And the similarity calculation logic generates a hash sequence through a mean hash algorithm and calculates a Hamming distance, and when the Hamming distance is greater than a set threshold value, the image is judged to be a non-similar image. The correlation algorithm is as follows:
(a) mean value hash algorithm
Zooming: the picture is scaled to 8 x 8, the structure is preserved, and the details are removed.
Graying: and converting into a 256-step gray scale map.
And (3) averaging: the average of all pixels of the gray map is calculated.
And (3) comparison: the pixel value greater than the average is noted as 1 and conversely as 0 for a total of 64 bits.
Generating a hash: and combining the 1 and 0 generated in the steps in sequence.
(b) Hamming distance calculation
The Hamming Distance/Hamming Distance is used for calculating the similarity of two vectors; that is, by comparing whether each bit of the vector is the same or not, if different, the hamming distance is added by 1, so as to obtain the hamming distance. The higher the vector similarity, the smaller the corresponding hamming distance. For example, positions 10001001 and 10110001 differ by 3.
The lower gastrointestinal examination is divided into two stages of endoscope entering and endoscope withdrawing, the endoscope entering stage does not need to be identified, and the critical point for distinguishing the endoscope entering and the endoscope withdrawing is whether the lens reaches the cecum or not, so the identification accuracy of the cecum is particularly important.
Combining the lower digestive tract part identification model A, and simultaneously reducing the error rate of the blind returning valve identification as much as possible by increasing the threshold, for example, the threshold is set to be 0.9, which means that the output probability of the current frame image model exceeds 90% before the blind returning valve is considered to be reached.
(4) And the image real-time identification module is used for detecting the endoscope image of the alimentary canal in real time based on the lower alimentary canal part identification model A and the lower alimentary canal part identification model B in the process of endoscope withdrawal operation, eliminating the images belonging to the auxiliary category, and outputting the primary category and the secondary category to which the images belong if the probability that the N continuous non-similar images are all in the same category exceeds a preset threshold value.
In a specific implementation, when the lower gastrointestinal tract endoscope image is identified by using the lower gastrointestinal tract part identification model a and the category to which the lower gastrointestinal tract endoscope image belongs is output as a colon, after the auxiliary category image is excluded, if the probability that the N consecutive non-similar images simultaneously belong to the colon exceeds a first preset threshold (for example, 0.9), the lower gastrointestinal tract endoscope image is secondarily judged by using the lower gastrointestinal tract part identification model B, and when the probability that the N consecutive non-similar images output as the same secondary category exceeds a second preset threshold (for example, 0.8), the lens is judged to have reached the secondary category part.
It should be noted that, in other embodiments, the values of the first preset threshold and the second preset threshold may be selectively set by a person skilled in the art according to actual situations.
The embodiment acquires the lower digestive tract part image and marks out the category to be identified and the auxiliary category for identifying the interference image; the category to be identified comprises a primary category and a secondary category, and the secondary category belongs to a subcategory of the primary category; the lower digestive tract part identification model A is trained by using the lower digestive tract part images marked with the auxiliary categories and the lower digestive tract part images marked with the primary categories, so that the influence of blurred images, defocused images, reflection artifacts, residual sundries, effusion and the like on image identification is reduced, and the accuracy of lower digestive tract part identification is improved; training a lower digestive tract part recognition model B by marking a lower digestive tract part image of a secondary category, and using a two-stage recognition model to improve the recognition accuracy;
in the embodiment, the probability that the N continuous non-similar images are in the same category exceeds a preset threshold value is used as the final category of the image, so that the position where the current endoscope head reaches is determined, and the accuracy of judging the position of the endoscope head is improved;
this embodiment can each anatomical site under intelligent recognition among the alimentary canal endoscopy to with its sign, discover under the endoscope that it is convenient for confirm its colorectal part of place after the pathological change, can discern the interference image and get rid of through alimentary canal position recognition model A down moreover, do not carry out one-level classification and secondary classification, for example: the endoscope inspection quality of the lower digestive tract is guaranteed due to the fact that the endoscope body slips or is poor in filling, the detection rate of pathological changes is improved, the anatomical part where the pathological changes are located can be identified in real time to be marked in time, follow-up treatment and follow-up visit of a patient are facilitated, and the computer-assisted endoscope inspection quality is guaranteed.
Example 3
The present embodiment provides a computer-readable storage medium on which a computer program is stored, characterized in that the program realizes the steps in the lower gastrointestinal endoscope image recognition method shown in fig. 1 when executed by a processor.
The embodiment acquires the lower digestive tract part image and marks out the category to be identified and the auxiliary category for identifying the interference image; the category to be identified comprises a primary category and a secondary category, and the secondary category belongs to a subcategory of the primary category; the lower digestive tract part identification model A is trained by using the lower digestive tract part images marked with the auxiliary categories and the lower digestive tract part images marked with the primary categories, so that the influence of blurred images, defocused images, reflection artifacts, residual sundries, effusion and the like on image identification is reduced, and the accuracy of lower digestive tract part identification is improved; training a lower digestive tract part recognition model B by marking a lower digestive tract part image of a secondary category, and using a two-stage recognition model to improve the recognition accuracy;
in the embodiment, the probability that the N continuous non-similar images are in the same category exceeds a preset threshold value is used as the final category of the image, so that the position where the current endoscope head reaches is determined, and the accuracy of judging the position of the endoscope head is improved;
this embodiment can each anatomical site under intelligent recognition among the alimentary canal endoscopy to with its sign, discover under the endoscope that it is convenient for confirm its colorectal part of place after the pathological change, can discern the interference image and get rid of through alimentary canal position recognition model A down moreover, do not carry out one-level classification and secondary classification, for example: the endoscope inspection quality of the lower digestive tract is guaranteed due to the fact that the endoscope body slips or is poor in filling, the detection rate of pathological changes is improved, the anatomical part where the pathological changes are located can be identified in real time to be marked in time, follow-up treatment and follow-up visit of a patient are facilitated, and the computer-assisted endoscope inspection quality is guaranteed.
Example 4
The present embodiment provides a computer apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the lower gastrointestinal endoscope image recognition method as shown in fig. 1 when executing the program.
The embodiment acquires the lower digestive tract part image and marks out the category to be identified and the auxiliary category for identifying the interference image; the category to be identified comprises a primary category and a secondary category, and the secondary category belongs to a subcategory of the primary category; the lower digestive tract part identification model A is trained by using the lower digestive tract part images marked with the auxiliary categories and the lower digestive tract part images marked with the primary categories, so that the influence of blurred images, defocused images, reflection artifacts, residual sundries, effusion and the like on image identification is reduced, and the accuracy of lower digestive tract part identification is improved; training a lower digestive tract part recognition model B by marking a lower digestive tract part image of a secondary category, and using a two-stage recognition model to improve the recognition accuracy;
in the embodiment, the probability that the N continuous non-similar images are in the same category exceeds a preset threshold value is used as the final category of the image, so that the position where the current endoscope head reaches is determined, and the accuracy of judging the position of the endoscope head is improved;
this embodiment can each anatomical site under intelligent recognition among the alimentary canal endoscopy to with its sign, discover under the endoscope that it is convenient for confirm its colorectal part of place after the pathological change, can discern the interference image and get rid of through alimentary canal position recognition model A down moreover, do not carry out one-level classification and secondary classification, for example: the endoscope inspection quality of the lower digestive tract is guaranteed due to the fact that the endoscope body slips or is poor in filling, the detection rate of pathological changes is improved, the anatomical part where the pathological changes are located can be identified in real time to be marked in time, follow-up treatment and follow-up visit of a patient are facilitated, and the computer-assisted endoscope inspection quality is guaranteed.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.
Claims (7)
1. A method for recognizing images of a lower alimentary canal endoscope is characterized by comprising the following steps:
acquiring a lower digestive tract part image, marking a class to be identified and an auxiliary class for identifying an interference image, and dividing the class into a training set and a testing set; the category to be identified comprises a primary category and a secondary category, and the secondary category belongs to a subcategory of the primary category;
training and testing a lower digestive tract part recognition model A and a lower digestive tract part recognition model B by utilizing a training set and a testing set; the lower digestive tract part identification model A is used for identifying a primary class and an auxiliary class, and the lower digestive tract part identification model B is used for identifying a secondary class;
in the process of endoscope entering operation, if the trained lower gastrointestinal tract part recognition model A recognizes that the type of the current lower gastrointestinal tract endoscope image is a cecum in a first-class category, and the probability that the cecum exists in N continuous non-similar images exceeds a preset threshold value, starting endoscope withdrawing operation; n is a positive integer greater than or equal to 3; the process of judging the image as the non-similar image comprises the following steps: generating a hash sequence through a mean hash algorithm and calculating a Hamming distance, and judging the image to be a non-similar image when the Hamming distance is greater than a set Hamming distance threshold;
in the process of endoscope withdrawal operation, real-time detection is carried out on an endoscope image of the alimentary tract based on a lower alimentary tract part identification model A and a lower alimentary tract part identification model B, images belonging to auxiliary categories are excluded, and if the probability that the continuous N non-similar images are all in the same category exceeds a preset threshold value, a primary category and a secondary category to which the images belong are output;
in the process of endoscope withdrawing operation, when a lower gastrointestinal endoscope image is identified by using a lower gastrointestinal part identification model A and the category to which the lower gastrointestinal endoscope image belongs is output as a colon, after the auxiliary category image is eliminated, if the probability that N continuous non-similar images simultaneously belong to the colon exceeds a first preset threshold, a lower gastrointestinal part identification model B is used for carrying out secondary judgment on the lower gastrointestinal endoscope image, and when the probability that the N continuous non-similar images output as the same secondary category exceeds a second preset threshold, the lens is judged to reach the secondary category part.
2. The lower alimentary tract endoscope image recognition method according to claim 1 wherein the primary categories include ileocecal valve, colon, sigmoid colon, and rectum; the secondary classes belong to the sub-classes of the colon, including ascending, transverse and descending; the auxiliary category is a preset interference image category and is used for eliminating interference images;
the auxiliary categories comprise appendix opening, effusion, lens distance smaller than a preset value from the intestinal wall, lens shielding, intestinal cavity contraction, incomplete intestinal cavity and fuzziness.
3. The lower digestive tract endoscope image recognition method of claim 1 wherein the lower digestive tract part recognition model a employs an image classification model provided by a deep learning framework Keras application module;
and (3) training the local area in a characteristic supervision mode by using a multi-branch structure and simultaneously utilizing the local information and the global information of the lower digestive tract endoscope image in the training set through a fine-grained classification network DFL-CNN to obtain a lower digestive tract part recognition model B.
4. A lower gastrointestinal endoscope image recognition system using the lower gastrointestinal endoscope image recognition method according to any one of claims 1 to 3, comprising:
the image marking module is used for acquiring the lower digestive tract part image, marking the category to be identified and the auxiliary category for identifying the interference image, and dividing the category into a training set and a testing set; the category to be identified comprises a primary category and a secondary category, and the secondary category belongs to a subcategory of the primary category;
a model training module for training and testing a lower digestive tract part recognition model A and a lower digestive tract part recognition model B by using a training set and a test set; the lower digestive tract part identification model A is used for identifying a primary class and an auxiliary class, and the lower digestive tract part identification model B is used for identifying a secondary class;
a withdrawal operation starting judgment module, which is used for starting withdrawal operation if the trained lower gastrointestinal tract part recognition model A recognizes that the type of the current lower gastrointestinal tract endoscope image is a cecum in a first-class type and the probability that the cecum exists in the continuous N non-similar images exceeds a preset threshold value in the process of the retraction operation; n is a positive integer greater than or equal to 3;
the image real-time identification module is used for detecting an endoscope image of the alimentary tract in real time based on the lower alimentary tract part identification model A and the lower alimentary tract part identification model B in the process of endoscope withdrawal operation, eliminating images belonging to auxiliary categories, and outputting a primary category and a secondary category to which the images belong if the probability that the N continuous non-similar images are all in the same category exceeds a preset threshold value;
in the image real-time identification module, in the process of endoscope withdrawal operation, when a lower gastrointestinal endoscope image is identified by using a lower gastrointestinal part identification model A and the category to which the lower gastrointestinal endoscope image belongs is output as a colon, after an auxiliary category image is excluded, if the probability that N continuous non-similar images simultaneously belong to the colon exceeds a first preset threshold, a lower gastrointestinal part identification model B is used for carrying out secondary judgment on the lower gastrointestinal endoscope image, and when the probability that the N continuous non-similar images output as the same secondary category exceeds a second preset threshold, the lens is judged to reach the secondary category part;
in the step of judging whether the image is a non-similar image or not, the step of judging whether the image is a non-similar image comprises the following steps:
and generating a hash sequence by a mean hash algorithm and calculating a Hamming distance, and judging the image to be a non-similar image when the Hamming distance is greater than a set Hamming distance threshold value.
5. The lower gastrointestinal endoscope image recognition system of claim 4 wherein in the image labeling module, the primary categories include ileocecal valve, colon, sigmoid colon, and rectum; the secondary classes belong to the sub-classes of the colon, including ascending, transverse and descending; the auxiliary category is a preset interference image category and is used for eliminating interference images;
in the image labeling module, the auxiliary categories comprise appendix opening, effusion, lens and intestinal wall distance smaller than a preset value, lens shielding, intestinal cavity contraction, incomplete intestinal cavity and fuzziness;
in the model training module, a lower digestive tract part recognition model A adopts an image classification model provided by a deep learning framework Keras application module;
in the model training module, a multi-branch structure is used for simultaneously utilizing local information and global information of a lower digestive tract endoscope image in a training set through a fine-grained classification network DFL-CNN, and training is carried out on a local area in a feature supervision mode, so that a lower digestive tract part recognition model B is obtained.
6. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the steps of the lower gastrointestinal endoscope image recognition method according to claims 1-3.
7. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program performs the steps in the lower gastrointestinal endoscope image recognition method according to claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911327593.1A CN110991561B (en) | 2019-12-20 | 2019-12-20 | Method and system for identifying images of endoscope in lower digestive tract |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911327593.1A CN110991561B (en) | 2019-12-20 | 2019-12-20 | Method and system for identifying images of endoscope in lower digestive tract |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110991561A CN110991561A (en) | 2020-04-10 |
| CN110991561B true CN110991561B (en) | 2020-12-25 |
Family
ID=70073634
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911327593.1A Active CN110991561B (en) | 2019-12-20 | 2019-12-20 | Method and system for identifying images of endoscope in lower digestive tract |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110991561B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111612034B (en) * | 2020-04-15 | 2024-04-12 | 中国科学院上海微系统与信息技术研究所 | Method and device for determining object recognition model, electronic equipment and storage medium |
| CN111739007B (en) * | 2020-06-22 | 2024-01-26 | 中南民族大学 | Endoscopic image recognition method, device, storage medium and apparatus |
| CN113139937B (en) * | 2021-04-13 | 2022-11-01 | 上海市第一人民医院 | Digestive tract endoscope video image identification method based on deep learning |
| CN115511770A (en) * | 2021-06-07 | 2022-12-23 | 深圳开立生物医疗科技股份有限公司 | Endoscope image processing method, endoscope image processing device, electronic device and readable storage medium |
| CN113793335B (en) * | 2021-11-16 | 2022-02-08 | 武汉大学 | Method and device for identifying alimentary tract tumor infiltration layer, computer equipment and medium |
| CN114782388A (en) * | 2022-04-29 | 2022-07-22 | 小荷医疗器械(海南)有限公司 | Endoscope advance and retreat time determining method and device based on image recognition |
| CN116681681B (en) * | 2023-06-13 | 2024-04-02 | 富士胶片（中国）投资有限公司 | Endoscopic image processing method, device, user equipment and medium |
| CN116740475B (en) * | 2023-08-15 | 2023-10-17 | 苏州凌影云诺医疗科技有限公司 | Digestive tract image recognition method and system based on state classification |
| CN117392449A (en) * | 2023-10-24 | 2024-01-12 | 青岛美迪康数字工程有限公司 | Enteroscopy part identification method, device and equipment based on endoscopic image features |
| CN117372787B (en) * | 2023-12-05 | 2024-02-20 | 同方赛威讯信息技术有限公司 | Image multi-category identification method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722735A (en) * | 2012-05-24 | 2012-10-10 | 西南交通大学 | Endoscopic image lesion detection method based on fusion of global and local features |
| CN108615037A (en) * | 2018-05-31 | 2018-10-02 | 武汉大学人民医院（湖北省人民医院） | Controllable capsule endoscopy operation real-time auxiliary system based on deep learning and operating method |
| CN109344287A (en) * | 2018-09-05 | 2019-02-15 | 腾讯科技（深圳）有限公司 | A kind of information recommendation method and relevant device |
| CN109447985A (en) * | 2018-11-16 | 2019-03-08 | 青岛美迪康数字工程有限公司 | Colonoscopic images analysis method, device and readable storage medium storing program for executing |
| CN109919212A (en) * | 2019-02-26 | 2019-06-21 | 中山大学肿瘤防治中心 | The multi-dimension testing method and device of tumour in digestive endoscope image |
| CN110097080A (en) * | 2019-03-29 | 2019-08-06 | 广州思德医疗科技有限公司 | A kind of construction method and device of tag along sort |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9743824B2 (en) * | 2013-09-04 | 2017-08-29 | Siemens Medical Solutions Usa, Inc. | Accurate and efficient polyp detection in wireless capsule endoscopy images |
| CN108968951B (en) * | 2018-08-15 | 2021-06-22 | 武汉中旗生物医疗电子有限公司 | Electrocardiogram detection method, device and system |
| CN110504029B (en) * | 2019-08-29 | 2022-08-19 | 腾讯医疗健康(深圳)有限公司 | Medical image processing method, medical image identification method and medical image identification device |
| CN110363768B (en) * | 2019-08-30 | 2021-08-17 | 重庆大学附属肿瘤医院 | Early cancer focus range prediction auxiliary system based on deep learning |
-
2019
- 2019-12-20 CN CN201911327593.1A patent/CN110991561B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722735A (en) * | 2012-05-24 | 2012-10-10 | 西南交通大学 | Endoscopic image lesion detection method based on fusion of global and local features |
| CN108615037A (en) * | 2018-05-31 | 2018-10-02 | 武汉大学人民医院（湖北省人民医院） | Controllable capsule endoscopy operation real-time auxiliary system based on deep learning and operating method |
| CN109344287A (en) * | 2018-09-05 | 2019-02-15 | 腾讯科技（深圳）有限公司 | A kind of information recommendation method and relevant device |
| CN109447985A (en) * | 2018-11-16 | 2019-03-08 | 青岛美迪康数字工程有限公司 | Colonoscopic images analysis method, device and readable storage medium storing program for executing |
| CN109919212A (en) * | 2019-02-26 | 2019-06-21 | 中山大学肿瘤防治中心 | The multi-dimension testing method and device of tumour in digestive endoscope image |
| CN110097080A (en) * | 2019-03-29 | 2019-08-06 | 广州思德医疗科技有限公司 | A kind of construction method and device of tag along sort |
Non-Patent Citations (2)
| Title |
|---|
| Review on the Applications of Deep Learning in the Analysis of Gastrointestinal Endoscopy Images;WENJU DU et al.;《IEEE Access》;20191010;第7卷;全文 * |
| 医用内窥镜图像计算机辅助诊断研究进展;敬雪平等;《Computer Engineering and Applications计算机工程与应用》;20180201;第54卷(第3期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110991561A (en) | 2020-04-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110991561B (en) | Method and system for identifying images of endoscope in lower digestive tract | |
| CN111048170B (en) | Digestive endoscopy structured diagnosis report generation method and system based on image recognition | |
| US10860930B2 (en) | Learning method, image recognition device, and computer-readable storage medium | |
| CN111128396B (en) | Digestive tract disease auxiliary diagnosis system based on deep learning | |
| US20190311475A1 (en) | Image diagnosis learning device, image diagnosis device, image diagnosis method, and recording medium for storing program | |
| US9171477B2 (en) | Method and system for recognizing and assessing surgical procedures from video | |
| Quellec et al. | Real-time task recognition in cataract surgery videos using adaptive spatiotemporal polynomials | |
| EP2557539B1 (en) | Image processing apparatus, image processing method, and image processing program | |
| JP6639523B2 (en) | Automatic learning image sorting device, automatic learning image sorting method, and automatic learning image sorting program | |
| Nguyen et al. | Colorectal segmentation using multiple encoder-decoder network in colonoscopy images | |
| JPWO2016185617A1 (en) | Image processing apparatus, image processing method, and image processing program | |
| US20120177259A1 (en) | Image processing device, image processing method, computer-readable recording device | |
| KR102176139B1 (en) | Apparatus and method for segmenting images using consecutive deep encoder-decoder network | |
| KR102332088B1 (en) | Apparatus and method for polyp segmentation in colonoscopy images through polyp boundary aware using detailed upsampling encoder-decoder networks | |
| CN111179252A (en) | Cloud platform-based digestive tract disease focus auxiliary identification and positive feedback system | |
| CN111723852B (en) | Robust training method for target detection network | |
| CN111444817B (en) | Character image recognition method and device, electronic equipment and storage medium | |
| CN111798408A (en) | Endoscope interference image detection and grading system and method | |
| Hegenbart et al. | Systematic assessment of performance prediction techniques in medical image classification a case study on celiac disease | |
| Ratheesh et al. | Advanced algorithm for polyp detection using depth segmentation in colon endoscopy | |
| CN111612749B (en) | Focus detection method and device based on lung image | |
| Li et al. | Detection of endoscope withdrawal time in colonoscopy videos | |
| CN114693912A (en) | Endoscope inspection system with eyeball tracking function, storage medium and equipment | |
| CN111126474B (en) | Confocal laser micro-endoscope digestive tract image identification method and system | |
| Zhou et al. | Wireless capsule endoscopy video automatic segmentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Li Yanqing Inventor after: Li Guangchao Inventor after: Feng Jian Inventor after: Zuo Xiuli Inventor after: Yang Xiaoyun Inventor after: Shao Xuejun Inventor after: Lai Yonghang Inventor after: Xin Wei Inventor after: Li Zhen Inventor before: Li Yanqing Inventor before: Li Guangchao Inventor before: Feng Jian Inventor before: Zuo Xiuli Inventor before: Yang Xiaoyun Inventor before: Shao Xuejun Inventor before: Lai Yonghang Inventor before: Xin Wei Inventor before: Li Zhen |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |