CN113887677B - Classification method, apparatus, apparatus and medium for images of intrapapillary capillaries - Google Patents

Abstract

The embodiment of the invention discloses a method, a device, equipment and a medium for classifying capillary vessel images in epithelial papillae. The method comprises the following steps: inputting an endoscopic image of capillary vessels in the epithelial papilla into a pre-trained neural network model to obtain an effective area of the endoscopic image; acquiring a target blood vessel region in an endoscope image from the effective region according to a connected domain algorithm; and acquiring the characteristic diameter, the characteristic distortion quantitative value, the characteristic area ratio, the centroid eccentricity and the whole image density of the target blood vessel in the target blood vessel region, and inputting the characteristic diameter, the characteristic distortion quantitative value, the characteristic area ratio, the centroid eccentricity and the whole image density into a pre-trained classification model to obtain a classification result of the endoscope image. The method extracts the target blood vessel in the endoscopic image based on the neural network technology and acquires five indexes for evaluating the target blood vessel, and further realizes accurate judgment of the esophageal cancer infiltration depth according to the five indexes, so that the efficiency and the accuracy of the judgment of the esophageal cancer infiltration depth are improved.

Description

Classification method, apparatus, apparatus and medium for images of intrapapillary capillaries

technical field

The invention relates to the technical field of medical assistance, in particular to a method, device, equipment and medium for classifying capillary images in epithelial papillae.

Background technique

Epithelial papillary capillary loops (IntraaPillary Capillary Loops, IPCL) are microvessels formed perpendicular to the branching vessels under the squamous epithelium. Among them, the degree of morphological changes of capillary loops in the epithelial papilla is a key indicator for the diagnosis of the depth of esophageal cancer invasion under endoscopy. The destruction of IPCL structure increases with tumor infiltration, so identifying changes in IPCL is important in evaluating esophagus-pharyngeal lesions. plays an important role in. When squamous cell carcinoma invades the mucosa propria, irregular microvessels form in the form of loops with enlarged diameters; when the tumor invades the muscularis mucosa or superficially invades the submucosa, the abnormal microvessel loops disappear, showing obvious The tumor deeply invades the submucosa, and the structure of IPCL is usually completely destroyed, and the diameter of the blood vessel is at least 3 times that of the superficial infiltrating blood vessel, accompanied by abnormal tumor angiogenesis with enlarged diameter and various shapes. . Therefore, IPCL vessels of esophageal squamous cell carcinoma exhibit abnormal degrees of irregularity such as dilation, tortuosity, caliber changes, and abnormal shapes.

At present, the nature of esophageal lesions is usually determined by using narrow-band imaging (NBI) technology to observe the capillary loops in the epithelial papilla. Specifically, the blood vessels on the mucosal surface are stained brown to form a high contrast with the background tissue to examine the capillaries in detail. Changes, and then propose classification criteria according to the abnormal degree of IPCL to predict the depth of invasion of esophageal cancer. However, due to the difficulty of homogenization of the evaluation criteria for the degree of vascular atypia under dye magnification and the great influence of the assessor’s subjectivity, it is difficult to achieve homogenization and high-precision evaluation of the degree of esophageal IPCL abnormality under dye magnification endoscopy. The technical problems of poor accuracy and low efficiency when judging the depth of esophageal cancer invasion.

SUMMARY OF THE INVENTION

The embodiments of the present invention provide a method, device, equipment and medium for classifying intrapapillary capillary images, which solve the technical problem that the depth of esophageal cancer invasion cannot be accurately identified under chromoendoscopy in the prior art.

In a first aspect, an embodiment of the present invention provides a method for classifying intrapapillary capillary images, which includes:

Input the endoscopic image of the capillaries in the epithelial papilla into the pre-trained neural network model to obtain the effective area of the endoscopic image;

Obtain the target blood vessel region in the endoscopic image from the effective region according to a connected domain algorithm;

obtaining the characteristic diameter, characteristic distortion quantification value, characteristic area ratio, centroid eccentricity, and whole image density of the target blood vessel in the target blood vessel region;

Inputting the characteristic diameter, the characteristic distortion quantification value, the characteristic area ratio, the centroid eccentricity and the whole image density into a pre-trained classification model to obtain the endoscope Image classification results.

In a second aspect, an embodiment of the present invention provides an apparatus for classifying intrapapillary capillary images, which includes:

a first input unit, configured to input the endoscopic image of the capillaries in the epithelial papilla into a pre-trained neural network model to obtain an effective area of the endoscopic image;

a first acquiring unit, configured to acquire a target blood vessel region in the endoscopic image from the effective region according to a connected domain algorithm;

a second acquiring unit, configured to acquire the characteristic diameter, characteristic distortion quantification value, characteristic area ratio, centroid eccentricity, and whole image density of the target blood vessel in the target blood vessel region;

The second input unit is used for inputting the characteristic diameter, the characteristic distortion quantization value, the characteristic area ratio, the centroid eccentricity and the whole image density to the pre-trained classification model , the classification result of the endoscopic image is obtained.

In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes The computer program implements the method for classifying epithelial intrapapillary capillary images according to the first aspect.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when executed by a processor, the computer program causes the processor to execute the above-mentioned first step. In one aspect, the method for classifying intrapapillary capillary images of the epithelium.

Embodiments of the present invention provide a method, device, equipment and medium for classifying intrapapillary capillary images. The method uses neural network technology to extract target blood vessels from endoscopic images to obtain characteristic diameters, characteristic diameters, and characteristics of target blood vessels for evaluation. Five indicators of distortion quantification value, characteristic area ratio, centroid eccentricity, and whole image density, and then classify the target blood vessels according to the five indicators, and determine the intrapapillary capillaries according to the classification results of the target blood vessels The current shape of the loop can be used to accurately judge the depth of esophageal cancer invasion and improve the efficiency and accuracy of esophageal cancer invasion depth judgment.

Description of drawings

In order to explain the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention, which are of great significance to the art For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic flowchart of a method for classifying intrapapillary capillary images according to an embodiment of the present invention;

2 is a schematic diagram of a sub-flow of a method for classifying intrapapillary capillary images according to an embodiment of the present invention;

3 is another schematic flowchart of a method for classifying intrapapillary capillary images according to an embodiment of the present invention;

4 is another schematic flowchart of a method for classifying intrapapillary capillary images according to an embodiment of the present invention;

5 is another schematic flowchart of a method for classifying intrapapillary capillary images according to an embodiment of the present invention;

6 is another schematic flowchart of a method for classifying intrapapillary capillary images according to an embodiment of the present invention;

7 is another schematic flowchart of a method for classifying intrapapillary capillary images according to an embodiment of the present invention;

8 is a schematic block diagram of an apparatus for classifying intrapapillary capillary images according to an embodiment of the present invention;

9 is a schematic block diagram of a computer device provided by an embodiment of the present invention;

10 is a specific application flowchart provided by an embodiment of the present invention;

11 is a flowchart of obtaining an effective area in an endoscopic image according to an embodiment of the present invention;

12 is a coordinate diagram for judging whether each pixel of a target blood vessel is located in a target blood vessel area by using an area method according to an embodiment of the present invention;

13 is an effect diagram of acquiring a target blood vessel in an endoscopic image according to an embodiment of the present invention;

14 is a schematic diagram of measuring the diameter of a target blood vessel in an endoscopic image by using Halcon according to an embodiment of the present invention;

15 is a schematic diagram of the distortion quantification of a target blood vessel in an endoscopic image according to an embodiment of the present invention;

FIG. 16 is a schematic diagram of a geometric center of an endoscopic image and an equivalent centroid of an effective area according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It is to be understood that, when used in this specification and the appended claims, the terms "comprising" and "comprising" indicate the presence of the described features, integers, steps, operations, elements and/or components, but do not exclude one or The presence or addition of a number of other features, integers, steps, operations, elements, components, and/or sets thereof.

It is also to be understood that the terminology used in this specification of the present invention is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise.

It should also be further understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items .

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of a method for classifying intrapapillary capillary images according to an embodiment of the present invention. The method for classifying epithelial capillary images in the embodiment of the present invention is applied to a terminal device, and the method is executed by application software installed in the terminal device. The terminal device is a terminal device capable of accessing the Internet, such as a desktop computer, a notebook computer, a tablet computer, or a mobile phone and other terminal devices.

The classification method of the epithelial intrapapillary capillary images will be described in detail below.

As shown in FIG. 1, the method includes the following steps S110-S140.

S110. Input the endoscopic image of the capillaries in the epithelial papilla into a pre-trained neural network model to obtain an effective area of the endoscopic image.

Wherein, the endoscopic image is obtained by NBI technology and there are capillaries within the epithelial papillary in the endoscopic image, and the neural network model is pre-trained and is used for extracting from the endoscopic image the intramapillary capillaries containing the epithelium The characteristic information of capillaries, that is, the effective area.

Specifically, the neural network model is constructed by a neural network with an image segmentation function, and the neural network can be either a Unet neural network or a Mask-RCNN neural network.

It should be noted that, in the process of constructing the neural network model, the neural network specifically adopted by the neural network model can be selected according to the actual situation, which is not specifically limited in the present invention.

In other inventive embodiments, as shown in FIG. 2 , before step S110 , the steps further include: S210 and S220 .

S210, input the training sample into the neural network model to obtain the mean square error loss of the neural network model;

S220. Update the network parameters of the neural network model according to the mean square error loss.

In this embodiment, as shown in FIG. 11 , FIG. 11 is a flowchart of acquiring an effective area in an endoscopic image according to an embodiment of the present invention. The neural network model is constructed by the Unet neural network, wherein the Unet neural network is obtained by improving the fully convolutional neural network, and the Unet neural network is achieved by strengthening the connection between layers, plus upsampling and downconvolution. Sufficient extraction of features to achieve accurate segmentation with fewer training samples. The neural network model adopts the Unet neural network, which can effectively extract the feature information containing the capillaries in the epithelial papilla from the endoscopic image.

Specifically, the training sample is also obtained by NBI technology, and the training sample contains characteristic information of capillaries in the epithelial papilla. After the training sample is input into the neural network model, it is output through the neural network model. The network parameters of the neural network model can be updated, thereby realizing the training of the neural network model.

The function that generates the mean squared error loss is:

，真实值为

。 Among them, m is the number of input training samples, and the predicted value of the neural network model is

, the real value is

.

In other inventive embodiments, as shown in FIG. 3 , before step S210 , the steps further include: S310 and S320 .

S310, performing video decoding on the endoscopic video of the capillaries in the epithelial papilla to obtain a decoded video image;

S320. Label the video decoded image to obtain the training sample.

In this embodiment, the training sample is obtained from an esophageal endoscopic video captured by NBI technology, and after acquiring the video of the endoscopic image, the terminal device performs decoding processing on the video to obtain the video decoded image, Then, the video decoded image is marked to outline the outline of blood vessels in the video decoded image to form a training sample for training the neural network model, so that after the training sample is input into the neural network model , the mean square error loss of the neural network model can be obtained, so as to realize the training of the neural network model.

S120. Acquire a target blood vessel region in the endoscopic image from the effective region according to a connected domain algorithm.

Specifically, the connected domain algorithm is also called a connected domain labeling algorithm, that is, the target blood vessel in the endoscopic image is marked in the effective area, and the connected domain algorithm is used to obtain the target blood vessel from the effective area. After the target blood vessel area, five indicators for judging the depth of esophageal cancer infiltration in the endoscopic image can be obtained, and then a linear weighted calculation can be performed according to the five indicators, and the epithelial papilla in the endoscopic image can be classified. The current shape of the inner capillary loop, thereby realizing the judgment of the infiltration depth of the esophageal cancer in the endoscopic image.

In other inventive embodiments, as shown in FIG. 4 , step S120 includes sub-steps S121 and S122.

S121, traversing the effective area to obtain the connected domain area and the minimum circumscribed horizontal rectangle of the target blood vessel;

S122. Determine the target blood vessel region according to the area of the connected domain and the minimum circumscribed horizontal rectangle.

Specifically, by traversing the effective area, the connected domain area and the minimum circumscribed horizontal rectangle of all blood vessels in the effective area are obtained, and then the connected domain area and the minimum circumscribed horizontal rectangle of the target blood vessel are determined therefrom. Wherein, when determining the connected domain area of the target blood vessel and the minimum circumscribed horizontal rectangle, the blood vessel where the largest connected domain except the background area is located among the connected domain areas of all blood vessels is used as the target blood vessel, and the minimum value of the blood vessel is The circumscribed horizontal rectangle is the smallest circumscribed horizontal rectangle of the target blood vessel.

， 所述目标血管的最小外接水平矩形内某一像素点坐标为

，若该像素点在所述目标 血管的连通域内部，则其与连通域所有相邻顶点组成的三角形面积和为多边形面积，则需 满足如下等式： In addition, after the position of the target blood vessel is determined, the target blood vessel region needs to be further determined. Specifically, by traversing all the pixels in the minimum circumscribed horizontal rectangle of the target blood vessel, and then using the area method as shown in Figure 12 to determine whether each pixel is in the connected domain of the target blood vessel, and then determining whether the pixel is in the connected domain of the target blood vessel, as shown in Figure 13 target vessel area shown. The specific process is: assuming that the vertex coordinates of the connected domain of the target blood vessel are

, the coordinates of a certain pixel in the minimum circumscribed horizontal rectangle of the target blood vessel are

, if the pixel is inside the connected domain of the target blood vessel, the sum of the triangle area formed by it and all the adjacent vertices of the connected domain is the polygonal area, and the following equation must be satisfied:

If the interior of the minimum circumscribed horizontal rectangle of the target blood vessel does not satisfy the above equation, the pixel value thereof is set as the background pixel of the endoscopic image.

S130. Obtain the characteristic diameter, characteristic distortion quantification value, characteristic area ratio, centroid eccentricity, and whole image density of the target blood vessel in the target blood vessel region.

Specifically, the characteristic diameter, the characteristic distortion quantification value, the characteristic area ratio, the centroid eccentricity, and the entire image density are all used to judge the endoscopic image. The index of the current shape of the capillary loop in the epithelial papillary, through these five indicators, the current shape of the capillary loop in the epithelial papillary can be identified in the endoscopic image, and then the esophageal cancer infiltration in the endoscopic image can be realized. in-depth judgment.

In other inventive embodiments, as shown in FIG. 5 , step S130 includes sub-steps S131 , S132 , S133 , S134 and S135 .

S131. Generate the characteristic diameter according to the largest class average diameter and the smallest class average diameter of the target blood vessel.

Specifically, the maximum class average diameter is the average value of diameters at the target blood vessel at the position belonging to the largest class, and the minimum class average diameter is the average value of the diameters at the target blood vessel at the position belonging to the smallest class. The characteristic diameter can be calculated from the largest class average diameter and the smallest class average diameter.

，然后从中获取所有最大类直径以及所有最小类直径并计算出所述最大类平均直径和所述最小类平均直径，最后根据所述最大类平均直径、所述最小类平均直径便可计算出所述特征性直径。具体计算公式如下：In this embodiment, as shown in FIG. 14 , FIG. 14 is a schematic diagram of using Halcon to measure the diameter of a target blood vessel in an endoscopic image according to an embodiment of the present invention. Measure diameters at multiple locations on the target vessel by invoking the diameter measurement toolkit in Halcon

, then obtain all the largest class diameters and all smallest class diameters and calculate the largest class average diameter and the smallest class average diameter, and finally calculate the largest class average diameter and the smallest class average diameter according to the largest class average diameter and the smallest class average diameter. Describe the characteristic diameter. The specific calculation formula is as follows:

为最大类平均直径，

为最小类平均直径，D为特征性直径。in,

is the largest class mean diameter,

is the smallest class mean diameter, and D is the characteristic diameter.

In addition, because Halcon is affected by the color difference between the target blood vessel and the background color, and the curve fitting ability, it is necessary to divide the blood vessels with complex shapes into several segments, and then measure the blood vessel diameter for each segment. Diameters at multiple sites on the target vessel.

S132 , respectively acquiring multiple quantification values of tortuosity and multiple area ratios of the target blood vessel.

Specifically, the plurality of distortion quantification values and the plurality of area ratios of the target blood vessel are respectively the distortion quantification values and area ratios at different parts of the target blood vessel, and the distortion quantification is used to represent The tortuosity of the blood vessel, the area fraction is used to characterize the area fraction of the blood vessel in its smallest circumscribed horizontal rectangle.

S133. Obtain the characteristic distortion quantization value and the characteristic area proportion from the plurality of distortion quantization values and the plurality of area proportions.

，然后采用几何平 均数计算公式来得到所述特征性面积占比量。 In this embodiment, after obtaining the plurality of quantification values of tortuosity and the plurality of area ratios, the quantified value of characteristic tortuosity can be calculated by using a preset quantification formula of blood vessel tortuosity, At the same time, the area proportion of each part in the target blood vessel is calculated through the preset area proportion formula

, and then use the geometric mean calculation formula to obtain the characteristic area proportion.

FIG. 15 is a schematic diagram of the distortion quantification of a target blood vessel in an endoscopic image according to an embodiment of the present invention. As shown in Figure 15, the principle of quantifying the distortion of the target blood vessel is as follows: parallel to the width and height of the minimum circumscribed rectangle of the target blood vessel, a new line intersects the target blood vessel. More will be explained. According to this principle, the more complex the target blood vessel is, the greater the density of points falling on the line will be when the pixels on the target blood vessel are projected to the width and height of the minimum circumscribed rectangle of the target blood vessel. Among them, the quantification formula of vascular tortuosity is:

为血管扭曲性量化值，n为微血管管壁内外两侧像素点总个数，

和

分 布别为血管的最小外接矩形的长和宽。 in,

is the quantification value of the tortuosity of the blood vessel, n is the total number of pixels on the inner and outer sides of the microvascular wall,

and

The distributions are the length and width of the smallest circumscribed rectangle of the blood vessel, respectively.

The area ratio formula is:

为血管中心线上每个像素点处的血管直径，

和

分布别为血管的最 小外接矩形的长和宽。 in,

is the blood vessel diameter at each pixel on the blood vessel centerline,

and

The distributions are the length and width of the smallest circumscribed rectangle of the blood vessel, respectively.

The formula for calculating the geometric mean is:

In addition, before calculating the characteristic tortuosity quantification value and the characteristic area ratio by using the above-mentioned blood vessel tortuosity quantification formula and area ratio formula, respectively, it is necessary to quantify the plurality of tortuosity quantification values and all the tortuosity quantification values. The multiple area proportions are screened, so that the finally generated characteristic distortion quantification value and the characteristic area proportions are more accurate, thereby improving the accuracy of judging the depth of esophageal cancer invasion. Wherein, the screening process can be performed with reference to step S131b, which is not specifically limited here.

S134. Generate the centroid eccentricity according to the equivalent centroid of the effective area and the geometric center of the endoscopic image.

In this embodiment, the equivalent centroid is that the mass of the blood vessel particle in the effective region is equal to the total mass of the particle system, and the geometric center is the exact center position of the endoscopic image. As shown in FIG. 16 , FIG. 16 is a schematic diagram of a geometric center of an endoscopic image and an equivalent centroid of an effective area according to an embodiment of the present invention. After the effective area is acquired, theoretically, if the capillaries in the endoscopic image are not abnormal, their distribution will be relatively uniform in the entire field of view, and the equivalent centroids of all blood vessels should be within the geometrical geometry of the entire field of view. center, or close to the geometric center; and when the capillaries in the epithelial papilla are abnormal in the endoscopic image, the blood vessels in the field of view will disappear, or be squeezed by the lesion to change the distribution of blood vessels, then all the effective areas in the There is a high probability that the equivalent centroid of the blood vessel will deviate far from the geometric center. After the equivalent centroid of the effective area and the geometric center of the endoscopic image are obtained, the centroid eccentricity of the target blood vessel can be calculated by using a preset centroid eccentricity calculation formula. Among them, the calculation formula of centroid eccentricity is:

为有效区域的等效质心，

为内镜图像中 每根血管的质心，

为内镜图像中每根血管面积，

和

均可在连通域的基础 上获得，

为内镜图像的几何中心，

。 Among them, e is the eccentricity of the center of mass,

is the equivalent centroid of the effective area,

is the centroid of each vessel in the endoscopic image,

is the area of each vessel in the endoscopic image,

and

can be obtained on the basis of connected domains,

is the geometric center of the endoscopic image,

.

S135. Obtain the whole image density according to a preset blood vessel whole image density formula.

In this embodiment, the blood vessel whole image density formula is a density formula pre-built according to the structure of the blood vessel in the endoscopic image, and the blood vessel whole image density formula is:

为血管整图中每根血管面积，可通过连通域获得，W、H分别为内镜图 像的宽和高。 in,

is the area of each blood vessel in the whole image of blood vessels, which can be obtained by the connected domain, where W and H are the width and height of the endoscopic image, respectively.

In other inventive embodiments, as shown in FIG. 6 , before step S131 , the steps further include: S131a , S131b , S131c and S131d .

S131a, obtaining multiple diameters of each blood vessel in the target blood vessel;

S131b, screening the multiple diameters of each branch to obtain the diameter after screening;

S131c, according to the K-means algorithm, the diameters after the screening are clustered to obtain the clustering result of each diameter after the screening;

S131d. Generate the largest class average diameter and the smallest class average diameter according to the clustering result.

Specifically, the target blood vessel in the endoscopic image is composed of a main blood vessel and a plurality of branch blood vessels. By measuring each blood vessel in the target blood vessel multiple times, each blood vessel in the target blood vessel can be obtained. Multiple diameters of blood vessels, and then screen out the normal values and use the K-means algorithm to cluster them into three categories, then the classification of each diameter after screening can be completed. The average diameter of each class is calculated, from which the largest class average diameter and the smallest class average diameter can be obtained.

Among them, the K-means algorithm is also called the fast clustering method. In the nearest cluster, the data is divided into K clusters to complete a division, and the new cluster formed is not the optimal division. Therefore, in the new cluster generated, the center point of each cluster is recalculated, and then the division is performed again until The result of each division remains the same.

In other inventive embodiments, as shown in FIG. 7 , step S131b includes sub-steps S131b1 and S131b2.

S131b1, obtaining the average diameter of each blood vessel and the variance of the diameter of each blood vessel;

S131b2 , screening multiple diameters of each blood vessel according to the average diameter of each blood vessel and the variance of the diameters of each blood vessel to obtain the screened diameter.

In this embodiment, the average diameter of each blood vessel in the target blood vessel and the variance of each blood vessel diameter are calculated, and then the screening of multiple diameters of each blood vessel can be completed according to the average diameter and variance.

The formula for calculating the average diameter is:

The formula for calculating the variance of each vessel diameter is:

或

时，剔除直径为

所述的数值，便可完成对所述目标血管中每根血管的多个直径的筛选。when

or

, the removal diameter is

With the value, the screening of multiple diameters of each blood vessel in the target blood vessel can be completed.

S140. Input the characteristic diameter, the characteristic distortion quantization value, the characteristic area ratio, the centroid eccentricity, and the whole image density into the pre-trained classification model to complete the classification of all The classification of target blood vessels in the endoscopic image is performed, and then the classification result of the endoscopic image is obtained.

，其中，

，

，

，

，

分别为 所述特征性直径、所述特征性扭曲性量化值、所述特征性面积占比量、所述质心偏心距以及 所述整图密度，ζ为血管异常程度系数，λ₁、λ₂、λ₃、λ₄、λ₅分别为各自的权重。其中，λ₁、λ₂、λ₃、 λ₄、λ₅可采用网格搜索法、贪婪搜索法等确定，在此不做具体限制。 Wherein, the classification model is pre-trained and used to classify the target blood vessels in the endoscopic image, and the classification model can be Logistic regression, support vector machine, extreme gradient boosting, decision tree, random forest, BP Any one of the neural networks is constructed, and the calculation formula of its linear weighted fitting is:

,in,

,

,

,

,

are the characteristic diameter, the characteristic distortion quantification value, the characteristic area ratio, the centroid eccentricity, and the whole image density, respectively, ζ is the degree coefficient of abnormality of blood vessels, λ ₁ , λ ₂ , λ ₃ , λ ₄ , and λ ₅ are their respective weights. Among them, λ ₁ , λ ₂ , λ ₃ , λ ₄ , and λ ₅ can be determined by grid search method, greedy search method, etc., and no specific limitation is made here.

Specifically, after the five index values of the target blood vessel in the endoscopic image are input into the classification model, the abnormality degree coefficient of the target blood vessel can be obtained, and then according to the position of the abnormality degree coefficient of the target blood vessel The abnormal level of the target blood vessel is determined by the numerical range, and then the classification of the endoscopic image can be completed.

For example, when the abnormality degree coefficient of the target blood vessel is less than or equal to 0.35, it can be determined that the target blood vessel in the endoscopic image is normal; when the abnormality degree coefficient of the target blood vessel is between 0.35 and 0.89, it can be determined that the target blood vessel is normal. The target blood vessel in the endoscopic image is generally abnormal; when the abnormality degree coefficient of the target blood vessel is between 0.89 and 1.25, it can be determined that the target blood vessel in the endoscopic image is relatively abnormal; When the abnormality degree coefficient is greater than 1.25, it can be determined that the target blood vessel in the endoscopic image is very abnormal.

In the method for classifying intrapapillary capillary images provided by the embodiment of the present invention, the effective area of the endoscopic image is obtained by inputting the endoscopic image of the intrapapillary capillary into a pre-trained neural network model; Obtain the target blood vessel area in the endoscopic image from the effective area according to the connected domain algorithm; obtain the characteristic diameter, characteristic distortion quantification value, characteristic area ratio, characteristic diameter of the target blood vessel in the target blood vessel area, The centroid eccentricity and the whole image density; the characteristic diameter, the characteristic distortion quantification value, the characteristic area ratio, the centroid eccentricity and the whole image density are input to the pre-trained classification In the model, the classification result of the endoscopic image is obtained. The invention realizes accurate esophageal cancer infiltration depth judgment through quantitative evaluation of the abnormal degree of capillaries in the esophageal epithelial papilla, improves the efficiency and accuracy of esophageal cancer infiltration depth judgment, and provides a reliable basis for clinical treatment decisions of esophageal cancer patients.

For a specific application of the method for classifying intrapapillary capillary images provided by the embodiment of the present invention, reference may be made to FIG. 10 , and FIG. 10 is a flowchart of a specific application provided by the embodiment of the present invention. As can be seen from FIG. 10 , the esophagus stained enlarged image of the capillaries in the esophageal epithelial papilla is obtained by electronic chromoendoscopy, that is, the endoscopic image, and then blood vessel segmentation processing will be performed in the endoscopic image to obtain blood vessels The segmentation map, that is, the effective area of the endoscopic image, and then extract a single blood vessel from the effective area, that is, the target blood vessel, and obtain the quantification of the whole image distribution of the blood vessel and the quantification of the whole image density of the blood vessel from the blood vessel segmentation map, that is, the The two indicators of the target blood vessel’s centroid eccentricity and the density of the whole image are described, and then the target blood vessel is quantified by blood vessel diameter, blood vessel distortion, and blood vessel area ratio, and the singular values that do not meet the requirements are eliminated to obtain the target. The characteristic diameter of the blood vessel, the characteristic distortion quantification value, and the characteristic area ratio. Finally, the five indicators for evaluating the target blood vessel are fitted and classified by the machine learning method to obtain the classification of the intrapapillary capillary images of the epithelium. The abnormality of the capillaries in the epithelial papilla can be judged.

An embodiment of the present invention further provides an apparatus 100 for classifying intra-papillary capillary images, which is used to perform any of the foregoing methods for classifying intra-papillary capillary images.

Specifically, please refer to FIG. 8 . FIG. 8 is a schematic block diagram of an apparatus 100 for classifying capillary images in epithelial papilla according to an embodiment of the present invention.

As shown in FIG. 8 , the apparatus 100 for classifying intrapapillary capillary images includes: a first input unit 110 , a first acquisition unit 120 , a second acquisition unit 130 and a second input unit 140 .

The first input unit 110 is configured to input the endoscopic image of the capillaries in the epithelial papilla into the pre-trained neural network model to obtain the effective area of the endoscopic image.

In another embodiment, the apparatus 100 for classifying intrapapillary capillary images further includes: a third input unit and an update unit.

The third input unit is used for inputting the training samples into the neural network model to obtain the mean square error loss of the neural network model; the updating unit is used for updating the average square error loss of the neural network model according to the mean square error loss. Network parameters.

In another embodiment, the apparatus 100 for classifying intrapapillary capillary images further includes: a decoding unit and a labeling unit.

The decoding unit is used for performing video decoding on the endoscopic video of the capillaries in the epithelial papilla to obtain a decoded video image; the labeling unit is used for labeling the decoded video image to obtain the training sample.

The first acquiring unit 120 is configured to acquire the target blood vessel region in the endoscopic image from the effective region according to a connected domain algorithm.

In another embodiment, the first obtaining unit 120 includes: a traversing unit and a determining unit.

a traversing unit, configured to traverse the effective area to obtain the connected domain area and the minimum circumscribed horizontal rectangle of the target blood vessel; a determination unit, configured to determine the target blood vessel area according to the connected domain area and the minimum circumscribed horizontal rectangle .

The second acquiring unit 130 is configured to acquire the characteristic diameter, characteristic distortion quantification value, characteristic area ratio, centroid eccentricity, and integral image density of the target blood vessel in the target blood vessel region.

In another embodiment, the second obtaining unit 130 includes: a first generating unit, a third obtaining unit, a fourth obtaining unit, a second generating unit, and a fifth obtaining unit.

The first generating unit is configured to generate the characteristic diameter according to the largest class average diameter and the smallest class average diameter of the target blood vessel; the third acquiring unit is configured to separately obtain a plurality of tortuosity quantification values of the target blood vessel and A plurality of area proportions; a fourth acquisition unit, configured to obtain the characteristic distortion quantization value and the characteristic area proportion from the plurality of distortion quantization values and the plurality of area proportions ratio; a second generating unit for generating the centroid eccentricity according to the equivalent centroid of the effective area and the geometric center of the endoscopic image; a fifth acquiring unit for generating the eccentricity according to a preset blood vessel overall image density The formula obtains the whole image density.

In another embodiment, the apparatus 100 for classifying intrapapillary capillary images further includes: a sixth acquiring unit, a first screening unit, a clustering unit and a third generating unit.

a sixth obtaining unit, used to obtain multiple diameters of each blood vessel in the target blood vessel; a first screening unit, used to screen the multiple diameters of each branch to obtain the diameter after screening; a clustering unit, For clustering the diameters after the screening according to the K-means algorithm, to obtain the clustering result of each diameter after screening; the third generating unit, for generating the maximum class average diameter and The smallest class mean diameter.

In another embodiment, the first screening unit includes: a seventh obtaining unit and a second screening unit.

The seventh obtaining unit is used to obtain the average diameter of each blood vessel and the variance of the diameter of each blood vessel; the second screening unit is used to obtain the average diameter of each blood vessel and the variance of each blood vessel according to the Variance of diameters Screen the diameters of each blood vessel to obtain the screened diameters.

The second input unit 140 is configured to input the characteristic diameter, the characteristic distortion quantization value, the characteristic area ratio, the centroid eccentricity and the whole image density into the pre-trained classification In the model, the classification result of the endoscopic image is obtained.

The apparatus 100 for classifying intrapapillary capillary images provided by the embodiment of the present invention is configured to input the endoscopic image of the intrapapillary capillary into the pre-trained neural network model to obtain the effective area of the endoscopic image. Obtain the target blood vessel area in the endoscopic image from the effective area according to the connected domain algorithm; obtain the characteristic diameter, characteristic distortion quantification value, and characteristic area ratio of the target blood vessel in the target blood vessel area , the centroid eccentricity and the whole image density; input the characteristic diameter, the characteristic distortion quantization value, the characteristic area ratio, the centroid eccentricity and the whole image density into the pre-trained In the classification model, the classification result of the endoscopic image is obtained.

Please refer to FIG. 9. FIG. 9 is a schematic block diagram of a computer device provided by an embodiment of the present invention.

Referring to FIG. 9 , the device 500 includes a processor 502 , a memory and a network interface 505 connected through a system bus 501 , wherein the memory may include a storage medium 503 and an internal memory 504 .

The storage medium 503 can store an operating system 5031 and a computer program 5032 . The computer program 5032, when executed, can cause the processor 502 to perform a classification method of epithelial intrapapillary capillary images.

The processor 502 is used to provide computing and control capabilities to support the operation of the entire device 500 .

The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the computer program 5032 can cause the processor 502 to perform a method for classifying intrapapillary capillary images.

The network interface 505 is used for network communication, such as providing transmission of data information. Those skilled in the art can understand that the structure shown in FIG. 9 is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the device 500 to which the solution of the present invention is applied. The specific device 500 may be Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

Wherein, the processor 502 is used to run the computer program 5032 stored in the memory, so as to realize the following function: input the endoscopic image of the capillaries in the epithelial papilla into the pre-trained neural network model to obtain the endoscopic image The effective area of proportion, centroid eccentricity and whole image density; input the characteristic diameter, the characteristic distortion quantification value, the characteristic area proportion, the centroid eccentricity and the whole image density into In the pre-trained classification model, the classification result of the endoscopic image is obtained.

Those skilled in the art can understand that the embodiment of the device 500 shown in FIG. 9 does not constitute a limitation on the specific structure of the device 500. In other embodiments, the device 500 may include more or less components than those shown. Either some components are combined, or different component arrangements. For example, in some embodiments, the device 500 may only include the memory and the processor 502. In such an embodiment, the structures and functions of the memory and the processor 502 are the same as those of the embodiment shown in FIG. 8 , which will not be repeated here.

It should be understood that, in this embodiment of the present invention, the processor 502 may be a central processing unit (Central Processing Unit, CPU), and the processor 502 may also be other general-purpose processors 502, digital signal processors 502 (Digital Signal Processor, DSP) , Application Specific Integrated Circuit (ASIC), off-the-shelf Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. Wherein, the general-purpose processor 502 may be a microprocessor 502 or the processor 502 may also be any conventional processor 502 or the like.

In another embodiment of the present invention, a computer storage medium is provided. The storage medium may be a non-volatile computer-readable storage medium or a volatile storage medium. The storage medium stores a computer program 5032, wherein when the computer program 5032 is executed by the processor 502, the following steps are implemented: input the endoscopic image of the capillaries in the epithelial papillary into a pre-trained neural network model, and obtain the endoscopic image of the endoscopic image. Effective area; obtain the target blood vessel area in the endoscopic image from the effective area according to the connected domain algorithm; obtain the characteristic diameter, characteristic distortion quantification value, characteristic area proportion of the target blood vessel in the target blood vessel area ratio, centroid eccentricity and whole image density; input the characteristic diameter, the characteristic distortion quantification value, the characteristic area ratio, the centroid eccentricity and the whole image density into the In the trained classification model, the classification result of the endoscopic image is obtained.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices, devices and units, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. Interchangeability, the above description has generally described the components and steps of each example in terms of function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only logical function division. In actual implementation, there may be other division methods, or units with the same function may be grouped into one Units, such as multiple units or components, may be combined or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may also be electrical, mechanical or other forms of connection.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solutions in the embodiments of the present invention.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a storage medium. Based on such understanding, the technical solution of the present invention is essentially or a part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a device 500 (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: a U disk, a removable hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk and other media that can store program codes.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of various equivalents within the technical scope disclosed by the present invention. Modifications or substitutions should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (9)

1. a classification method of capillary images in epithelial papilla, is characterized in that, comprises: Input the endoscopic image of the capillaries in the epithelial papilla into the pre-trained neural network model to obtain the effective area of the endoscopic image; Obtain the target blood vessel region in the endoscopic image from the effective region according to a connected domain algorithm; obtaining the characteristic diameter, characteristic distortion quantification value, characteristic area ratio, centroid eccentricity, and whole image density of the target blood vessel in the target blood vessel region; Inputting the characteristic diameter, the characteristic distortion quantification value, the characteristic area ratio, the centroid eccentricity and the whole image density into a pre-trained classification model to obtain the endoscope Image classification results; Wherein, obtaining the characteristic diameter, characteristic distortion quantification value, characteristic area ratio, centroid eccentricity and whole image density of the target blood vessel in the target blood vessel region includes: generating the characteristic diameter according to the largest class average diameter and the smallest class average diameter of the target blood vessel; Obtaining a plurality of tortuosity quantification values and a plurality of area proportions of the target blood vessel respectively; wherein, each area proportion is calculated according to a preset area proportion formula; Calculate a plurality of tortuosity quantification values according to a preset blood vessel tortuosity quantification formula to obtain the characteristic tortuosity quantification value; Calculate the multiple area proportions according to a preset geometric mean calculation formula to obtain the characteristic area proportions; generating the centroid eccentricity according to the equivalent centroid of the effective area and the geometric center of the endoscopic image; Obtain the whole image density according to a preset blood vessel whole image density formula; The calculation formula of the characteristic diameter is:

in,

is the largest class mean diameter,

is the average diameter of the smallest class; The characteristic distortion quantification formula is:

in,

is the quantification value of the tortuosity of the blood vessel, n is the total number of pixels on the inner and outer sides of the microvascular wall,

and

The distribution is the length and width of the smallest circumscribed rectangle of the blood vessel, respectively; The area ratio formula is:

in,

is the blood vessel diameter at each pixel on the blood vessel centerline,

and

The distributions are the length and width of the smallest circumscribed rectangle of the blood vessel, respectively, and _Δi is the area ratio of each part in the target blood vessel; The geometric mean calculation formula is:

Wherein, Δ is the characteristic distortion quantity; The calculation formula of the centroid eccentricity is:

Among them, e is the eccentricity of the center of mass,

is the equivalent centroid of the effective area,

is the centroid of each vessel in the endoscopic image,

is the area of each vessel in the endoscopic image,

and

Obtained on the basis of the connected domain,

is the geometric center of the endoscopic image,

; The blood vessel whole image density formula is:

in,

is the area of each blood vessel in the whole image of blood vessels, which can be obtained by the connected domain, where W and H are the width and height of the endoscopic image, respectively. 2 . The method for classifying intrapapillary capillary images according to claim 1 , wherein the endoscopic images of intrapapillary capillaries are input into a pre-trained neural network model to obtain the endoscopic image. 3 . Before the effective area of the mirror image, it also includes: Input the training sample into the neural network model to obtain the mean square error loss of the neural network model; The network parameters of the neural network model are updated according to the mean square error loss. 3. The method for classifying intrapapillary capillary images according to claim 2, characterized in that, before the training samples are input into the neural network model, before the mean square error loss of the neural network model is obtained ,Also includes: performing video decoding on the endoscopic video of the capillaries in the epithelial papilla to obtain a video decoded image; Annotate the video decoded image to obtain the training sample. 4 . The method for classifying intrapapillary capillary images according to claim 1 , wherein the extraction of the target blood vessel region in the endoscopic image from the effective region according to a connected domain algorithm comprises: 5 . Traverse the effective area to obtain the connected domain area and the minimum circumscribed horizontal rectangle of the target blood vessel; The target blood vessel area is determined according to the area of the connected domain and the minimum circumscribed horizontal rectangle. 5 . The method for classifying intrapapillary capillary images according to claim 1 , wherein, before generating the characteristic diameter according to the largest class average diameter and the smallest class average diameter of the target blood vessel, further 5 . include: obtaining a plurality of diameters of each blood vessel in the target blood vessel; Screen multiple diameters of each branch to obtain the diameter after screening; According to the K-means algorithm, the diameters after screening are clustered to obtain the clustering results of each diameter after screening; The largest class average diameter and the smallest class average diameter are generated according to the clustering result. 6. The method for classifying epithelial intrapapillary capillary images according to claim 5, wherein the screening of the multiple diameters of each blood vessel to obtain the screened diameters comprises: Obtain the average diameter of each blood vessel and the variance of the diameter of each blood vessel; Screen the diameters of each blood vessel according to the average diameter of each blood vessel and the variance of the diameter of each blood vessel to obtain the screened diameter. 7. A device for classifying intrapapillary capillary images, comprising: a first input unit, configured to input the endoscopic image of the capillaries in the epithelial papilla into a pre-trained neural network model to obtain an effective area of the endoscopic image; a first acquiring unit, configured to acquire a target blood vessel region in the endoscopic image from the effective region according to a connected domain algorithm; a second acquiring unit, configured to acquire the characteristic diameter, characteristic distortion quantification value, characteristic area ratio, centroid eccentricity, and whole image density of the target blood vessel in the target blood vessel region; The second input unit is used for inputting the characteristic diameter, the characteristic distortion quantization value, the characteristic area ratio, the centroid eccentricity and the whole image density to the pre-trained classification model , obtain the classification result of the endoscopic image; Wherein, obtaining the characteristic diameter, characteristic distortion quantification value, characteristic area ratio, centroid eccentricity, and whole image density of the target blood vessel in the target blood vessel area, including: generating the characteristic diameter according to the largest class average diameter and the smallest class average diameter of the target blood vessel; Obtaining a plurality of tortuosity quantification values and a plurality of area proportions of the target blood vessel respectively; wherein, each area proportion is calculated according to a preset area proportion formula; Calculate a plurality of tortuosity quantification values according to a preset blood vessel tortuosity quantification formula to obtain the characteristic tortuosity quantification value; Calculate the multiple area proportions according to a preset geometric mean calculation formula to obtain the characteristic area proportions; generating the centroid eccentricity according to the equivalent centroid of the effective area and the geometric center of the endoscopic image; Obtain the whole image density according to a preset blood vessel whole image density formula; The calculation formula of the characteristic diameter is:

in,

is the largest class mean diameter,

is the average diameter of the smallest class; The characteristic distortion quantification formula is:

in,

is the quantification value of the tortuosity of the blood vessel, n is the total number of pixels on the inner and outer sides of the microvascular wall,

and

The distribution is the length and width of the smallest circumscribed rectangle of the blood vessel, respectively; The area ratio formula is:

in,

is the blood vessel diameter at each pixel on the blood vessel centerline,

and

The distributions are the length and width of the smallest circumscribed rectangle of the blood vessel, respectively, and _Δi is the area ratio of each part in the target blood vessel; The geometric mean calculation formula is:

Wherein, Δ is the characteristic distortion quantity; The calculation formula of the centroid eccentricity is:

Among them, e is the eccentricity of the center of mass,

is the equivalent centroid of the effective area,

is the centroid of each vessel in the endoscopic image,

is the area of each vessel in the endoscopic image,

and

Obtained on the basis of the connected domain,

is the geometric center of the endoscopic image,

; The blood vessel whole image density formula is:

in,

is the area of each blood vessel in the whole image of blood vessels, which can be obtained by the connected domain, where W and H are the width and height of the endoscopic image, respectively. 8. A computer device comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements the computer program as claimed in the claims The method for classifying epithelial intrapapillary capillary images according to any one of 1 to 6. 9. A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, the computer program, when executed by a processor, causes the processor to execute any one of claims 1 to 6 The classification method of epithelial intrapapillary capillary images described in .

Images