WO2021031066A1 - Cartilage image segmentation method and apparatus, readable storage medium, and terminal device - Google Patents

Abstract

The present application is applicable to the technical field of image processing, and in particular, to a cartilage image segmentation method and apparatus. The method comprises: obtaining a target cartilage image to be segmented; inputting the target cartilage image to a preset cartilage image segmentation model, the cartilage image segmentation model comprising an atrous convolution module, an atrous pyramid pooling module connected to the atrous convolution module, an attention mechanism module connected to the atrous pyramid pooling module, and a fusion module separately connected to the atrous convolution module and the attention mechanism module; extracting features of the target cartilage image by means of the atrous convolution module to obtain a first feature map; pooling the first feature map by means of the atrous pyramid pooling module to obtain a second feature map, and weighting the second feature map by means of the attention mechanism module to obtain a third feature map; and up-sampling the third feature map by means of the fusion module, and fusing a fourth feature map obtaining by sampling with the first feature map to obtain a cartilage segmentation result.

Description

Cartilage image segmentation method, device, readable storage medium and terminal equipment Technical field

This application belongs to the field of image processing technology, and in particular relates to a method, device, computer-readable storage medium, and terminal equipment for cartilage image segmentation.

Background technique

In the medical field, it is often necessary to segment the cartilage image to obtain the segmented cartilage, so that parameters such as the thickness and volume of the cartilage can be calculated to facilitate the assessment of the cartilage condition, thereby facilitating the diagnosis of cartilage diseases. The current cartilage image segmentation method is mainly based on the convolutional neural network model. Due to the particularity of cartilage characteristics, the existing segmentation method based on the convolutional neural network model still has the problem of low segmentation accuracy.

technical problem

The embodiments of the present application provide a cartilage image segmentation method, device, computer-readable storage medium, and terminal equipment, which can solve the problem of low accuracy of cartilage image segmentation.

Technical solutions

In the first aspect, an embodiment of the present application provides a cartilage image segmentation method, including:

Acquiring an image of the target cartilage to be segmented;

Input the target cartilage image to a preset cartilage image segmentation model, where the cartilage image segmentation model includes a cavity convolution module, a pyramid cavity pooling module connected to the cavity convolution module, and the pyramid cavity An attention mechanism module connected to the pooling module and a fusion module connected to the hollow convolution module and the attention mechanism module respectively;

Performing feature extraction on the target cartilage image through the cavity convolution module to obtain a first feature map corresponding to the target cartilage image;

The first feature map is pooled by the pyramid hole pooling module to obtain a second feature map corresponding to the target cartilage image, and the second feature map is weighted by the attention mechanism module Processing to obtain a third feature map corresponding to the target cartilage image;

Up-sampling the third feature map by the fusion module, and fusing the sampled fourth feature map with the first feature map to obtain the target cartilage image output by the cartilage image segmentation model Cartilage segmentation results.

In a second aspect, an embodiment of the present application provides a cartilage image segmentation device, including:

The target image acquisition module is used to acquire the target cartilage image to be segmented;

The target image input module is used to input the target cartilage image to a preset cartilage image segmentation model, wherein the cartilage image segmentation model includes a cavity convolution module and a pyramid cavity pooling connected to the cavity convolution module Module, an attention mechanism module connected to the pyramid hole pooling module, and a fusion module connected to the hole convolution module and the attention mechanism module respectively;

A feature extraction module, configured to perform feature extraction on the target cartilage image through the cavity convolution module to obtain a first feature map corresponding to the target cartilage image;

The pooling weighting processing module is used to perform pooling processing on the first feature map through the pyramid hole pooling module to obtain a second feature map corresponding to the target cartilage image, and to perform the pooling process through the attention mechanism module Performing weighting processing on the second feature map to obtain a third feature map corresponding to the target cartilage image;

The result output module is used to up-sample the third feature map through the fusion module, and fuse the fourth feature map obtained by sampling with the first feature map to obtain the output of the cartilage image segmentation model The cartilage segmentation result of the target cartilage image.

In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, The cartilage image segmentation method described in the first aspect is realized.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium that stores a computer program that, when executed by a processor, implements the cartilage image segmentation method described in the first aspect above .

In a fifth aspect, embodiments of the present application provide a computer program product, which when the computer program product runs on a terminal device, causes the terminal device to execute the cartilage image segmentation method described in the first aspect.

Beneficial effect

In the embodiment of this application, multi-scale image information is extracted from the target cartilage image through the hole convolution module and the pyramid hole pooling module, and the multi-scale image information is fused through the fusion module, which can effectively retain the detailed information of the image and improve The image boundary segmentation capability improves the segmentation accuracy of cartilage images. In addition, the weighting of image information through the attention mechanism module can effectively enhance the segmentation ability of cartilage images and improve the accuracy and precision of cartilage image segmentation.

Description of the drawings

FIG. 1 is a schematic flowchart of a cartilage image segmentation method provided by an embodiment of the present application;

2 is a structural block diagram of a cartilage image segmentation model provided by an embodiment of the present application;

3 is a schematic structural diagram of a hole convolution module provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of convolution of a channel hole convolution layer provided by an embodiment of the present application;

FIG. 5 is a schematic flowchart of obtaining a second feature map in an application scenario by the cartilage image segmentation method provided by an embodiment of the present application;

Fig. 6 is a schematic structural diagram of a pyramid cavity pooling module provided by an embodiment of the present application;

Fig. 6a is a schematic diagram of convolution with different sampling rates provided by an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an attention mechanism module provided by an embodiment of the present application;

FIG. 8 is a schematic flowchart of obtaining a third feature map in an application scenario by the cartilage image segmentation method provided by an embodiment of the present application;

Figure 9a is a cartilage segmentation diagram of artificially labeled gold standard;

9b is a cartilage segmentation diagram segmented by the cartilage image segmentation method in an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a cartilage image segmentation device provided by an embodiment of the present application;

FIG. 11 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.

Embodiments of the invention

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are proposed for a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obstructing the description of this application.

It should be understood that when used in the specification and appended claims of this application, the term "comprising" indicates the existence of the described features, wholes, steps, operations, elements and/or components, but does not exclude one or more other The existence or addition of features, wholes, steps, operations, elements, components, and/or collections thereof.

In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

Fig. 1 shows a schematic flowchart of a cartilage image segmentation method provided by an embodiment of the present application, and the cartilage image segmentation method includes:

Step S101: Obtain a target cartilage image to be segmented;

The execution subject of the embodiments of the present application may be a terminal device, and the terminal device includes, but is not limited to, computing devices such as desktop computers, notebooks, palmtop computers, and cloud servers. When cartilage segmentation is required, the target cartilage image to be segmented may be sent to the terminal device, where the target cartilage image may be a magnetic resonance imaging (MRI) image containing cartilage, for example, an MRI of a knee joint image.

Step S102. Input the target cartilage image to a preset cartilage image segmentation model, where the cartilage image segmentation model includes a cavity convolution module, a pyramid cavity pooling module connected to the cavity convolution module, and The attention mechanism module connected to the pyramid hole pooling module and the fusion module respectively connected to the hole convolution module and the attention mechanism module;

Specifically, after the terminal device obtains the target cartilage image, it can call the cartilage image segmentation model as shown in FIG. 2 and can input the target cartilage image into the cartilage image segmentation model.

In a possible implementation manner, the inputting the target cartilage image to a preset cartilage image segmentation model may include:

Step a: Obtain the original resolution and original sampling distance corresponding to the target cartilage image;

Step b: Determine the target sampling distance according to the original resolution, the original sampling distance, and the preset target resolution;

Step c: Use the target sampling distance to resample the target cartilage image, and input the resampled target cartilage image into a preset cartilage image segmentation model.

Wherein, the determining the target sampling distance according to the original resolution, the original sampling distance, and the preset target resolution includes:

Determine the target sampling distance according to the following formula:

Spacing is the target sampling distance, spacing' is the original sampling distance, ImageRe' is the original resolution, and ImageRe is the target resolution.

For the above steps a to c, the original resolution can be any resolution, the original sampling distance can be any distance, and the target resolution can be a resolution of 513×513 pixels to determine the target The sampling distance is used to perform image resampling to crop the target cartilage image to the target resolution, which is convenient for the cartilage image segmentation model to perform image feature extraction, improves the segmentation efficiency of the cartilage image segmentation model, and can reduce the The size of the target cartilage image is limited to facilitate user use and improve user experience.

Step S103: Perform feature extraction on the target cartilage image through the cavity convolution module to obtain a first feature map corresponding to the target cartilage image;

It should be understood that after the terminal device inputs the target cartilage image into the cartilage image segmentation model, the cavity convolution module of the cartilage image segmentation model can extract image features of the target cartilage image to obtain The first feature map corresponding to the target cartilage image.

In a possible implementation manner, as shown in FIG. 3, the hole convolution module is a convolution module based on the Xception network structure, where the Xception network structure includes a flat hole convolution layer and a channel hole convolution layer The sampling rate of the planar hole convolution layer is 1 or 3, and the sampling rate of the channel hole convolution layer is 6. It should be understood that the size of the convolution kernel of the planar hole convolution layer and the channel hole convolution layer may both be 3×3.

Specifically, the Xception network structure may include an input unit, an intermediate processing unit, and an output unit. The input unit may include a first 3×3 convolutional layer connected in series (that is, a convolutional layer with a convolution kernel size of 3×3). , The following similar expressions have the same meaning), the second 3×3 convolutional layer, the first convolution subunit, the second convolution subunit, and the third convolution subunit, wherein the first convolution subunit, Both the second convolution subunit and the third convolution subunit can include the first 1×1 convolution layer and the concatenated first plane hole convolution layer (the plane convolution shown in Figure 3), the second plane hole Convolutional layer (planar convolution shown in Figure 3) and the first channel convolutional layer (hole convolution shown in Figure 3); the intermediate processing unit may include 16 fourth convolutions connected in series Unit, the fourth convolution subunit may include three third planar hole convolution layers connected in series; the output unit may include a fifth convolution subunit and a sixth convolution subunit connected in series, the fifth The convolution subunit may include a second 1×1 convolution layer and a fourth planar hole convolution layer, a fifth planar hole convolution layer, and a second channel hole convolution layer connected in series. The sixth convolution subunit may It includes a concatenated sixth plane hole convolution layer, a third channel hole convolution layer and a seventh plane hole convolution layer.

It should be noted that after the first 1×1 convolutional layer, the second 1×1 convolutional layer, and the first channel hole convolutional layer, a standardized output layer and a Relu activation layer may be connected in sequence. . As shown in FIG. 4, the first channel hole convolution layer, the second channel hole convolution layer, and the third channel hole convolution layer mainly adopt depth-wise hole convolution, which can be specifically formed by convolution The core size is 3x3 hole convolution and the cross-module convolution core size is 1x1 convolution in series, which can reduce model parameters and improve convolution efficiency.

As shown in FIG. 3, after the terminal device inputs the target cartilage image into the cartilage image segmentation model, the input unit of the cavity convolution module may first perform feature extraction on the target cartilage image, and may The extracted feature map A is input to the intermediate processing unit of the hole convolution module, and the intermediate processing unit can perform feature extraction on the feature map A, and can input the extracted feature map B to the hole convolution The output unit of the module, the output unit can further perform feature extraction on the feature map B to obtain the first feature map corresponding to the target cartilage image.

Specifically, the process for the input unit to perform feature extraction on the target cartilage image may be: the first 3×3 convolutional layer of the input unit may first perform feature extraction on the target cartilage image, and may extract The feature map R of the input unit is input to the second 3×3 convolutional layer of the input unit; the second 3×3 convolutional layer can perform further feature extraction on the feature map R, and the extracted feature map T Input to the first convolution subunit of the input unit. Here, the first 1×1 convolution layer of the first convolution subunit can perform feature extraction on the feature map T to obtain the extracted feature map T1, and the first convolution subunit The planar cavity convolution layer can also perform feature extraction on the feature map T to obtain the extracted feature map T2, and can input the feature map T2 to the second planar cavity convolution layer of the first convolution subunit. The second plane hole convolution layer can perform feature extraction on the feature map T2, and can input the extracted feature map T21 into the first channel hole convolution layer of the first convolution subunit, and the first channel hole The convolutional layer can perform feature extraction on the feature map T21 to obtain the feature map S. Subsequently, the first convolution subunit may fuse the feature map T1 and the feature map S, and input the feature map L obtained by the fusion into the second convolution subunit of the input unit. The second convolution subunit can perform feature extraction on the feature map L, and input the extracted feature map H into the third convolution subunit of the input unit, and the third convolution subunit can The feature map H performs feature extraction to obtain the feature map A extracted by the input unit. Here, the feature extraction process performed by the second convolution subunit on the feature map L and the feature extraction process performed by the third convolution subunit on the feature map H are the same as those performed by the first convolution subunit on the feature map. The feature extraction process performed by T is similar, and the basic principle is the same. For the sake of brevity, it will not be repeated here.

Correspondingly, the process of the intermediate processing unit performing feature extraction on the feature map A may be: the first third planar hole convolution layer of the first fourth convolution subunit can first perform feature extraction on the feature map A, And the extracted feature map A1 can be input to the second third plane hole convolution layer of the first fourth convolution subunit, and the second third plane hole convolution layer can compare the feature map A1 Perform feature extraction, and input the extracted feature map A11 into the third third plane hole convolution layer of the first fourth convolution subunit, and the third third plane hole convolution layer can be Feature map A11 performs feature extraction to obtain feature map G; then the first fourth convolution subunit can fuse feature map A and feature map G, and can input the fused feature map K to the second The fourth convolution subunit, the second and fourth convolution subunit can perform feature extraction on the feature map K, and input the extracted feature map to the third and fourth convolution subunit, and so on, until Input the feature map to the sixteenth fourth convolution subunit, and perform feature extraction on the feature map through the sixteenth fourth convolution subunit, so as to obtain the feature map B extracted by the intermediate processing unit. Here, the second and fourth convolution subunits, the third and fourth convolution subunits,..., the process of feature extraction by the sixteenth and fourth convolution subunits is the same as that of the first and fourth convolution subunits. The feature extraction process of the feature map A by the unit is similar, and the basic principle is the same. For the sake of brevity, it will not be repeated here.

Correspondingly, the feature extraction process of the feature map B by the output unit may be as follows: the fifth convolution subunit of the output unit can first perform feature extraction on the feature map B, and can input the extracted feature map F to The sixth convolution subunit of the output unit, where the fifth convolution subunit performs feature extraction on the feature map B and the first convolution subunit of the input unit performs feature map T The extraction process is similar, and the basic principle is the same. For the sake of brevity, I will not repeat it here. Subsequently, the sixth plane hole convolution layer of the sixth convolution subunit can perform feature extraction on the feature map F, and can input the extracted feature map F1 into the third channel hole of the sixth convolution subunit Convolutional layer, the third channel hole convolution layer can perform feature extraction on the feature map F1, and can input the extracted feature map F11 to the seventh plane hole convolution layer of the sixth convolution subunit, The feature map F11 can be extracted through the seventh plane cavity convolution layer to obtain the first feature map corresponding to the target cartilage image.

It should be noted that the Xception network structure in the embodiment of the present application is similar to the residual network structure, so the hole convolution module based on the Xception network structure can effectively reduce the gradient attenuation rate, avoid the degradation of the network structure, and ensure cartilage image segmentation Accuracy. In addition, the hole convolution module adopts the hole convolution layer with different sampling rates to extract the features of the target cartilage image, which can increase the receptive field, increase the amount of information contained in the feature map, and effectively retain the detailed information of the image. Improve the segmentation accuracy of cartilage images.

Step S104: Perform pooling processing on the first feature map by the pyramid hole pooling module to obtain a second feature map corresponding to the target cartilage image, and perform a pooling process on the second feature map by the attention mechanism module The image is weighted to obtain a third feature image corresponding to the target cartilage image;

It should be understood that the cavity convolution module, the pyramid cavity pooling module, and the attention mechanism module of the cartilage image segmentation model are sequentially connected in series. Therefore, the cavity convolution module extracts the first feature map corresponding to the target cartilage image After that, the first feature map can be input to the pyramid hole pooling module, and the pyramid hole pooling module can perform pooling processing on the first feature map to obtain the first feature map Corresponding to the second feature map, the second feature map can be input to the attention mechanism module, and the attention mechanism module can perform weighting processing on the second feature map to obtain the target cartilage The third feature map corresponding to the image uses the pyramid hole pooling module to extract image information at multiple scales, which improves the boundary segmentation capability of the image and improves the segmentation accuracy of the cartilage image. At the same time, the attention mechanism module is used to weight the image information. It can effectively improve the segmentation accuracy and segmentation accuracy of cartilage images.

In a possible implementation manner, the pyramid hole pooling module includes a plurality of first convolution branches parallel to each other;

Specifically, as shown in FIG. 5, performing pooling processing on the first feature map by the pyramid hole pooling module to obtain a second feature map corresponding to the target cartilage image may include:

Step S501: Perform feature sampling on the first feature map through each of the first convolution branches to obtain a first sampling feature map, a second sampling feature map, and a third sampling feature corresponding to the first feature map. Figure and the fourth sampling feature map;

Step S502, splicing the first sampling feature map, the second sampling feature map, the third sampling feature map, and the fourth sampling feature map to obtain a spliced splicing feature map;

Step S503: Perform average pooling processing on the stitched feature map to obtain a second feature map corresponding to the target cartilage image.

It should be noted that the first convolution branch includes a first hole convolution unit, a second hole convolution unit, and a third hole convolution unit with different sampling rates, and the first hole convolution unit and the first hole convolution unit Two hole convolution units are connected, and the second hole convolution unit is connected to the third hole convolution unit.

Specifically, as shown in FIG. 6, the pyramid hole pooling module may include four first convolution branches parallel to each other, and each first convolution branch may include a first hole convolution unit and a second convolution unit connected in series in sequence. A hole convolution unit and a third hole convolution unit, wherein the first hole convolution unit may include a convolution layer with a sampling rate of 6, as shown in FIG. 6a, a first Relu activation layer and a first dropout layer, The second hole convolution unit may include a convolution layer with a sampling rate of 12 as shown in FIG. 6a, a second Relu activation layer, and a second dropout layer, and the third hole convolution unit may include the convolution layer shown in FIG. 6a. The sample rate shown is 18 convolutional layers. Optionally, the pyramid hole pooling module may further include a splicing layer connected to the convolutional layer with a sampling rate of 18 and an average pooling layer connected to the splicing layer.

It should be understood that after the pyramid hole pooling module obtains the first feature map extracted by the hole convolution module, it can perform feature sampling on the first feature map through four parallel first convolution branches respectively. Obtain a first sampling feature map, a second sampling feature map, a third sampling feature map, and a fourth sampling feature map corresponding to the first feature map, respectively. Here, the process of performing feature sampling on the first feature map by the first convolution branch to obtain the first sampling feature map may be as follows: first, the first convolutional layer with a sampling rate of 6 can The feature map performs feature sampling, and the sampled feature map C obtained by sampling can be input to the first Relu activation layer; secondly, the sampled feature map C can be processed through the first Relu activation layer, and the processed feature map C can be processed The sampling feature map C1 is input to the first dropout layer; the sampling feature map C1 can be processed again through the first dropout layer, and the processed sampling feature map C2 can be input to the convolutional layer with a sampling rate of 12 The sampling feature map C2 can then be further sampled through the convolutional layer with a sampling rate of 12, and the sampling feature map C3 obtained by sampling can be input to the second Relu activation layer; then the second Relu activation layer can be used The Relu activation layer processes the sampled feature map C3, and can input the processed sampled feature map C4 to the second dropout layer, so as to process the sampled feature map C4 through the second dropout layer, and can process The obtained sampling feature map C5 is input to the convolutional layer with a sampling rate of 18; finally, the sampling feature map C5 can be sampled by the convolutional layer with a sampling rate of 18 to obtain the first feature map corresponding to the first feature map. A sampling feature map. Wherein, the process of performing feature sampling on the first feature map by each of the other first convolution branches to obtain the second sampling feature map, the third sampling feature map, and the fourth sampling feature map is the same as that obtained above. The process of the first sampling feature map is similar, and the basic principle is the same. For the sake of brevity, it will not be repeated here.

It should be noted that after the first convolution branch obtains the first sampling feature map, the second sampling feature map, the third sampling feature map, and the fourth sampling feature map, all The first sampling feature map, the second sampling feature map, the third sampling feature map, and the fourth sampling feature map are respectively input to the splicing layer of the pyramid hole pooling module, and the splicing layer is sufficient The first sampling feature map, the second sampling feature map, the third sampling feature map, and the fourth sampling feature map are spliced together. For example, the first sampling feature map, the second sampling feature map can be combined The sampling feature map, the third sampling feature map, and the fourth sampling feature map are summed and spliced to obtain the spliced splicing feature map, and the obtained splicing feature map can be input to the pyramid hole pooling module The average pooling layer of, the average pooling layer can perform average pooling processing on the stitched feature map, so as to obtain the second feature map corresponding to the target cartilage image.

In a possible implementation, as shown in FIG. 7, the attention mechanism module may include multiple second convolution branches parallel to each other, for example, may include three second convolution branches parallel to each other, where: Each of the second convolution branches may include a convolution layer with a convolution kernel size of 1×1 and a step size of 2.

Specifically, as shown in FIG. 8, performing weighting processing on the second feature map by the attention mechanism module to obtain a third feature map corresponding to the target cartilage image may include:

Step S801: Perform convolution processing on the second feature map through each of the second convolution branches to obtain the first convolution feature map, the second convolution feature map, and the first convolution feature map corresponding to the second feature map. Three convolution feature maps;

Step S802: Perform transposition processing on the first convolution feature map, and perform matrix multiplication processing on the transposed feature map obtained by the transposition and the second convolution feature map to obtain a fifth feature map;

Step S803: Perform normalization processing on the fifth feature map, and perform matrix multiplication processing on the normalized fifth feature map and the third convolution feature map to obtain the second feature map Corresponding weighting coefficient matrix;

Step S804: Perform weighting processing on the second feature map through the weighting coefficient matrix to obtain a third feature map corresponding to the target cartilage image.

For steps S801 to S804, after the pyramid hole pooling module obtains the second feature map corresponding to the target cartilage image, it can input the second feature map to the attention mechanism module, and the attention The mechanism module can first perform convolution processing on the second feature map through three parallel 1×1 convolution layers to obtain three convolution feature maps corresponding to the second feature map, that is, through three 1×1 convolutional layers. The convolutional layer performs dimensionality reduction processing on the second feature map to generate a first convolution feature map, a second convolution feature map, and a third convolution feature map that retain detailed information; then the first convolution feature map can be The product feature map is transposed, and the transposed feature map obtained by the transposition can be subjected to matrix multiplication processing with the second convolution feature map to obtain a fifth feature map; subsequently, the fifth feature map can be Perform normalization processing, for example, the fifth feature map can be normalized through the softmax function, and the normalized fifth feature map can be matrix-multiplied by the third convolution feature map Processing, the weighting coefficient matrix corresponding to the second feature map is obtained, that is, the attention of each position in the feature map relative to other positions is obtained; finally, the second feature map can be weighted by the weighting coefficient matrix, To obtain the third feature map corresponding to the target cartilage image, for example, a weighted third feature map can be obtained by summing the second feature map and the weighting coefficient matrix on the basis of preset coefficients.

Step S105: Up-sampling the third feature map by the fusion module, and fusing the sampled fourth feature map with the first feature map to obtain the target output by the cartilage image segmentation model Cartilage segmentation result of cartilage image.

In a possible implementation manner, the third feature map is upsampled by the fusion module, and the fourth feature map obtained by sampling is fused with the first feature map to obtain the cartilage The cartilage segmentation result of the target cartilage image output by the image segmentation model may include:

Step d: Perform bilinear up-sampling on the third feature map by the fusion module to obtain the fourth feature map;

Step e: Perform convolution processing on the first feature map through the third convolution branch of the fusion module to obtain a sixth feature map corresponding to the first feature map;

Step f: Perform fusion processing on the fourth feature map and the sixth feature map to obtain a fused seventh feature map;

Step g: Perform bilinear upsampling on the seventh feature map to obtain the cartilage segmentation result of the target cartilage image output by the cartilage image segmentation model.

For the above steps d to g, it should be understood that the bilinear upsampling may be 4 times bilinear upsampling, and the third convolution branch of the fusion module may include a convolution kernel with a size of 1×1. Layers, wherein the third convolution branch can be connected to the hole convolution module to obtain the first feature map output by the hole convolution module, and can perform convolution processing on the first feature map , Thereby obtaining a sixth feature map corresponding to the first feature map. The fusion module may further perform fusion processing on the fourth feature map and the sixth feature map, that is, perform layer-wise splicing on the fourth feature map and the sixth feature map, so as to preserve the void volume. The useful information in the feature map output by the product module improves the segmentation accuracy and segmentation accuracy of the cartilage image; finally, the size of the original feature map can be restored by performing 4-fold bilinear upsampling on the fused seventh feature map.

It should be noted that, in the embodiment of the present application, the cartilage image segmentation model can be obtained through training in the following steps:

Step h: Acquire a first preset number of first training cartilage images;

Step i. Expand the first training cartilage image by using a preset expansion method to obtain a second preset number of second training cartilage images, where the second training cartilage image includes the first training cartilage image, and The second preset quantity is greater than the first preset quantity;

Step j: Use the second training cartilage image and a preset loss function to train the cartilage image segmentation model, and the loss function is:

Where, B is the number of training cartilage images, N is the number of pixels of each training cartilage image, p _ij is the probability that the j-th pixel of the i-th training cartilage image belongs to the cartilage, α=0.75, γ=2.

For the above steps h to j, when training the cartilage image segmentation model, a first preset number (such as 440) of first training cartilage images of different resolutions may be acquired from the medical imaging control system mimics; , Each first training cartilage image can be preprocessed, and the original resolution and original sampling distance corresponding to each first training cartilage image can be obtained, and the original resolution, original sampling distance and original sampling distance corresponding to each first training cartilage image can be obtained. The preset target resolution determines the corresponding target sampling distance, and uses each target sampling distance to resample the corresponding first training cartilage image so that all the first training cartilage images are at the same origin and the same direction. Here, The method for determining the target sampling distance at the location is the same as the method for determining the target sampling distance described above; then, the first training software can be expanded by performing symmetric, stretching, and rotating radiation transformations on each first training cartilage image along the xy plane. The number of images is expanded to obtain a second preset number of second training cartilage images, where the second training cartilage images can include the first training cartilage images; finally, the second training can be used under the guidance of the Focal loss method The cartilage image trains the cartilage image segmentation model to obtain the optimal model parameters. Focal loss can be used to define the loss function loss during the cartilage image segmentation model training, and the loss function is minimized based on the Adam batch gradient descent algorithm loss to obtain the optimal model parameters.

It should be understood that a dropout layer with a dropout rate of 0.9 can also be used in the training process to improve training efficiency.

Table 1 below shows the comparison results of the cartilage image segmentation method in the embodiment of the present application, the cartilage image method based on the Deeplab v3 structure, and the cartilage image method based on the U-net structure, where Dice is the similarity coefficient ( Dice similarity coefficient, DSC) is a parameter for evaluating the effect of cartilage segmentation. The larger the Dice, the higher the segmentation accuracy, and the smaller the average surface distance, the higher the segmentation accuracy. It can be seen from the comparison results shown in Table 1 below that the cartilage image segmentation method in the embodiment of the present application is significantly better than the cartilage image method based on the Deeplab v3 structure and the cartilage image method based on the U-net structure.

Table 1

To	Dice(DSC)	Average surface distance (PG, GP)
The method of the embodiment of the application	77%	(0.69mm, 0.49mm)
Deeplab v3	65%	(2.56mm, 1.49mm)
U-net	55%	(2.93mm, 1.85mm)

In addition, FIG. 9a and FIG. 9b show a schematic diagram of the comparison of the cartilage segmentation results of the artificial labeling gold standard and the cartilage image segmentation method in the embodiment of the present application. Among them, FIG. 9a is the artificial labeling gold standard, and FIG. 9b is the embodiment of the application. The cartilage segmentation results of the cartilage image segmentation method in FIG. 9a and FIG. 9b show that the cartilage image segmentation method in the embodiment of the present application can reach the segmentation accuracy of the artificial labeling gold standard.

In the embodiment of this application, multi-scale image information is extracted from the target cartilage image through the hole convolution module and the pyramid hole pooling module, and the multi-scale image information is fused through the fusion module, which can effectively retain the detailed information of the image and improve The image boundary segmentation capability improves the segmentation accuracy of cartilage images. In addition, the weighting of image information through the attention mechanism module can effectively enhance the segmentation ability of cartilage images and improve the accuracy and precision of cartilage image segmentation.

Corresponding to the cartilage image segmentation method described in the above embodiment, FIG. 10 shows a structural block diagram of a cartilage image segmentation device provided by an embodiment of the present application. For ease of description, only the parts related to the embodiment of the present application are shown.

10, the cartilage image segmentation device includes:

The target image acquisition module 1001 is used to acquire the target cartilage image to be segmented;

The target image input module 1002 is configured to input the target cartilage image into a preset cartilage image segmentation model, wherein the cartilage image segmentation model includes a cavity convolution module and a pyramid cavity pool connected to the cavity convolution module A transformation module, an attention mechanism module connected to the pyramid hole pooling module, and a fusion module respectively connected to the hole convolution module and the attention mechanism module;

The feature extraction module 1003 is configured to perform feature extraction on the target cartilage image through the cavity convolution module to obtain a first feature map corresponding to the target cartilage image;

Pooling weighting processing module 1004, configured to perform pooling processing on the first feature map by the pyramid hole pooling module to obtain a second feature map corresponding to the target cartilage image, and pass the attention mechanism module Weighting the second feature map to obtain a third feature map corresponding to the target cartilage image;

The result output module 1005 is configured to up-sample the third feature map through the fusion module, and fuse the fourth feature map obtained by sampling with the first feature map to obtain the cartilage image segmentation model output The result of cartilage segmentation of the target cartilage image.

In a possible implementation manner, the target image input module 1002 includes:

An original sampling distance obtaining unit, configured to obtain the original resolution and original sampling distance corresponding to the target cartilage image;

A target sampling distance determining unit, configured to determine a target sampling distance according to the original resolution, the original sampling distance, and a preset target resolution;

The image resampling unit is configured to resample the target cartilage image by using the target sampling distance, and input the resampled target cartilage image into a preset cartilage image segmentation model.

Optionally, the target sampling distance determining unit is specifically configured to determine the target sampling distance according to the following formula:

Spacing is the target sampling distance, spacing' is the original sampling distance, ImageRe' is the original resolution, and ImageRe is the target resolution.

In a possible implementation manner, the hole convolution module is a convolution module based on an Xception network structure, wherein the Xception network structure includes a flat hole convolution layer and a channel hole convolution layer, and the flat hole convolution The sampling rate of the accumulation layer is 1 or 3, and the sampling rate of the channel hole convolutional layer is 6.

Optionally, the pyramid hole pooling module includes a plurality of first convolution branches parallel to each other;

The pooling weighting processing module 1004 includes:

The feature sampling unit is configured to perform feature sampling on the first feature map through each of the first convolution branches to obtain the first sampling feature map, the second sampling feature map, and the first feature map corresponding to the first feature map. Three sampling feature map and fourth sampling feature map;

A feature splicing unit, configured to splice the first sampling feature map, the second sampling feature map, the third sampling feature map, and the fourth sampling feature map to obtain a spliced splicing feature map;

The average pooling unit is configured to perform average pooling processing on the stitched feature map to obtain a second feature map corresponding to the target cartilage image.

It should be noted that the first convolution branch includes a first hole convolution unit, a second hole convolution unit, and a third hole convolution unit with different sampling rates, and the first hole convolution unit and the first hole convolution unit Two hole convolution units are connected, and the second hole convolution unit is connected to the third hole convolution unit.

In a possible implementation manner, the attention mechanism target includes multiple second convolution branches parallel to each other;

The pooling weighting processing module 1004 includes:

The first convolution processing unit is configured to perform convolution processing on the second feature map through each of the second convolution branches to obtain the first convolution feature map and the second feature map corresponding to the second feature map. Convolution feature map and the third convolution feature map;

The matrix multiplication unit is configured to perform transposition processing on the first convolution feature map, and perform matrix multiplication processing on the transposed feature map obtained by the transposition and the second convolution feature map to obtain a fifth feature Figure;

The normalization processing unit is configured to perform normalization processing on the fifth feature map, and perform matrix multiplication processing on the fifth feature map after the normalization processing and the third convolution feature map to obtain the The weighting coefficient matrix corresponding to the second feature map;

The weighting processing unit is configured to perform weighting processing on the second feature map through the weighting coefficient matrix to obtain a third feature map corresponding to the target cartilage image.

Optionally, the result output module 1005 includes:

The first upsampling unit is configured to perform bilinear upsampling on the third feature map by the fusion module to obtain the fourth feature map;

A second convolution processing unit, configured to perform convolution processing on the first feature map through the third convolution branch of the fusion module to obtain a sixth feature map corresponding to the first feature map;

A fusion processing unit, configured to perform fusion processing on the fourth feature map and the sixth feature map to obtain a fused seventh feature map;

The second up-sampling unit is configured to perform bilinear up-sampling on the seventh feature map to obtain the cartilage segmentation result of the target cartilage image output by the cartilage image segmentation model.

In a possible implementation manner, the cartilage image segmentation device includes:

A training image acquisition module for acquiring a first preset number of first training cartilage images;

The training image expansion module is used to expand the first training cartilage image by using a preset expansion method to obtain a second preset number of second training cartilage images, where the second training cartilage image includes the first training cartilage Images, the second preset number is greater than the first preset number;

The segmentation model training module is used to train the cartilage image segmentation model by using the second training cartilage image and a preset loss function, and the loss function is:

Where, B is the number of training cartilage images, N is the number of pixels of each training cartilage image, p _ij is the probability that the j-th pixel of the i-th training cartilage image belongs to the cartilage, α=0.75, γ=2.

It should be noted that the information interaction and execution process between the above-mentioned devices/units are based on the same concept as the method embodiments of this application, and their specific functions and technical effects can be found in the method embodiments. I won't repeat it here.

FIG. 11 is a schematic structural diagram of a terminal device provided by an embodiment of this application. As shown in FIG. 11, the terminal device 11 of this embodiment includes: at least one processor 1100 (only one is shown in FIG. 11), a processor, a memory 1101, and a processor stored in the memory 1101 and capable of being processed in the at least one processor. A computer program 1102 running on the processor 1100, and when the processor 1100 executes the computer program 1102, the steps in any of the foregoing cartilage image segmentation method embodiments are implemented.

The terminal device 11 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The terminal device may include, but is not limited to, a processor 1100 and a memory 1101. Those skilled in the art can understand that FIG. 11 is only an example of the terminal device 11, and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, input and output devices may also be included.

The so-called processor 1100 may be a central processing unit (Central Processing Unit, CPU), and the processor 1100 may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), and application specific integrated circuits (Application Specific Integrated Circuits). , ASIC), ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.

The memory 1101 may be an internal storage unit of the terminal device 11 in some embodiments, such as a hard disk or memory of the terminal device 11. In other embodiments, the memory 1101 may also be an external storage device of the terminal device 11, for example, a plug-in hard disk equipped on the terminal device 11, a smart media card (SMC), a secure digital (Secure Digital, SD) card, Flash Card, etc. Further, the memory 1101 may also include both an internal storage unit of the terminal device 11 and an external storage device. The memory 1101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 1101 can also be used to temporarily store data that has been output or will be output.

The embodiments of the present application also provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be realized.

The embodiments of the present application provide a computer program product. When the computer program product runs on a terminal device, the terminal device can realize the steps in the foregoing method embodiments when executed.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the implementation of all or part of the processes in the above-mentioned embodiment methods in the present application can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by the processor, the steps of the foregoing method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate forms. The computer-readable medium may at least include: any entity or device capable of carrying computer program code to the photographing device/terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), and random access memory (RAM, Random Access Memory), electric carrier signal, telecommunications signal and software distribution medium. Such as U disk, mobile hard disk, floppy disk or CD-ROM, etc. In some jurisdictions, according to legislation and patent practices, computer-readable media cannot be electrical carrier signals and telecommunication signals.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail or recorded in an embodiment, reference may be made to related descriptions of other embodiments.

A person of ordinary skill in the art may be aware that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed apparatus/equipment and method may be implemented in other ways. For example, the device/equipment embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or Components can be combined or integrated into another system, or some features can be omitted or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still implement the foregoing The technical solutions recorded in the examples are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in Within the scope of protection of this application.

Claims (15)

1. A method for segmentation of cartilage image, characterized in that it comprises:

   Acquiring an image of the target cartilage to be segmented;

   Input the target cartilage image to a preset cartilage image segmentation model, where the cartilage image segmentation model includes a cavity convolution module, a pyramid cavity pooling module connected to the cavity convolution module, and the pyramid cavity An attention mechanism module connected to the pooling module and a fusion module connected to the hollow convolution module and the attention mechanism module respectively;

   Performing feature extraction on the target cartilage image through the cavity convolution module to obtain a first feature map corresponding to the target cartilage image;

   The first feature map is pooled by the pyramid hole pooling module to obtain a second feature map corresponding to the target cartilage image, and the second feature map is weighted by the attention mechanism module Processing to obtain a third feature map corresponding to the target cartilage image;

   Up-sampling the third feature map by the fusion module, and fusing the sampled fourth feature map with the first feature map to obtain the target cartilage image output by the cartilage image segmentation model Cartilage segmentation results.
2. The cartilage image segmentation method according to claim 1, wherein the inputting the target cartilage image into a preset cartilage image segmentation model comprises:

   Acquiring the original resolution and original sampling distance corresponding to the target cartilage image;

   Determining the target sampling distance according to the original resolution, the original sampling distance, and the preset target resolution;

   The target cartilage image is resampled by using the target sampling distance, and the target cartilage image obtained by the resampling is input into a preset cartilage image segmentation model.
3. The cartilage image segmentation method according to claim 2, wherein the determining the target sampling distance according to the original resolution, the original sampling distance, and a preset target resolution comprises:

   Determine the target sampling distance according to the following formula:

   

   Spacing is the target sampling distance, spacing' is the original sampling distance, ImageRe' is the original resolution, and ImageRe is the target resolution.
4. The cartilage image segmentation method according to claim 1, wherein the cavity convolution module is a convolution module based on an Xception network structure, wherein the Xception network structure includes a planar cavity convolution layer and a channel cavity convolution The sampling rate of the planar hole convolutional layer is 1 or 3, and the sampling rate of the channel hole convolutional layer is 6.
5. The cartilage image segmentation method according to claim 1, wherein the pyramid hole pooling module includes a plurality of first convolution branches parallel to each other;

   The performing pooling processing on the first feature map by the pyramid hole pooling module to obtain the second feature map corresponding to the target cartilage image includes:

   Perform feature sampling on the first feature map through each of the first convolution branches to obtain the first sampling feature map, the second sampling feature map, the third sampling feature map, and the first feature map corresponding to the first feature map. Four sampling feature maps;

   Splicing the first sampling feature map, the second sampling feature map, the third sampling feature map, and the fourth sampling feature map to obtain a spliced feature map after splicing;

   Performing mean pooling processing on the stitched feature map to obtain a second feature map corresponding to the target cartilage image.
6. The cartilage image segmentation method according to claim 5, wherein the first convolution branch includes a first hole convolution unit, a second hole convolution unit, and a third hole convolution unit with different sampling rates, so The first hole convolution unit is connected to the second hole convolution unit, and the second hole convolution unit is connected to the third hole convolution unit.
7. The cartilage image segmentation method according to claim 1, wherein the attention mechanism target includes a plurality of second convolution branches parallel to each other;

   The performing weighting processing on the second feature map by the attention mechanism module to obtain the third feature map corresponding to the target cartilage image includes:

   Perform convolution processing on the second feature map through each of the second convolution branches to obtain the first convolution feature map, the second convolution feature map, and the third convolution corresponding to the second feature map. Feature map

   Performing transposition processing on the first convolution feature map, and performing matrix multiplication processing on the transposed feature map obtained by the transposition and the second convolution feature map to obtain a fifth feature map;

   Perform a normalization process on the fifth feature map, and perform a matrix multiplication process on the fifth feature map after the normalization process and the third convolution feature map to obtain a weight corresponding to the second feature map Coefficient matrix

   Perform weighting processing on the second feature map through the weighting coefficient matrix to obtain a third feature map corresponding to the target cartilage image.
8. The cartilage image segmentation method according to claim 1, wherein the third feature map is up-sampled by the fusion module, and the fourth feature map obtained by the sampling is compared with the first feature map. Performing fusion to obtain the cartilage segmentation result of the target cartilage image output by the cartilage image segmentation model includes:

   Performing bilinear upsampling on the third feature map by the fusion module to obtain the fourth feature map;

   Performing convolution processing on the first feature map through the third convolution branch of the fusion module to obtain a sixth feature map corresponding to the first feature map;

   Performing fusion processing on the fourth feature map and the sixth feature map to obtain a fused seventh feature map;

   Performing bilinear upsampling on the seventh feature map to obtain a cartilage segmentation result of the target cartilage image output by the cartilage image segmentation model.
9. The cartilage image segmentation method according to any one of claims 1 to 8, wherein the cartilage image segmentation model is obtained through training in the following steps:

   Acquiring a first preset number of first training cartilage images;

   The first training cartilage image is expanded using a preset expansion method to obtain a second preset number of second training cartilage images, where the second training cartilage image includes the first training cartilage image, and the second preset Set the number to be greater than the first preset number;

   Use the second training cartilage image and a preset loss function to train the cartilage image segmentation model, and the loss function is:

   

   Where, B is the number of training cartilage images, N is the number of pixels of each training cartilage image, p _ij is the probability that the j-th pixel of the i-th training cartilage image belongs to the cartilage, α=0.75, γ=2.
10. A cartilage image segmentation device, characterized in that it comprises:

    The target image acquisition module is used to acquire the target cartilage image to be segmented;

    The target image input module is used to input the target cartilage image to a preset cartilage image segmentation model, wherein the cartilage image segmentation model includes a cavity convolution module and a pyramid cavity pooling connected to the cavity convolution module Module, an attention mechanism module connected to the pyramid hole pooling module, and a fusion module connected to the hole convolution module and the attention mechanism module respectively;

    A feature extraction module, configured to perform feature extraction on the target cartilage image through the cavity convolution module to obtain a first feature map corresponding to the target cartilage image;

    The pooling weighting processing module is used to perform pooling processing on the first feature map through the pyramid hole pooling module to obtain a second feature map corresponding to the target cartilage image, and to perform the pooling process through the attention mechanism module Performing weighting processing on the second feature map to obtain a third feature map corresponding to the target cartilage image;

    The result output module is used to up-sample the third feature map through the fusion module, and fuse the fourth feature map obtained by sampling with the first feature map to obtain the output of the cartilage image segmentation model The cartilage segmentation result of the target cartilage image.
11. 10. The cartilage image segmentation device of claim 10, wherein the target image input module comprises:

    An original sampling distance obtaining unit, configured to obtain the original resolution and original sampling distance corresponding to the target cartilage image;

    A target sampling distance determining unit, configured to determine a target sampling distance according to the original resolution, the original sampling distance, and a preset target resolution;

    The image resampling unit is configured to resample the target cartilage image by using the target sampling distance, and input the resampled target cartilage image into a preset cartilage image segmentation model.
12. The cartilage image segmentation device according to claim 10, wherein the cavity convolution module is a convolution module based on the Xception network structure, wherein the Xception network structure includes a planar cavity convolution layer and a channel cavity convolution The sampling rate of the planar hole convolutional layer is 1 or 3, and the sampling rate of the channel hole convolutional layer is 6.
13. The cartilage image segmentation device according to any one of claims 10 to 12, wherein the cartilage image segmentation device comprises:

    A training image acquisition module for acquiring a first preset number of first training cartilage images;

    The training image expansion module is used to expand the first training cartilage image by using a preset expansion method to obtain a second preset number of second training cartilage images, where the second training cartilage image includes the first training cartilage Images, the second preset number is greater than the first preset number;

    The segmentation model training module is used to train the cartilage image segmentation model by using the second training cartilage image and a preset loss function, and the loss function is:

    

    Where, B is the number of training cartilage images, N is the number of pixels of each training cartilage image, p _ij is the probability that the j-th pixel of the i-th training cartilage image belongs to the cartilage, α=0.75, γ=2.
14. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program as claimed in claims 1 to 9. Any of the cartilage image segmentation methods.
15. A computer-readable storage medium storing a computer program, wherein the computer program implements the cartilage image segmentation method according to any one of claims 1 to 9 when the computer program is executed by a processor.

Images