WO2020199477A1 - 基于多模型融合的图像标注方法、装置、计算机设备及存储介质 - Google Patents

基于多模型融合的图像标注方法、装置、计算机设备及存储介质 Download PDF

Info

Publication number
WO2020199477A1
WO2020199477A1 PCT/CN2019/103139 CN2019103139W WO2020199477A1 WO 2020199477 A1 WO2020199477 A1 WO 2020199477A1 CN 2019103139 W CN2019103139 W CN 2019103139W WO 2020199477 A1 WO2020199477 A1 WO 2020199477A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
segmentation
classification
fusion
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/103139
Other languages
English (en)
French (fr)
Inventor
李风仪
南洋
侯晓帅
吕传峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Priority to SG11202008418PA priority Critical patent/SG11202008418PA/en
Publication of WO2020199477A1 publication Critical patent/WO2020199477A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • This application relates to the field of image processing technology, and in particular to an image annotation method, device, computer equipment, and storage medium based on multi-model fusion.
  • CNN deep convolutional neural network
  • the embodiments of the present application provide an image labeling method, device, computer equipment, and storage medium based on multi-model fusion, which can better realize the labeling of the region of interest in the image to be labelled, and the labeling has a higher overlap rate and precision.
  • this application provides an image annotation method based on multi-model fusion, and the method includes:
  • the contour of the fusion image is extracted, and the region of interest in the image to be marked is labeled according to the contour.
  • this application provides an image tagging device based on multi-model fusion, the device including:
  • the preprocessing module is used to obtain the image to be annotated and preprocess the image to be annotated to obtain several example images;
  • the two-classification module is used to input each of the instance images into a classification model based on the DenseNet network for two-class classification;
  • the classification splicing module is used to splice the binary classification results corresponding to the several instance images to obtain the classification result image;
  • the binarization segmentation module is used to input each of the instance images into a segmentation model based on the Vnet network and the FPN network for binarization segmentation;
  • a segmentation splicing module for splicing binarized segmentation results corresponding to the several instance images to obtain a segmentation result image
  • a fusion module configured to calculate a binarized fusion image according to the classification result image and the segmentation result image
  • the labeling module is used to extract the contour of the fusion image and label the region of interest in the image to be labelled according to the contour.
  • the present application provides a computer device that includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and when the computer is executed
  • the program implements the above-mentioned image annotation method based on multi-model fusion.
  • the present application provides a computer-readable storage medium that stores a computer program, and if the computer program is executed by a processor, the above-mentioned image annotation method based on multi-model fusion is realized.
  • This application discloses an image labeling method, device, device and storage medium based on multi-model fusion, through the DenseNet network-based classification model to perform two classifications of the instance image of the image to be labeled to obtain the classification result image, and through the Vnet network and The segmentation model of the FPN network performs binary segmentation on the instance image of the image to be labeled to obtain the segmentation result image, and then combines the classification result image and the segmentation result image to extract the contour of the region of interest in the image to be labeled; the coincidence rate of the label , The precision rate is high, which can reduce the false positive rate.
  • FIG. 1 is a schematic flowchart of an image labeling method based on multi-model fusion according to an embodiment of this application;
  • Fig. 2 is a schematic diagram of an application scenario of an image labeling method in an embodiment
  • Figure 3 is a schematic diagram of an image to be annotated
  • FIG. 4 is a schematic diagram of an example image obtained by processing the image to be annotated in FIG. 3;
  • Figure 5 is a schematic diagram of the structure of the DenseNet network
  • FIG. 6 is a schematic diagram of a sub-process of performing two classifications by a classification model in an embodiment
  • Figure 7 is a schematic diagram of the structure of dense blocks in the DenseNet network
  • FIG. 8 is a schematic diagram of a sub-process for performing two classifications by a classification model in an embodiment
  • Figure 9 is a schematic diagram of a classification result image
  • Figure 10 is a schematic diagram of the structure of the Vnet network in the segmentation model
  • Figure 11 is a schematic diagram of the structure of the FPN network
  • FIG. 12 is a schematic diagram of a sub-process of the segmentation model for binary segmentation
  • FIG. 13 is a schematic diagram of a sub-process of the FPN network outputting binarized segmentation results according to the results of sampling processing on the Vnet network;
  • FIG. 14 is a schematic diagram of binarized segmentation results corresponding to example images
  • Figure 15 is a schematic diagram of a segmentation result image
  • 16 is a schematic flowchart of an image labeling method based on multi-model fusion according to another embodiment of the application.
  • FIG. 17 is a schematic diagram of a sub-process of obtaining a binarized fused image by taking the intersection of the classification result image and the segmentation result image;
  • FIG. 19 is a schematic diagram of marking a region of interest in an image to be marked.
  • 20 is a schematic structural diagram of an image tagging device based on multi-model fusion provided by an embodiment of the application;
  • FIG. 21 is a schematic structural diagram of an image tagging device based on multi-model fusion according to another embodiment of the application.
  • FIG. 22 is a schematic structural diagram of a computer device provided by an embodiment of this application.
  • the embodiments of the present application provide an image annotation method, device, equipment and storage medium based on multi-model fusion.
  • the image labeling method based on multi-model fusion can be applied to a terminal or a server to implement labeling of a region of interest in an image.
  • the image annotation method based on multi-model fusion is used for servers, and certainly can be used for terminals, such as mobile phones, notebooks, and desktops.
  • terminals such as mobile phones, notebooks, and desktops.
  • the following embodiments will introduce in detail an image labeling method based on multi-model fusion applied to a server.
  • FIG. 1 is a schematic flowchart of an image labeling method based on multi-model fusion according to an embodiment of the present application.
  • the image annotation method based on multi-model fusion includes the following steps:
  • Step S110 Obtain an image to be annotated and preprocess the image to be annotated to obtain several example images.
  • the image to be annotated is an image stored locally by an apparatus for implementing an image annotation method based on multi-model fusion, an image obtained by the apparatus from the network, and an image obtained by the apparatus from an input device connected to it. Images, images acquired by the device from other electronic devices, etc.
  • the server obtains the image to be annotated from the terminal, both the server and the terminal are connected to the Internet, and the user uploads the image to be annotated to the server on the terminal;
  • the area of interest can also be sent back to the terminal.
  • the image to be annotated can be a color image, a grayscale image, etc.
  • some preprocessing can be performed on the image to be labeled, such as size adjustment, cropping, rotation, and/or image algorithm processing, etc.; the image algorithm processing includes: adjusting color temperature algorithm, adjusting exposure algorithm, adjusting contrast algorithm , High light recovery algorithm, low light compensation algorithm, white balance algorithm, sharpness adjustment algorithm, fog algorithm index, natural saturation adjustment algorithm, etc.
  • the image to be annotated acquired by the server is specifically a pathological image of skin basal cell carcinoma.
  • the size of the image to be annotated is relatively large, and the image to be annotated can be processed into a number of example images of preset sizes, namely, patch or instance, through preprocessing such as image cutting.
  • Figure 4 shows an example of two example images. It should be noted that the image to be annotated and the example image shown in FIG. 3 and FIG. 4 do not reflect the size and proportional relationship between the actual image to be annotated and the example image.
  • Step S120 Input each of the example images into a classification model based on the DenseNet network to perform two classifications.
  • the DenseNet network is a convolutional neural network with dense connections, which can alleviate the problem of gradient disappearance, strengthen feature propagation, encourage feature reuse, and greatly reduce the amount of parameters.
  • a DenseNet network includes multiple dense blocks.
  • the two adjacent dense blocks include transition blocks, namely transition layers.
  • step S120 inputs each of the example images into a classification model based on the DenseNet network for two classification, which specifically includes:
  • Step S121 If it is predicted that the example image has a region of interest based on the classification model of the DenseNet network, the binary classification result corresponding to the example image is the first value block.
  • the classification model based on the DenseNet network detects that a certain instance image has a region of interest, such as a cancerous area, the classification model divides this instance image into the first value block during classification.
  • the size of the first value block is the same as the size of the example image, that is, the length and the width are equal.
  • the first value is 255, and in a grayscale or black and white image, the first value block is a white block.
  • Step S122 If it is predicted that there is no region of interest in the example image based on the classification model of the DenseNet network, the binary classification result corresponding to the example image is the second value block.
  • the classification model based on the DenseNet network detects that an instance image does not have a region of interest, such as a cancerous area, the classification model divides this instance image into a second value block during classification.
  • the size of the second value block is the same as the size of the example image, that is, the length and the width are equal.
  • the second value is 0, and in a grayscale or black and white image, the second value block is a black block.
  • the region of interest is a cancerous region. If the example image is predicted to have an area of interest based on the classification model of the DenseNet network, the binary classification result corresponding to the example image is a positive block; if the classification model based on the DenseNet network predicts The example image does not have a region of interest, and the binary classification result corresponding to the example image is a negative block.
  • the binary classification result corresponding to the example image is the first value block. If the classification model based on the DenseNet network predicts that the proportion of the region of interest in the example image is not greater than the preset threshold, the binary classification result corresponding to the example image is the second value block.
  • Figure 7 shows the structure diagram of a dense block in the DenseNet network.
  • the input of each layer of the network is the connection of the output of all the previous layers, namely Concatenate, and the feature map learned by this layer will also be directly connected. Pass it to the following layers as input; thus, the DenseNet network can maximize the information flow between all layers in the network.
  • this structure can on the one hand alleviate the problem of gradient dissipation during the training process, because each layer will receive the gradient signals of all subsequent layers during the back propagation, so it will not appear as the network depth increases.
  • the problem that the gradient closer to the input layer is getting smaller and smaller, so that the features of the bottom layer are also reflected in the high layer; on the other hand, because a large number of features are reused, a large number of features can be generated using a small number of convolution kernels. The size is relatively small.
  • the input instance image is classified into two categories, including:
  • the size of the input example image is [512, 512, 3], that is, the length and width of the example image are both 512 pixels, and there are three channels, for example, the example image is an RGB three-channel image.
  • preprocessing the input example image includes the following steps:
  • Zero padding and convolution using the zero padding layer ZeroPadding2D to process the example image to the size of [518,518,3], and using the convolution layer Conv2D to convolve the zero padding image to output [256,256,64] Feature image.
  • the zero padding layer is mainly used to pad zeros on both sides of a single matrix, so as to control the size of the feature map during subsequent processing such as convolution.
  • the normalization layer is mainly used to normalize the data distribution state to a standard normal distribution, so as to speed up the convergence speed when training the model.
  • the position and number of the normalization layer are not limited.
  • the normalization layer can be set at the beginning of the DenseNet network, before the convolutional layer, or after the convolutional layer.
  • setting the normalization layer after the convolution layer can improve the convergence speed of the structured feature data obtained by convolution.
  • Non-linear processing using activation layer Activation to perform non-linear processing on batch-standardized feature images of [256, 256, 64].
  • the specific nonlinear processing can be implemented according to the nonlinear function ReLU.
  • the non-linear function ReLU is an element-level operation that can set all pixel values less than 0 in the feature map to zero.
  • the purpose of nonlinear processing is to introduce nonlinearity into the DenseNet network.
  • Zero padding and pooling using the zero padding layer ZeroPadding2D to process the non-linearly processed feature image of [256, 256, 64] into the size of [258, 258, 64], and using the pooling layer MaxPooling2D to treat the zero padding Feature image pooling, output [128, 128, 64] feature image.
  • this step of pooling uses maxpooling; maximum pooling has invariance, which includes translation invariance, rotation invariance, and scale ) Invariance; Maximum pooling can also reduce parameters and calculations while retaining the main features, prevent overfitting, and improve the generalization ability of the model.
  • the channel number channel of the feature image is increased according to the first dense block Dense Block.
  • the first dense block loop performs the following operations several times: Batch Normalization, Activation non-linear processing of the activation layer, Conv convolution processing of the convolutional layer, Batch Normalization of the batch normalization, activation non-linear processing of the activation layer, Convolution layer Conv convolution processing and connection layer Concatenate connection processing.
  • the connection layer Concatenate establishes a connection between the layer behind the connection layer and all the layers in front of the connection layer, which will reduce the number of network layers and facilitate network training. Each time the loop is completed, the image size remains unchanged, and the number of channels increases by 32.
  • the transition block cyclically executes the following operations several times: Batch Normalization, activation layer activation nonlinear processing, conv layer Conv convolution processing, pooling processing; among them, the convolution kernel of the convolution layer is 1 ⁇ 1, Pooling adopts Average Pooling, and the pooling window is 2 ⁇ 2; while reducing the dimensionality, it is more advantageous to pass the information to the next step for feature extraction, which can reduce the increase in the variance of the estimated value caused by the limited neighborhood size The error of feature extraction, more retain the background information of the image. Each time the loop is completed, the size of the feature image becomes half of the original, and the number of channels remains unchanged.
  • S14 According to the second dense block in the DenseNet network, increase the number of channels of the output characteristic image of the first transition block, and reduce the size of the output characteristic image of the second dense block according to the second transition block.
  • S15 According to the third dense block in the DenseNet network, increase the number of channels of the output feature image of the second transition block, and reduce the size of the output feature image of the third dense block according to the third transition block.
  • the output of the fourth dense block dense block4 in the DenseNet network is a feature image of [16, 16, 1024], that is, the length and width are 16 pixels, and there are 1024 channels.
  • batch normalization is performed on the feature image of [16, 16, 1024] output by the fourth dense block, and then GlobalAvergate Pooling is performed, and then the output is a one-dimensional feature vector with a length of 1024.
  • the instance image is predicted according to the one-dimensional feature vector with a length of 1024, and the result of the two classification is specifically the first value block or the second value block.
  • Step S130 Combine the two classification results corresponding to the several instance images to obtain a classification result image.
  • the image to be labeled as shown in FIG. 3 after preprocessing the image to be labeled as shown in FIG. 3 to obtain several instance images, predict each instance image based on the classification model of the DenseNet network, and stitch the prediction results of each instance image to obtain a classification result image, such as Shown in Figure 9.
  • the black area in FIG. 9 includes a number of second value blocks, corresponding to an example image where the area of interest does not exist; the white area in FIG. 9 includes a number of first value blocks, which corresponds to an example image where the area of interest exists.
  • Step S140 Input each of the example images into a segmentation model based on the Vnet network and the FPN network for binarization segmentation.
  • Vnet network is suitable for medical image segmentation of three-dimensional structure, and can realize end-to-end image semantic segmentation of 3D images.
  • Fig. 10 is a schematic structural diagram of a Vnet network.
  • the Vnet network includes a down-sampling (encoder) part and an up-sampling (decoder) part, and the whole can be regarded as a V-shape.
  • an FPN (feature pyramid networks) network includes a bottom-up line, a top-down line, and a lateral connection.
  • the area enlarged in the dashed box in Figure 11 is the horizontal connection.
  • the main function of the 1 ⁇ 1 convolution kernel is to reduce the number of convolution kernels, that is, to reduce the number of feature images (feature map) without changing the features The size of the image.
  • the bottom-up part of the FPN network is actually the normal forward propagation process of the neural network.
  • the feature map is calculated by the convolution kernel, it usually becomes smaller and smaller.
  • the top-down process is carried out by upsampling, and the more abstract and semantic high-level feature maps are up-sampled; then by horizontally connecting the feature to the previous layer feature, the up-sampling result is horizontally connected
  • the feature images of the same size generated from the bottom up are merged.
  • the feature images used in each layer of prediction (predict) are combined with features of different resolutions and different semantic strengths, which can complete the object of the corresponding resolution size. Check to ensure that each layer has appropriate resolution and strong semantic features.
  • the two horizontally connected features have the same spatial size, and the bottom layer is used to locate detailed information.
  • FPN mainly solves the multi-scale problem in object detection. Through simple network connection changes, the performance of small object detection is greatly improved without basically increasing the calculation amount of the original model.
  • the segmentation model based on Vnet network and FPN network uses the network structure of Vnet network combined with feature pyramid network to improve the effect of binarization segmentation.
  • the step S140 of inputting each of the example images into the segmentation model based on the Vnet network and the FPN network to perform binarization segmentation specifically includes:
  • Step S141 Input the example image into the Vnet network of the segmentation model.
  • the size of the input example image is [1024, 1024, 3], where 3 represents 3 channels such as RGB channels.
  • Step S142 Based on the Vnet network, sequentially perform non-linear activation, several down-sampling and several up-sampling processing on the example image.
  • the nonlinear activation processing includes performing batch normalization and convolution operations on the input example images, and then processing them through the PReLU activation layer; there are 16 filters for the convolution operation, and the size of the convolution kernel It is 5 ⁇ 5, and padding can be used to ensure that the feature image size remains unchanged.
  • the feature image after nonlinear activation processing is sequentially down-sampled (encoder) processed 5 times through a convolutional layer.
  • the first four downsamplings are batch normalization and PReLU activation, and then the input feature image of this downsampling and the corresponding position of the processed feature image are added, and the image is reduced after a dimensionality reduction operation ,
  • the reduction factor is 2. No dimensionality reduction operation is performed during the fifth downsampling.
  • the size of the feature image input for the first down-sampling is [1024, 1024, 16], and the size of the output feature image is [512, 512, 32];
  • the size of the feature image input for the second down-sampling is [512, 512, 32], the size of the output feature image is [256, 256, 64];
  • the size of the input feature image for the third downsampling is [256, 256, 64], the size of the output feature image is [128, 128, 128];
  • the size of the feature image input for the fourth downsampling is [128, 128, 128], and the size of the output feature image is [64, 64, 256];
  • the size of the input for the fifth downsampling The size of the feature image is [64, 64, 256], the size of the output feature image is [64, 64, 256], and no dimensionality reduction operation is performed.
  • up-sampling processing is performed on the feature image output after several down-sampling.
  • the inputs are merged together, and then batch normalization and PReLU activation are performed on the merged result, and then the output of the batch normalization and PReLU activation and the output of the previous processing are subjected to feature summation (element-wise sum); then through deconvolution ( The de-convolutional) layer performs upsampling operations.
  • the input of the first up-sampling is the input of the fourth down-sampling [128, 128, 128] and the output of the fifth down-sampling process [64, 64, 256], which is obtained by convolution [128, 128] , 128], the size of the output feature image is [256, 256, 64];
  • the input of the second upsampling is the input of the third downsampling [256, 256, 64] and the output of the first upsampling [256 ,256,64], the output size is [512,512,32];
  • the input of the third upsampling is the input of the second downsampling [512,512,32] and the output of the second upsampling [512 ,512,32], the output size is [1024,1024,16];
  • the input of the fourth upsampling process is the input of the first downsampling [1024,1024,16] and the output of the third upsampling [ 1024, 1024, 16], the output size is [102
  • Step S143 Based on the FPN network of the segmentation model, output the result of the binarization segmentation according to the result of each of the upsampling processing.
  • the feature image after upsampling is predicted according to the FPN network.
  • step S143 is based on the FPN network of the segmentation model, and outputs the result of the binarization segmentation according to the results of each of the upsampling processing, which specifically includes the following steps:
  • step S1431 a convolution operation is performed on the output of the first up-sampling process in the Vnet network.
  • the number of convolution kernels filter is 128, the size of the input feature image is [256, 256, 64], and the size of the output feature image is [256, 256, 128].
  • Step S1432 perform a convolution operation on the output of the second up-sampling process in the Vnet network.
  • the number filter of convolution kernels is 128.
  • Step S1433 Up-sampling processing is performed on the output convolution operation result of the first up-sampling process, and the result of the up-sampling processing is added to the output convolution operation result of the second up-sampling process.
  • the output of the second up-sampling process [512, 512, 32] after the convolution operation is [512, 512, 128]
  • the output of step S1431 [256, 256, 128] after the up-sampling operation
  • the size is [512, 512, 128]
  • the output size of the addition of the two is [512, 512, 128].
  • Step S1434 Perform a convolution operation on the output of the third upsampling process in the Vnet network.
  • the number filter of convolution kernels is 128.
  • Step S1435 Upsampling is performed on the output convolution operation result of the second upsampling process, and the result of the upsampling process is added to the output convolution operation result of the third upsampling process.
  • the output [1024, 1024, 16] of the third up-sampling process has a size of [1024, 1024, 128] after the convolution operation, and the output of step S1432 [512, 512, 128] after the up-sampling operation
  • the size is [1024, 1024, 128]
  • the sum of the two output sizes is [1024, 1024, 128].
  • Step S1436 Perform a convolution operation on the output of the fourth upsampling process in the Vnet network.
  • the number filter of convolution kernels is 128.
  • Step S1437 Add the result of the output convolution operation of the third upsampling process and the result of the output convolution operation of the fourth upsampling process.
  • the output of the fourth up-sampling process [1024, 1024, 16] has a size of [1024, 1024, 128] after the convolution operation, and the output of step S1434 has a size of [1024, 1024, 128].
  • the size of the output of the addition is [1024, 1024, 128].
  • Step S1438 Perform 1 ⁇ 1 convolution on the added result to obtain a binarized segmentation result image.
  • a 1 ⁇ 1 convolution is performed on the output [1024, 1024, 128] of step S1437 to obtain a binary segmentation result image.
  • the instance image of the Vnet network in the input segmentation model and the result of the binarization segmentation that is, the length and width of the segmentation result image are equal.
  • the segmentation model predicts that the example image has a region of interest, the region corresponding to the region of interest in the result of the binarization segmentation is a first value region. If the segmentation model predicts that the example image has a non-interest area, the area corresponding to the non-interest area in the result of the binarization segmentation is a second value area.
  • the region corresponding to the region of interest in the binarized segmentation result output by the segmentation model is the first value region;
  • the region corresponding to the region of interest is the second value region.
  • the first value is 255, and in a grayscale or black-and-white image, the first value area is a white area; the second value is 0, and in a grayscale or black-and-white image, the second value area is a black area.
  • the region of interest is a cancerous region. If the segmentation model based on the Vnet network and the FPN network predicts that the example image has a region of interest, then the binarized segmentation result output by the segmentation model matches the region of interest.
  • the corresponding region is a positive region, and the region that does not correspond to the region of interest in the binarized segmentation result output by the segmentation model is a negative region.
  • Step S150 stitching the binarized segmentation results corresponding to the several example images to obtain a segmentation result image.
  • each instance image is binarized based on the segmentation model of the Vnet network and FPN network, and the corresponding instance image in Figure 3 is binarized segmentation
  • the result is shown in Figure 14.
  • the black area in FIG. 14 corresponds to the second value area in the binarized segmentation result, indicating that it is not an area of interest;
  • the white area in FIG. 14 corresponds to the first value area in the binarized segmentation result, indicating that it is of interest area.
  • the binarized segmentation results corresponding to the several example images are stitched together to obtain the segmentation result image.
  • Step S160 Calculate a binarized fusion image according to the classification result image and the segmentation result image.
  • the prediction effect of the region of interest in the image to be labeled is further improved.
  • step S160 calculates a binarized fusion image according to the classification result image and the segmentation result image, which specifically includes:
  • Step S161 Intersecting the classification result image and the segmentation result image to obtain the binarized fusion image. Exemplarily, if a certain position in the classification result image corresponds to the region of interest, and the corresponding position in the segmentation result image also corresponds to the region of interest, the corresponding position in the binarized fused image also corresponds to the region of interest.
  • step S161 takes the intersection of the classification result image and the segmentation result image to obtain the binarized fusion image, which specifically includes:
  • Step S1611 if the pixel with the first value in the classification result image also has the first value in the segmentation result image, set the corresponding pixel in the fusion image to the first value.
  • the corresponding position in the binarized fusion image is also white.
  • Step S1612 If the pixel in the classification result image has the second value, set the corresponding pixel in the fusion image to the second value.
  • Step S1613 If the pixel in the segmentation result image has the second value, set the corresponding pixel in the fused image to the second value.
  • the corresponding position in the binarized fusion image is black.
  • a multi-model fusion-based Image annotation methods before the intersection of the classification result image and the segmentation result image is obtained in step S161 to obtain the binarized fusion image, a multi-model fusion-based Image annotation methods also include:
  • Step S101 Expand the first value block in the classification result image.
  • the step S161 to take the intersection of the classification result image and the segmentation result image to obtain the binarized fused image specifically includes:
  • step S1601 an intersection of the expanded classification result image and the segmentation result image is taken to obtain the binarized fusion image.
  • Dilation can smooth the edges of the first value block in the classification result image, thereby smoothing the boundary between the first value and the second value in the fused image obtained by taking the intersection.
  • Step S170 Extract the contour of the fused image, and label the region of interest in the image to be labeled according to the contour.
  • the boundary of the first value and the second value in the fused image that is, the range defined by the boundary of the intersection, can be used as the range of the region of interest obtained by prediction.
  • the fused image generated by prediction extract all the contour points of the region of interest in the fused image to form the contour of the fused image; then generate an .xml file based on the contour points, name it after the image to be annotated, and expand
  • the name is .ndpa, which is the annotation file corresponding to the image to be annotated, and the annotation file is displayed in the original image, as shown in FIG. 19, and the region of interest in the image to be annotated can be marked according to the outline.
  • displaying the annotation file corresponding to the image to be annotated in the original image may be executed by the server in FIG. 2 or may be executed by the terminal in FIG. 2.
  • each of the instance images is predicted according to the classification model and segmentation model, and the corresponding results of each instance image are calculated according to the binary classification result and the binary segmentation result of each instance image.
  • Binarized fused image then merge the fused images corresponding to each instance image to obtain the binarized fused image corresponding to the image to be labeled.
  • the classification model based on the DenseNet network and the segmentation model based on the Vnet network and the FPN network can be trained together.
  • the training samples include the image to be segmented that does not contain the region of interest, and the image to be segmented that is marked with the region of interest; the loss function uses weightedloss and focalloss, and the two focuses are different.
  • the combined loss function can integrate different factors into the model training Process, which can improve the overlap rate of the model Dice, the precision rate IoU; reduce the false positive rate fpr, that is, the percentage of the actual area of interest but is judged to be positive; and by using precision rather than accuracy to evaluate the training effect , Improve the suppression of false positives, and use the combined loss function to further optimize the training effect.
  • the effect of suppressing false positives is almost 100%, and the prediction speed of the model reaches 90 seconds per sheet, which significantly reduces the doctor’s Work load.
  • the Dice of the model is 0.88, and the IoU is 0.83.
  • coincidence rate Dice is as follows:
  • A represents the target window generated by the model, namely the segmentation result (SegResult), and B represents the original marked window, namely Ground Truth.
  • the precision IoU represents the intersection of the segmentation result and the marked window than the union of the segmentation result and the marked window.
  • the image labeling method based on multi-model fusion uses the classification model based on the DenseNet network to classify the instance image of the image to be labeled to obtain the classification result image, and the image to be labeled using the segmentation model based on the Vnet network and the FPN network
  • the example image of the image is binarized to obtain the segmentation result image, and then the classification result image and the segmentation result image are fused to extract the contour of the region of interest in the image to be annotated; the coincidence rate and precision of the annotation are high, and the Reduce the false positive rate.
  • FIG. 20 is a schematic structural diagram of an image tagging device based on multi-model fusion according to an embodiment of the present application.
  • the image tagging device based on multi-model fusion may be configured in a server to execute the aforementioned image tagging method based on multi-model fusion.
  • the image tagging device based on multi-model fusion includes:
  • the preprocessing module 110 is configured to obtain an image to be annotated and preprocess the image to be annotated to obtain several example images.
  • the two classification module 120 is configured to input each of the example images into a classification model based on the DenseNet network for two classifications.
  • the two-classification module 120 includes:
  • the first classification sub-module 121 is configured to predict that the example image has a region of interest based on the classification model of the DenseNet network, and the second classification result corresponding to the example image is the first value block.
  • the second classification sub-module 122 is configured to predict that there is no region of interest in the example image based on the classification model of the DenseNet network, and the binary classification result corresponding to the example image is the second value block.
  • the classification splicing module 130 is configured to splice the two classification results corresponding to the several instance images to obtain a classification result image.
  • the binarization segmentation module 140 is configured to input each of the example images into a segmentation model based on the Vnet network and the FPN network for binarization segmentation.
  • the segmentation model predicts that the example image has a region of interest, the region corresponding to the region of interest in the result of the binarization segmentation is a first value region; if the segmentation model predicts the The example image has a non-interest area, and the area corresponding to the non-interest area in the result of the binarization segmentation is the second value area.
  • the binarization segmentation module 140 includes:
  • the input sub-module 141 is used to input the example image into the Vnet network of the segmentation model.
  • the Vnet sub-module 142 is configured to sequentially perform non-linear activation, several down-sampling and several up-sampling processing on the example image based on the Vnet network.
  • the FPN sub-module 143 is configured to output a binarized segmentation result based on the FPN network of the segmentation model according to the result of each up-sampling process.
  • the segmentation and splicing module 150 is configured to splice the binarized segmentation results corresponding to the several instance images to obtain the segmentation result image.
  • the fusion module 160 is configured to calculate a binarized fusion image according to the classification result image and the segmentation result image.
  • the fusion module 160 is configured to intersect the classification result image and the segmentation result image to obtain the binarized fusion image.
  • the fusion module 160 includes:
  • the first fusion sub-module 161 is configured to set the corresponding pixel in the fusion image to the first value if the pixel having the first value in the classification result image also has the first value in the segmentation result image.
  • the second fusion sub-module 162 is configured to, if the pixel in the classification result image has the second value, set the corresponding pixel in the fusion image to the second value.
  • the third fusion sub-module 163 is configured to set the corresponding pixel in the fusion image to the second value if the pixel in the segmentation result image has the second value.
  • the labeling module 170 is configured to extract the contour of the fusion image and label the region of interest in the image to be labelled according to the contour.
  • the image tagging device based on multi-model fusion further includes:
  • the expansion module 101 is used to expand the first value block in the classification result image.
  • the fusion module 160 is specifically configured to take the intersection of the expanded classification result image and the segmentation result image to obtain the binarized fusion image.
  • the method and device of this application can be used in many general or special computing system environments or configurations.
  • the foregoing method and apparatus may be implemented in the form of a computer program, and the computer program may run on the computer device as shown in FIG. 22.
  • FIG. 22 is a schematic structural diagram of a computer device according to an embodiment of the present application.
  • the computer equipment can be a server or a terminal.
  • the computer device includes a processor, a memory, and a network interface connected through a system bus, where the memory may include a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium can store an operating system and a computer program.
  • the computer program includes program instructions, and when the program instructions are executed, the processor can execute any image annotation method based on multi-model fusion.
  • the processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
  • the internal memory provides an environment for the operation of the computer program in the non-volatile storage medium.
  • the processor can execute any image annotation method based on multi-model fusion.
  • the network interface is used for network communication, such as sending assigned tasks.
  • the structure of the computer device is only a block diagram of a part of the structure related to the solution of the application, and does not constitute a limitation on the computer device to which the solution of the application is applied.
  • the specific computer device may include More or fewer components are shown in the figure, or some components are combined, or have different component arrangements.
  • the processor may be a central processing unit (Central Processing Unit, CPU), the processor may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), and application specific integrated circuits (Application Specific Integrated Circuits). Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor.
  • the processor is used to run a computer program stored in a memory to implement the following steps:
  • the contour of the fusion image is extracted, and the region of interest in the image to be marked is labeled according to the contour.
  • processor when used to implement the two-class classification of each of the instance images into the classification model based on the DenseNet network, it is specifically used to implement:
  • the binary classification result corresponding to the example image is the first value block
  • the binary classification result corresponding to the example image is the second value block.
  • the processor when configured to input each of the example images into a segmentation model based on the Vnet network and the FPN network for binarization segmentation, it is specifically used to achieve:
  • the FPN network based on the segmentation model outputs the result of binarization segmentation according to the results of each of the upsampling processing.
  • the processor when configured to calculate a binarized fusion image based on the classification result image and the segmentation result image, it is specifically configured to realize:
  • the intersection of the classification result image and the segmentation result image is taken to obtain the binarized fusion image.
  • the processor is further configured to expand the first value block in the classification result image.
  • the processor is used to realize the intersection of the classification result image and the segmentation result image to obtain the binarized fusion image, it is specifically used to realize: the expanded classification result image and the segmentation result The images are intersected to obtain the binarized fused image.
  • a computer-readable storage medium stores a computer program
  • the computer program includes program instructions
  • the processor executes the program instructions to implement any item provided in the embodiments of this application based on Multi-model fusion image annotation method.
  • the computer-readable storage medium may be the internal storage unit of the computer device described in the foregoing embodiment, such as the hard disk or memory of the computer device.
  • the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SMC), or a secure digital (Secure Digital, SD) equipped on the computer device. ) Card, Flash Card, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

一种基于多模型融合的图像标注方法、装置、计算机设备及存储介质，方法包括：预处理待标注图像得到实例图像；将实例图像进行二分类；将二分类结果拼接以获取分类结果图像；将实例图像二值化分割；将分割结果拼接得到分割结果图像；根据分类结果图像和分割结果图像计算二值化的融合图像；提取融合图像的轮廓标注感兴趣区域。

Description

基于多模型融合的图像标注方法、装置、计算机设备及存储介质
本申请要求于2019年4月4日提交中国专利局、申请号为201910268876.7、发明名称为“基于多模型融合的图像标注方法、装置、计算机设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，尤其涉及一种基于多模型融合的图像标注方法、装置、计算机设备及存储介质。
背景技术
在一些应用场景中，需要在图像中分割出感兴趣的目标区域；但是由于有些图像，如皮肤病变图像的细粒度变化，让系统自动对图像各区域进行分类十分困难。虽然现有的深度卷积神经网络(CNN)常被用于分类各种不同细粒度的对象，并且在多项任务中表现良好，但是仍存在准确率低的问题。
发明内容
本申请实施例提供一种基于多模型融合的图像标注方法、装置、计算机设备及存储介质，能够较佳地实现对待标注图像中感兴趣区域的标注，标注的重合率、查准率较高。
第一方面，本申请提供了一种基于多模型融合的图像标注方法，所述方法包括：
获取待标注图像并对所述待标注图像进行预处理以得到若干实例图像；
将各所述实例图像输入基于DenseNet网络的分类模型进行二分类；
将所述若干实例图像对应的二分类结果拼接以获取分类结果图像；
将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割；
将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像；
根据所述分类结果图像和所述分割结果图像计算二值化的融合图像；
提取所述融合图像的轮廓，并根据所述轮廓标注所述待标注图像中的感兴趣区域。
第二方面，本申请提供了一种基于多模型融合的图像标注装置，所述装置包括：
预处理模块，用于获取待标注图像并对所述待标注图像进行预处理以得到若干实例图像；
二分类模块，用于将各所述实例图像输入基于DenseNet网络的分类模型进行二分类；
分类拼接模块，用于将所述若干实例图像对应的二分类结果拼接以获取分类结果图像；
二值化分割模块，用于将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割；
分割拼接模块，用于将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像；
融合模块，用于根据所述分类结果图像和所述分割结果图像计算二值化的融合图像；
标注模块，用于提取所述融合图像的轮廓，并根据所述轮廓标注所述待标注图像中的感兴趣区域。
第三方面，本申请提供了一种计算机设备，所述计算机设备包括存储器和处理器；所述存储器用于存储计算机程序；所述处理器，用于执行所述计算机程序并在执行所述计算机程序时实现上述的基于多模型融合的图像标注方法。
第四方面，本申请提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，若所述计算机程序被处理器执行，实现上述的基于多模型融合的图像标注方法。
本申请公开了一种基于多模型融合的图像标注方法、装置、设备及存储介质，通过基于DenseNet网络的分类模型对待标注图像的实例图像进行二分类以获取分类结果图像，以及通 过基于Vnet网络和FPN网络的分割模型对待标注图像的实例图像进行二值化分割以获取分割结果图像，进而将分类结果图像和分割结果图像融合提取用于标注待标注图像中感兴趣区域的轮廓；标注的重合率、查准率较高，可降低假阳性率。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请一实施例的基于多模型融合的图像标注方法的流程示意图；
图2为一实施例中图像标注方法的应用场景示意图；
图3为待标注图像的示意图；
图4为图3中待标注图像处理得到的实例图像的示意图；
图5为DenseNet网络的结构示意图；
图6为一实施例中的分类模型进行二分类的子流程示意图；
图7为DenseNet网络中稠密块的结构示意图；
图8为一实施例中的分类模型进行二分类的子流程示意图；
图9为分类结果图像的示意图；
图10为分割模型中Vnet网络的结构示意图；
图11为FPN网络的结构示意图；
图12为分割模型进行二值化分割的子流程示意图；
图13为FPN网络根据Vnet网络上采样处理的结果输出二值化分割结果的子流程示意图；
图14为实例图像对应的二值化分割结果的示意图；
图15为分割结果图像的示意图；
图16为本申请另一实施例的基于多模型融合的图像标注方法的流程示意图；
图17为分类结果图像与分割结果图像取交集得到二值化的融合图像的子流程示意图；
图18为本申请再一实施例的基于多模型融合的图像标注方法的流程示意图；
图19为标注待标注图像中的感兴趣区域的示意图；
图20为本申请一实施例提供的基于多模型融合的图像标注装置的结构示意图；
图21为本申请另一实施例提供的基于多模型融合的图像标注装置的结构示意图；
图22为本申请一实施例提供的一种计算机设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
附图中所示的流程图仅是示例说明，不是必须包括所有的内容和操作/步骤，也不是必须按所描述的顺序执行。例如，有的操作/步骤还可以分解、组合或部分合并，因此实际执行的顺序有可能根据实际情况改变。另外，虽然在装置示意图中进行了功能模块的划分，但是在某些情况下，可以以不同于装置示意图中的模块划分。
本申请的实施例提供了一种基于多模型融合的图像标注方法、装置、设备及存储介质。其中，该基于多模型融合的图像标注方法可以应用于终端或服务器中，以实现对图像中感兴趣区域的标注。
例如，基于多模型融合的图像标注方法用于服务器，当然可以用于终端，比如手机、笔记本、台式机等。但为了便于理解，以下实施例将以应用于服务器的基于多模型融合的图像 标注方法进行详细介绍。
下面结合附图，对本申请的一些实施方式作详细说明。在不冲突的情况下，下述的实施例及实施例中的特征可以相互组合。
请参阅图1，图1是本申请的实施例提供的一种基于多模型融合的图像标注方法的流程示意图。
如图1所示，基于多模型融合的图像标注方法包括以下步骤：
步骤S110、获取待标注图像并对所述待标注图像进行预处理以得到若干实例图像。
在一些可选的实施例中，待标注图像为用于实现基于多模型融合的图像标注方法的装置在本地存储的图像、该装置从网络获取的图像、该装置从与其连接的输入装置获取的图像、该装置从其他电子设备获取的图像等。
如图2所示，服务器从终端获取待标注图像，服务器与终端均连接于互联网，用户在终端将待标注图像上传至服务器；服务器通过基于多模型融合的图像标注方法标注待标注图像中的感兴趣区域，还可以将标注结果传回该终端。
待标注图像可以为彩色图像、灰度图像等。获取到待标注图像后可以对待标注图像进行一些预处理，如尺寸调整、裁剪处理、旋转处理和/或图像算法处理等等；其中图像算法处理包括：调整色温算法、调整曝光算法、调整对比度算法、高光恢复算法、低光补偿算法、白平衡算法、调整清晰度算法、雾化算法索引、调整自然饱和度算法等。
示例性的，如图3所示为服务器获取的待标注图像，具体为皮肤基底细胞癌病理图像。
在一些实施例中，待标注图像的尺寸较大，可以通过图像切割等预处理将待标注图像处理为若干预设尺寸的实例图像，即patch或instance。如图4所示为两个实例图像的示例。需要注意的是，图3和图4中示出的待标注图像和实例图像不体现实际待标注图像和实例图像之间的大小、比例关系。
步骤S120、将各所述实例图像输入基于DenseNet网络的分类模型进行二分类。
基于预先训练好的基于DenseNet网络，如DenseNet121网络的分类(classification)模型对各所述实例图像进行预测。
DenseNet网络是一种具有密集连接的卷积神经网络，可以缓解梯度消失的问题，加强特征传播，鼓励特征复用，可以极大的减少参数量。如图5所示，一个DenseNet网络包括多个稠密块Dense Block。相邻两个Dense Block的之间包括过渡块transition block，即transition layers。
在一些可选的实施例中，如图6所示，步骤S120将各所述实例图像输入基于DenseNet网络的分类模型进行二分类，具体包括：
步骤S121、若基于DenseNet网络的分类模型预测所述实例图像存在感兴趣区域，所述实例图像对应的二分类结果为第一值区块。
如果基于DenseNet网络的分类模型检测到某实例图像存在感兴趣区域，如癌变区域，则分类模型分类时将这一实例图像划分为第一值区块。示例性的，第一值区块的大小与实例图像的大小相同，即长、宽相等。
示例性的，第一值为255，在灰度或黑白图像中，第一值区块为白色区块。
步骤S122、若基于DenseNet网络的分类模型预测所述实例图像不存在感兴趣区域，所述实例图像对应的二分类结果为第二值区块。
如果基于DenseNet网络的分类模型检测到某实例图像不存在感兴趣区域，如癌变区域，则分类模型分类时将这一实例图像划分为第二值区块。示例性的，第二值区块的大小与实例图像的大小相同，即长、宽相等。
示例性的，第二值为0，在灰度或黑白图像中，第二值区块为黑色区块。
示例性的，感兴趣区域为癌变区域，若基于DenseNet网络的分类模型预测所述实例图像存在感兴趣区域，所述实例图像对应的二分类结果为阳性区块；若基于DenseNet网络的分类 模型预测所述实例图像不存在感兴趣区域，所述实例图像对应的二分类结果为阴性区块。
在另一些实施例中，若基于DenseNet网络的分类模型预测所述实例图像中感兴趣区域的比例大于预设阈值，所述实例图像对应的二分类结果为第一值区块。若基于DenseNet网络的分类模型预测所述实例图像中感兴趣区域的比例不大于预设阈值，所述实例图像对应的二分类结果为第二值区块。
如图7所示为DenseNet网络中一个稠密块Dense Block的结构示意图。在各稠密块Dense Block中的任何两层之间都有直接的连接，即网络每一层的输入都是前面所有层输出的连接，即Concatenate，而该层所学习的特征图也会被直接传给其后面的各层作为输入；从而DenseNet网络可以最大化网络中所有层之间的信息流。
在一些实施例中，这种结构一方面可以减轻训练过程中梯度消散的问题，因为在反传时每一层都会接受其后所有层的梯度信号，所以不会出现随着网络深度的增加，靠近输入层的梯度越来越小的问题，使得底层的特征在高层中也有反映；另一方面由于大量的特征被复用，使得使用少量的卷积核就可以生成大量的特征，最终模型的尺寸比较小。
如图8所示，基于DenseNet网络的分类模型对输入的实例图像进行二分类，具体包括：
S11、对输入的实例图像进行预处理，以获取特征图像。
示例性的，输入的实例图像的大小为[512，512，3]，即实例图像的长和宽均为512个像素，有三个通道，例如实例图像为RGB三通道的图像。
具体的，对输入的实例图像进行预处理，包括以下步骤：
补零与卷积，利用补零层ZeroPadding2D将实例图像处理为[518，518，3]的大小，以及利用卷积层Conv2D对补零后的图像卷积，输出[256，256，64]的特征图像。补零层主要用于在单个矩阵的两边补零，以便后续卷积等处理时控制特征图的大小。
批标准化，利用正规化层Batch Normalization对[256，256，64]的特征图像进行批标准化，输出为批标准化了的[256，256，64]的特征图像。正规化层主要用于将数据分布状态归一化为标准正态分布，以利于在训练模型的时候加快收敛速度。具体的，对正规化层设置的位置以及数量不做限定，例如正规化层可以设置于DenseNet网络的开端、位于卷积层之前或者位于卷积层之后。示例性的，将正规化层设置于卷积层之后，可以提升卷积得到的结构化特征数据的收敛速度。
非线性处理，利用激活层Activation对批标准化了的[256，256，64]的特征图像进行非线性处理。具体的非线性处理可以根据非线性函数ReLU实现。非线性函数ReLU是一个元素级别的操作，可以将特征图中的所有小于0的像素值设置为零。非线性处理的目的是在DenseNet网络中引入非线性。
补零与池化，利用补零层ZeroPadding2D将非线性处理后的[256，256，64]的特征图像处理为[258，258，64]的大小，以及利用池化层MaxPooling2D对补零后的特征图像池化，输出[128，128，64]的特征图像。
具体的，这一步骤的池化采用最大值池化maxpooling；最大值池化具有不变(invariance)性，这种不变性包括平移(translation)不变性、旋转(rotation)不变性和尺度(scale)不变性；最大值池化还可以在保留主要特征的同时减少参数和计算量，防止过拟合，提高模型泛化能力。
S12、根据DenseNet网络中的第一个稠密块增加预处理后的特征图像的通道数。
具体的，如图5所示，根据第一个稠密块Dense Block增加特征图像的通道数channel。如图7所示，第一个稠密块循环执行若干次以下操作：批标准化Batch Normalization、激活层Activation非线性处理、卷积层Conv卷积处理、批标准化Batch Normalization、激活层Activation非线性处理、卷积层Conv卷积处理以及连接层Concatenate连接处理。其中连接层Concatenate将连接层后面一层和连接层前面所有层都建立一个连接，这样会减少网络网络层数，有利于网络的训练。每次循环完成，图像大小不变，通道数channel数量增加32。
S13、根据DenseNet网络中的第一个过渡块缩小所述第一个稠密块输出的特征图像的大小。
具体的，过渡块循环执行若干次以下操作：批标准化Batch Normalization、激活层Activation非线性处理、卷积层Conv卷积处理、池化处理；其中，卷积层的卷积核为1×1，池化采用均值池化AveragePooling，池化窗口为2×2；在减少维度的同时，更有利信息传递到下一步骤进行特征提取，能减小邻域大小受限造成的估计值方差增大引起的特征提取的误差，更多的保留图像的背景信息。每次循环完成，特征图像的大小变为原来的一半，通道数channel数量不变。
S14、根据DenseNet网络中的第二个稠密块增加所述第一个过渡块输出特征图像的通道数，以及根据第二个过渡块缩小所述第二个稠密块输出特征图像的大小。
S15、根据DenseNet网络中的第三个稠密块增加所述第二个过渡块输出特征图像的通道数，以及根据第三个过渡块缩小所述第三个稠密块输出特征图像的大小。
S16、根据DenseNet网络中的第四个稠密块增加所述第三个过渡块输出特征图像的通道数。
示例性的，DenseNet网络中的第四个稠密块dense block4的输出为[16，16，1024]的特征图像，即长和宽均为16个像素，有1024个通道。
S17、对所述第四个稠密块输出的特征图像进行批标准化以及全局平均池化。
示例性的，对所述第四个稠密块输出的[16，16，1024]的特征图像进行批标准化Batch Normalization，然后进行全局平均池化GlobalAvergatePooling，之后输出为长度为1024的一维特征向量。
S18、根据所述全局平均池化得到的特征向量对所述实例图像进行二分类。
示例性的，根据所述长度为1024的一维特征向量对所述实例图像进行预测，二分类的结果具体为第一值区块或第二值区块。
步骤S130、将所述若干实例图像对应的二分类结果拼接以获取分类结果图像。
示例性的，图3所示的待标注图像预处理得到若干实例图像后，基于DenseNet网络的分类模型对各实例图像进行预测，各实例图像的预测结果拼接后得到分类(classification)结果图像，如图9所示。图9中的黑色区域包括若干第二值区块，对应于不存在感兴趣区域的实例图像；图9中的白色区域包括若干第一值区块，对应于存在感兴趣区域的实例图像。
步骤S140、将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割。
Vnet网络适用于三维结构的医学影像分割，能够实现3D图像端到端的图像语义分割。如图10所示为一Vnet网络的结构示意图，Vnet网络包括下采样(encoder)部分和上采样(decoder)部分，整体可看作是一个V形。
如图11所示，FPN(feature pyramid networks，特征金字塔)网络包括一个自底向上的线路，一个自顶向下的线路以及横向连接(lateral connection)。图11虚线框中放大的区域就是横向连接，其中1×1的卷积核的主要作用是减少卷积核的个数，也就是减少了特征图像(feature map)的个数，并不改变特征图像的尺寸大小。
具体的，FPN网络自底向上的部分其实就是神经网络普通的正向传播过程，特征图经过卷积核计算，通常会越变越小。自顶向下的过程采用上采样(up sampling)进行，把更抽象、语义更强的高层特征图进行上采样；然后通过把该特征横向连接至前一层特征，横向连接将上采样的结果和自底向上生成的相同大小的特征图像进行融合(merge)，每一层预测(predict)所用的特征图像都融合了不同分辨率、不同语义强度的特征，可以完成对应分辨率大小的物体进行检测，保证每一层都有合适的分辨率以及强语义特征。横向连接的两层特征在空间尺寸上相同，利用底层定位细节信息。
FPN主要解决的是物体检测中的多尺度问题，通过简单的网络连接改变，在基本不增加 原有模型计算量的情况下，大幅度提升了小物体检测的性能。基于Vnet网络和FPN网络的分割(segmentation)模型利用Vnet网络结合特征金字塔网络的网络结构提升二值化分割的效果。
在一些可选的实施例中，如图12所示，步骤S140将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割具体包括：
步骤S141、将所述实例图像输入所述分割模型的Vnet网络。
示例性的，输入实例图像的大小为[1024，1024，3]，其中3代表3个通道如RGB通道。
步骤S142、基于所述Vnet网络对所述实例图像依次进行非线性激活、若干次下采样和若干次上采样处理。
示例性的，其中，非线性激活处理包括对输入的实例图像经进行批标准化Bach Normalization、卷积操作，然后再经过PReLU激活层处理；卷积操作的过滤器filter为16个，卷积核尺寸为5×5，可以通过填充padding保证特征图像大小不变。
示例性的，通过卷积(convolutional)层对非线性激活处理后的特征图像依次进行5次下采样(encoder)处理。其中，前四次下采样都先进行批标准化(Batch Normalization)和PReLU激活，再将本次下采样的输入特征图像和经过处理的特征图像对应位置相加后，经过一次降维操作，缩小图像，缩小因子为2。第五次下采样的时候不进行降维操作。具体的，第一次下采样输入的特征图像的大小为[1024，1024，16]，输出的特征图像的大小为[512，512，32]；第二次下采样输入的特征图像的大小为[512，512，32]，输出的特征图像的大小为[256，256，64]；第三次下采样输入的特征图像的大小为[256，256，64]，输出的特征图像的大小为[128，128，128]；第四次下采样输入的特征图像的大小为[128，128，128]，输出的特征图像的大小为[64，64，256]；第五次下采样输入的特征图像的大小为[64，64，256]，输出的特征图像的大小为[64，64，256]，不进行降维操作。
示例性的，对若干次下采样后输出的特征图像进行四次上采样(decoder)处理。每次上采样过程的输入有两个：通过细粒度特征转发(fine-grained features forwarding)传来的相应下采样过程的输入和前一处理过程的输出；先经过连接(concatenate)层将两个输入合并到一起，然后对合并结果进行批标准化和PReLU激活，然后再将批标准化和PReLU激活的输出和前一处理过程的输出进行特征求和(element-wise sum)；之后通过去卷积(de-convolutional)层进行上采样操作。
具体的，第一次上采样的输入为第四次下采样的输入[128，128，128]和第五次下采样过程的输出[64，64，256]经过卷积得到的[128，128，128]，输出特征图像的大小为[256，256，64]；第二次上采样的输入为第三次下采样的输入[256，256，64]和第一次上采样的输出[256，256，64]，输出的大小为[512，512，32]；第三次上采样的输入为第二次下采样的输入[512，512，32]和第二次上采样的输出[512，512，32]，输出的大小为[1024，1024，16]；第四次上采样过程的输入为第一次下采样的输入[1024，1024，16]和第三次上采样的输出[1024，1024，16]，输出的大小为[1024，1024，16]。
步骤S143、基于所述分割模型的FPN网络，根据各所述上采样处理的结果输出二值化分割的结果。
具体的，根据FPN网络对上采样处理后的特征图像进行预测。
示例性的，如图13所示，步骤S143基于所述分割模型的FPN网络，根据各所述上采样处理的结果输出二值化分割的结果，具体包括以下步骤：
步骤S1431、对Vnet网络中第一次上采样过程的输出做卷积操作。
示例性的，卷积核数目filter为128，输入特征图像的大小为[256，256，64]，输出特征图像的大小为[256，256，128]。
步骤S1432、对Vnet网络中第二次上采样过程的输出做卷积操作。
示例性的，卷积核数目filter为128。
步骤S1433、将对第一次上采样过程的输出卷积操作的结果进行上采样处理，并将上采样处理的结果与对第二次上采样过程的输出卷积操作的结果进行相加。
示例性的，第二次上采样过程的输出[512，512，32]卷积操作后的大小为[512，512，128]，步骤S1431的输出[256，256，128]上采样操作后的大小为[512，512，128]，二者相加的输出的大小为[512，512，128]。
步骤S1434、对Vnet网络中第三次上采样过程的输出做卷积操作。
示例性的，卷积核数目filter为128。
步骤S1435、将对第二次上采样过程的输出卷积操作的结果进行上采样处理，并将上采样处理的结果与对第三次上采样过程的输出卷积操作的结果进行相加。
示例性的，第三次上采样过程的输出[1024，1024，16]卷积操作后的大小为[1024，1024，128]，步骤S1432的输出[512，512，128]上采样操作后的大小为[1024，1024，128]，二者相加输出的大小为[1024，1024，128]。
步骤S1436、对Vnet网络中第四次上采样过程的输出做卷积操作。
示例性的，卷积核数目filter为128。
步骤S1437、将对第三次上采样过程的输出卷积操作的结果与对第四次上采样过程的输出卷积操作的结果进行相加。
示例性的，第四次上采样过程的输出[1024，1024，16]卷积操作后的大小为[1024，1024，128]，步骤S1434的输出的大小为[1024，1024，128]，二者相加的输出的大小为[1024，1024，128]。
步骤S1438、对相加的结果进行1×1卷积，以获取二值化的分割结果图像。
示例性的，对步骤S1437的输出[1024，1024，128]做1×1卷积，得到二值化的分割(segmentation)结果图像。
在一些实施例中，输入分割模型中Vnet网络的实例图像与所述二值化分割的结果，即分割结果图像的长、宽相等。
在一些实施例中，若所述分割模型预测所述实例图像存在感兴趣区域，所述二值化分割的结果中与所述感兴趣区域对应的区域为第一值区域。若所述分割模型预测所述实例图像存在非感兴趣区域，所述二值化分割的结果中与所述非感兴趣区域对应的区域为第二值区域。
示例性的，如果分割模型检测到某实例图像存在感兴趣区域，如癌变区域，则分割模型输出的二值化分割结果中与所述感兴趣区域对应的区域为第一值区域；不与所述感兴趣区域对应的区域为第二值区域。
示例性的，第一值为255，在灰度或黑白图像中，第一值区域为白色区域；第二值为0，在灰度或黑白图像中，第二值区域为黑色区域。
示例性的，感兴趣区域为癌变区域，若基于Vnet网络和FPN网络的分割模型预测所述实例图像存在感兴趣区域，则所述分割模型输出的二值化分割结果中与所述感兴趣区域对应的区域为阳性区域，所述分割模型输出的二值化分割结果中不与所述感兴趣区域对应的区域为阴性区域。
步骤S150、将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像。
示例性的，图3所示的待标注图像预处理得到若干实例图像后，基于Vnet网络和FPN网络的分割模型对各实例图像进行二值化分割，图3中相应实例图像的二值化分割结果如图14所示。图14中的黑色区域对应于二值化分割结果中的第二值区域，表示不是感兴趣区域；图14中的白色区域对应于二值化分割结果中的第一值区域，表示是感兴趣区域。
如图15所示为将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像。
步骤S160、根据所述分类结果图像和所述分割结果图像计算二值化的融合图像。
通过综合基于DenseNet网络的分类模型预测的分类结果和基于Vnet网络和FPN网络的分割模型预测的分割结果，进一步提升对待标注图像中感兴趣区域预测的效果。
在一些可选的实施例中，如图16所示，步骤S160根据所述分类结果图像和所述分割结果图像计算二值化的融合图像，具体包括：
步骤S161、将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。示例性的，如果分类结果图像中某位置对应于感兴趣区域，且分割结果图像中的相应位置也对应于感兴趣区域，则二值化的融合图像中的相应位置也对应于感兴趣区域。
示例性的，如图17所示，步骤S161将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像，具体包括：
步骤S1611、若所述分类结果图像中为第一值的像素在所述分割结果图像中也为第一值，将所述融合图像中相应的像素设为第一值。
示例性的，如果分类结果图像中某位置为白色，且分割结果图像中的相应位置也为白色，则二值化的融合图像中的相应位置也是白色。
步骤S1612、若所述分类结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值。
步骤S1613、若所述分割结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值。
示例性的，如果分类结果图像中某位置为黑色，或者分割结果图像中的相应位置为黑色，则二值化的融合图像中的相应位置就是黑色。
在一些可选的实施例中，如图18所示，在步骤S161将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像之前，基于多模型融合的图像标注方法还包括：
步骤S101、将所述分类结果图像中的第一值区块进行膨胀。
在本实施例中，步骤S161将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像，具体包括：
步骤S1601、将所述膨胀后的分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
膨胀可以让分类结果图像中第一值区块的边缘平滑，从而可以使得通过取交集得到的融合图像中第一值、第二值的边界平滑。
步骤S170、提取所述融合图像的轮廓，并根据所述轮廓标注所述待标注图像中的感兴趣区域。
融合图像中第一值、第二值的边界，即交集的边界限定的范围可以作为预测得到的感兴趣区域的范围。
示例性的，具体的，根据预测生成的融合图像，提取融合图像中感兴趣区域的所有轮廓点以构成融合图像的轮廓；然后基于轮廓点生成.xml文件，以待标注图像的名字命名，扩展名为.ndpa，即为待标注图像对应的标注文件，将该标注文件在原图像进行显示，如图19所示，就可以根据所述轮廓标注所述待标注图像中的感兴趣区域。
在一些实施例中，将待标注图像对应的标注文件在原图像进行显示可以由图2中的服务器执行，也可以由图2中的终端执行。
在另一些可行的实施例中，根据所述分类模型、分割模型对各所述实例图像进行预测，并分别根据各所述实例图像的二分类结果、二值化分割结果计算各实例图像对应的二值化的融合图像；然后将各实例图像对应的融合图像进行拼接后得到待标注图像对应的二值化的融合图像。
在一些可行的实施例中，基于DenseNet网络的分类模型和基于Vnet网络和FPN网络的分割模型可以一起训练。训练的样本包括不包含感兴趣区域的待分割图像、标记了感兴趣区域的待分割图像；损失函数采用weightedloss和focalloss，两者侧重点不一样，组合损失函数可以将不同的因素融合进模型训练过程，从而可以提升模型的重合率Dice、查准率IoU；降低假阳性率fpr，即实际无感兴趣区域但被判为阳性的百分比；而且通过使用精密度precision 而非准确度accuracy评价训练效果，提升了对假阳性的抑制，并利用组合损失函数进一步优化训练效果。
基于DenseNet网络的分类模型和基于Vnet网络和FPN网络的分割模型的预测结果融合后，几近达到百分百抑制假阳性的效果，且模型的预测速度达到90秒/每张，显著减少医生的工作负担。具体的，模型的Dice为0.88，IoU为0.83。
其中，重合率Dice的定义如下：
Figure PCTCN2019103139-appb-000001
A表示模型产生的目标窗口，即分割结果(SegResult)，B表示原来的标记窗口，即Ground Truth。
查准率IoU表示分割结果与标记窗口的交集比上分割结果与标记窗口的并集。
上述实施例提供的基于多模型融合的图像标注方法，通过基于DenseNet网络的分类模型对待标注图像的实例图像进行二分类以获取分类结果图像，以及通过基于Vnet网络和FPN网络的分割模型对待标注图像的实例图像进行二值化分割以获取分割结果图像，进而将分类结果图像和分割结果图像融合提取用于标注待标注图像中感兴趣区域的轮廓；标注的重合率、查准率较高，可降低假阳性率。
请参阅图20，图20是本申请一实施例提供的一种基于多模型融合的图像标注装置的结构示意图。示例性的，该基于多模型融合的图像标注装置可以配置于服务器中，用于执行前述的基于多模型融合的图像标注方法。
如图20所示，该基于多模型融合的图像标注装置，包括：
预处理模块110，用于获取待标注图像并对所述待标注图像进行预处理以得到若干实例图像。
二分类模块120，用于将各所述实例图像输入基于DenseNet网络的分类模型进行二分类。
具体的，如图21所示，二分类模块120包括：
第一分类子模块121、用于若基于DenseNet网络的分类模型预测所述实例图像存在感兴趣区域，所述实例图像对应的二分类结果为第一值区块。
第二分类子模块122、用于若基于DenseNet网络的分类模型预测所述实例图像不存在感兴趣区域，所述实例图像对应的二分类结果为第二值区块。
分类拼接模块130，用于将所述若干实例图像对应的二分类结果拼接以获取分类结果图像。
二值化分割模块140，用于将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割。
具体的，若所述分割模型预测所述实例图像存在感兴趣区域，所述二值化分割的结果中与所述感兴趣区域对应的区域为第一值区域；若所述分割模型预测所述实例图像存在非感兴趣区域，所述二值化分割的结果中与所述非感兴趣区域对应的区域为第二值区域。
具体的，如图21所示，二值化分割模块140包括：
输入子模块141、用于将所述实例图像输入所述分割模型的Vnet网络。
Vnet子模块142、用于基于所述Vnet网络对所述实例图像依次进行非线性激活、若干次下采样和若干次上采样处理。
FPN子模块143、用于基于所述分割模型的FPN网络，根据各所述上采样处理的结果输出二值化分割的结果。
分割拼接模块150，用于将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像。
融合模块160，用于根据所述分类结果图像和所述分割结果图像计算二值化的融合图像。
具体的，融合模块160用于将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
具体的，如图21所示，融合模块160包括：
第一融合子模块161、用于若所述分类结果图像中为第一值的像素在所述分割结果图像中也为第一值，将所述融合图像中相应的像素设为第一值。
第二融合子模块162、用于若所述分类结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值。
第三融合子模块163、用于若所述分割结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值。
标注模块170，用于提取所述融合图像的轮廓，并根据所述轮廓标注所述待标注图像中的感兴趣区域。
在一些实施例中，如图21所示，基于多模型融合的图像标注装置还包括：
膨胀模块101，用于将所述分类结果图像中的第一值区块进行膨胀。
融合模块160具体用于将所述膨胀后的分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
需要说明的是，所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，上述描述的装置和各模块、单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
本申请的方法、装置可用于众多通用或专用的计算系统环境或配置中。例如：个人计算机、服务器计算机、手持设备或便携式设备、平板型设备、多处理器系统、基于微处理器的系统、可编程的消费电子设备、网络PC、小型计算机、大型计算机、包括以上任何系统或设备的分布式计算环境等等。
示例性的，上述的方法、装置可以实现为一种计算机程序的形式，该计算机程序可以在如图22所示的计算机设备上运行。
请参阅图22，图22是本申请实施例提供的一种计算机设备的结构示意图。该计算机设备可以是服务器或终端。
参阅图22，该计算机设备包括通过系统总线连接的处理器、存储器和网络接口，其中，存储器可以包括非易失性存储介质和内存储器。
非易失性存储介质可存储操作系统和计算机程序。该计算机程序包括程序指令，该程序指令被执行时，可使得处理器执行任意一种基于多模型融合的图像标注方法。
处理器用于提供计算和控制能力，支撑整个计算机设备的运行。
内存储器为非易失性存储介质中的计算机程序的运行提供环境，该计算机程序被处理器执行时，可使得处理器执行任意一种基于多模型融合的图像标注方法。
该网络接口用于进行网络通信，如发送分配的任务等。本领域技术人员可以理解，该计算机设备的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
应当理解的是，处理器可以是中央处理单元(Central Processing Unit，CPU)，该处理器还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。其中，通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
其中，在一个实施例中，所述处理器用于运行存储在存储器中的计算机程序，以实现如下步骤：
获取待标注图像并对所述待标注图像进行预处理以得到若干实例图像；
将各所述实例图像输入基于DenseNet网络的分类模型进行二分类；
将所述若干实例图像对应的二分类结果拼接以获取分类结果图像；
将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割；
将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像；
根据所述分类结果图像和所述分割结果图像计算二值化的融合图像；
提取所述融合图像的轮廓，并根据所述轮廓标注所述待标注图像中的感兴趣区域。
示例性的，处理器用于实现将各所述实例图像输入基于DenseNet网络的分类模型进行二分类时，具体用于实现：
若基于DenseNet网络的分类模型预测所述实例图像存在感兴趣区域，所述实例图像对应的二分类结果为第一值区块；
若基于DenseNet网络的分类模型预测所述实例图像不存在感兴趣区域，所述实例图像对应的二分类结果为第二值区块。
示例性的，处理器用于实现将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割时，具体用于实现：
将所述实例图像输入所述分割模型的Vnet网络；
基于所述Vnet网络对所述实例图像依次进行非线性激活、若干次下采样和若干次上采样处理；
基于所述分割模型的FPN网络，根据各所述上采样处理的结果输出二值化分割的结果。
示例性的，处理器用于实现根据所述分类结果图像和所述分割结果图像计算二值化的融合图像时，具体用于实现：
将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
示例性的，处理器还用于实现将所述分类结果图像中的第一值区块进行膨胀。处理器用于实现将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像时，具体用于实现：将所述膨胀后的分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
通过以上的实施方式的描述可知，本领域的技术人员可以清楚地了解到本申请可借助软件加必需的通用硬件平台的方式来实现。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在存储介质中，如ROM/RAM、磁碟、光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例或者实施例的某些部分所述的方法，如：
一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序中包括程序指令，所述处理器执行所述程序指令，实现本申请实施例提供的任一项基于多模型融合的图像标注方法。
其中，所述计算机可读存储介质可以是前述实施例所述的计算机设备的内部存储单元，例如所述计算机设备的硬盘或内存。所述计算机可读存储介质也可以是所述计算机设备的外部存储设备，例如所述计算机设备上配备的插接式硬盘，智能存储卡(Smart Media Card，SMC)，安全数字(Secure Digital，SD)卡，闪存卡(Flash Card)等。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。

Claims (20)

  1. 一种基于多模型融合的图像标注方法，包括：
    获取待标注图像并对所述待标注图像进行预处理以得到若干实例图像；
    将各所述实例图像输入基于DenseNet网络的分类模型进行二分类；
    将所述若干实例图像对应的二分类结果拼接以获取分类结果图像；
    将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割；
    将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像；
    根据所述分类结果图像和所述分割结果图像计算二值化的融合图像；
    提取所述融合图像的轮廓，并根据所述轮廓标注所述待标注图像中的感兴趣区域。
  2. 如权利要求1所述的基于多模型融合的图像标注方法，其中，所述将各所述实例图像输入基于DenseNet网络的分类模型进行二分类，具体包括：
    若基于DenseNet网络的分类模型预测所述实例图像存在感兴趣区域，所述实例图像对应的二分类结果为第一值区块；
    若基于DenseNet网络的分类模型预测所述实例图像不存在感兴趣区域，所述实例图像对应的二分类结果为第二值区块。
  3. 如权利要求2所述的基于多模型融合的图像标注方法，其中，所述将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割，具体包括：
    将所述实例图像输入所述分割模型的Vnet网络；
    基于所述Vnet网络对所述实例图像依次进行非线性激活、若干次下采样和若干次上采样处理；
    基于所述分割模型的FPN网络，根据各所述上采样处理的结果输出二值化分割的结果。
  4. 如权利要求3所述的基于多模型融合的图像标注方法，其中，若所述分割模型预测所述实例图像存在感兴趣区域，所述二值化分割的结果中与所述感兴趣区域对应的区域为第一值区域；
    若所述分割模型预测所述实例图像存在非感兴趣区域，所述二值化分割的结果中与所述非感兴趣区域对应的区域为第二值区域。
  5. 如权利要求4所述的基于多模型融合的图像标注方法，其中，所述根据所述分类结果图像和所述分割结果图像计算二值化的融合图像，具体包括：
    将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
  6. 如权利要求5所述的基于多模型融合的图像标注方法，其中，所述将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像，具体包括：
    若所述分类结果图像中为第一值的像素在所述分割结果图像中也为第一值，将所述融合图像中相应的像素设为第一值；
    若所述分类结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值；
    若所述分割结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值。
  7. 如权利要求5所述的基于多模型融合的图像标注方法，其中，所述将所述分类结果图像与所述分割结果图像取交集之前，还包括：
    将所述分类结果图像中的第一值区块进行膨胀；
    所述将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像，具体包括：
    将所述膨胀后的分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
  8. 一种基于多模型融合的图像标注装置，包括：
    预处理模块，用于获取待标注图像并对所述待标注图像进行预处理以得到若干实例图像；
    二分类模块，用于将各所述实例图像输入基于DenseNet网络的分类模型进行二分类；
    分类拼接模块，用于将所述若干实例图像对应的二分类结果拼接以获取分类结果图像；
    二值化分割模块，用于将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割；
    分割拼接模块，用于将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像；
    融合模块，用于根据所述分类结果图像和所述分割结果图像计算二值化的融合图像；
    标注模块，用于提取所述融合图像的轮廓，并根据所述轮廓标注所述待标注图像中的感兴趣区域。
  9. 一种计算机设备，所述计算机设备包括存储器和处理器；
    所述存储器用于存储计算机程序；
    所述处理器，用于执行所述计算机程序并在执行所述计算机程序时实现如下步骤：
    获取待标注图像并对所述待标注图像进行预处理以得到若干实例图像；
    将各所述实例图像输入基于DenseNet网络的分类模型进行二分类；
    将所述若干实例图像对应的二分类结果拼接以获取分类结果图像；
    将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割；
    将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像；
    根据所述分类结果图像和所述分割结果图像计算二值化的融合图像；
    提取所述融合图像的轮廓，并根据所述轮廓标注所述待标注图像中的感兴趣区域。
  10. 如权利要求9所述的计算机设备，其中，所述处理器在实现所述将各所述实例图像输入基于DenseNet网络的分类模型进行二分类时，用于实现如下步骤：
    若基于DenseNet网络的分类模型预测所述实例图像存在感兴趣区域，所述实例图像对应的二分类结果为第一值区块；
    若基于DenseNet网络的分类模型预测所述实例图像不存在感兴趣区域，所述实例图像对应的二分类结果为第二值区块。
  11. 如权利要求10所述的计算机设备，其中，所述处理器在实现所述将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割时，用于实现如下步骤：
    将所述实例图像输入所述分割模型的Vnet网络；
    基于所述Vnet网络对所述实例图像依次进行非线性激活、若干次下采样和若干次上采样处理；
    基于所述分割模型的FPN网络，根据各所述上采样处理的结果输出二值化分割的结果；
    若所述分割模型预测所述实例图像存在感兴趣区域，所述二值化分割的结果中与所述感兴趣区域对应的区域为第一值区域；
    若所述分割模型预测所述实例图像存在非感兴趣区域，所述二值化分割的结果中与所述非感兴趣区域对应的区域为第二值区域。
  12. 如权利要求11所述的计算机设备，其中，所述处理器在实现所述根据所述分类结果图像和所述分割结果图像计算二值化的融合图像时，用于实现如下步骤：
    将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
  13. 如权利要求12所述的计算机设备，其中，所述处理器在实现所述将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像时，用于实现如下步骤：
    若所述分类结果图像中为第一值的像素在所述分割结果图像中也为第一值，将所述融合图像中相应的像素设为第一值；
    若所述分类结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值；
    若所述分割结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值。
  14. 如权利要求12所述的计算机设备，其中，所述处理器在实现所述将所述分类结果图像与所述分割结果图像取交集之前，用于实现如下步骤：
    将所述分类结果图像中的第一值区块进行膨胀；
    所述将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像，具体包括：
    将所述膨胀后的分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
  15. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于：若所述计算机程序被处理器执行，实现如下步骤：
    获取待标注图像并对所述待标注图像进行预处理以得到若干实例图像；
    将各所述实例图像输入基于DenseNet网络的分类模型进行二分类；
    将所述若干实例图像对应的二分类结果拼接以获取分类结果图像；
    将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割；
    将所述若干实例图像对应的二值化分割结果拼接以获取分割结果图像；
    根据所述分类结果图像和所述分割结果图像计算二值化的融合图像；
    提取所述融合图像的轮廓，并根据所述轮廓标注所述待标注图像中的感兴趣区域。
  16. 如权利要求15所述的存储介质，其中，所述处理器在实现所述将各所述实例图像输入基于DenseNet网络的分类模型进行二分类时，用于实现如下步骤：
    若基于DenseNet网络的分类模型预测所述实例图像存在感兴趣区域，所述实例图像对应的二分类结果为第一值区块；
    若基于DenseNet网络的分类模型预测所述实例图像不存在感兴趣区域，所述实例图像对应的二分类结果为第二值区块。
  17. 如权利要求16所述的存储介质，其中，所述处理器在实现所述将各所述实例图像输入基于Vnet网络和FPN网络的分割模型进行二值化分割时，用于实现如下步骤：
    将所述实例图像输入所述分割模型的Vnet网络；
    基于所述Vnet网络对所述实例图像依次进行非线性激活、若干次下采样和若干次上采样处理；
    基于所述分割模型的FPN网络，根据各所述上采样处理的结果输出二值化分割的结果；
    若所述分割模型预测所述实例图像存在感兴趣区域，所述二值化分割的结果中与所述感兴趣区域对应的区域为第一值区域；
    若所述分割模型预测所述实例图像存在非感兴趣区域，所述二值化分割的结果中与所述非感兴趣区域对应的区域为第二值区域。
  18. 如权利要求17所述的存储介质，其中，所述处理器在实现所述根据所述分类结果图像和所述分割结果图像计算二值化的融合图像时，用于实现如下步骤：
    将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
  19. 如权利要求18所述的存储介质，其中，所述处理器在实现所述将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像时，用于实现如下步骤：
    若所述分类结果图像中为第一值的像素在所述分割结果图像中也为第一值，将所述融合图像中相应的像素设为第一值；
    若所述分类结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值；
    若所述分割结果图像中的像素为第二值，将所述融合图像中相应的像素设为第二值。
  20. 如权利要求18所述的存储介质，其中，所述处理器在实现所述将所述分类结果图像与所述分割结果图像取交集之前，用于实现如下步骤：
    将所述分类结果图像中的第一值区块进行膨胀；
    所述将所述分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像，具体包括：
    将所述膨胀后的分类结果图像与所述分割结果图像取交集，以得到所述二值化的融合图像。
PCT/CN2019/103139 2019-04-04 2019-08-28 基于多模型融合的图像标注方法、装置、计算机设备及存储介质 Ceased WO2020199477A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
SG11202008418PA SG11202008418PA (en) 2019-04-04 2019-08-28 Multi-model fusion based method and device for image annotation, computer device, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910268876.7 2019-04-04
CN201910268876.7A CN110097564B (zh) 2019-04-04 2019-04-04 基于多模型融合的图像标注方法、装置、计算机设备及存储介质

Publications (1)

Publication Number Publication Date
WO2020199477A1 true WO2020199477A1 (zh) 2020-10-08

Family

ID=67444308

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/103139 Ceased WO2020199477A1 (zh) 2019-04-04 2019-08-28 基于多模型融合的图像标注方法、装置、计算机设备及存储介质

Country Status (3)

Country Link
CN (1) CN110097564B (zh)
SG (1) SG11202008418PA (zh)
WO (1) WO2020199477A1 (zh)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112215243A (zh) * 2020-10-30 2021-01-12 百度(中国)有限公司 图像特征提取方法、装置、设备及存储介质
CN114676759A (zh) * 2021-03-09 2022-06-28 北京航空航天大学 一种安检图像中违禁物品检测方法及装置
CN114782907A (zh) * 2022-03-29 2022-07-22 智道网联科技(北京)有限公司 无人车行驶环境识别方法、装置、设备和计算机可读存储介质
CN115861609A (zh) * 2022-11-15 2023-03-28 浙江大华技术股份有限公司 遥感图像的分割标注方法、电子设备及存储介质
CN116452599A (zh) * 2023-04-13 2023-07-18 江南大学 基于轮廓的图像实例分割方法及系统
CN116703941A (zh) * 2023-05-19 2023-09-05 依未科技(北京)有限公司 眼底渗出病灶提取方法及装置
CN116843899A (zh) * 2023-07-12 2023-10-03 中南大学深圳研究院 一种雾天交通场景图像语义分割方法及相关设备
CN117237203A (zh) * 2023-07-24 2023-12-15 西北工业大学 针对高动态条件下低信噪比模糊星图的智能化盲复原方法
WO2024045826A1 (zh) * 2022-08-29 2024-03-07 上海智臻智能网络科技股份有限公司 建筑图纸的空间面积计算方法及系统、建筑图纸的处理方法、设备和存储介质
CN121722253A (zh) * 2026-02-12 2026-03-24 厦门铠甲网络股份有限公司 基于人体姿态识别的云游戏交互方法和系统

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110097564B (zh) * 2019-04-04 2023-06-16 平安科技（深圳）有限公司 基于多模型融合的图像标注方法、装置、计算机设备及存储介质
CN110569875B (zh) * 2019-08-07 2022-04-19 清华大学无锡应用技术研究院 一种基于特征复用的深度神经网络目标检测方法
CN110607229A (zh) * 2019-09-29 2019-12-24 深圳赛动生物自动化有限公司 细胞转移观察机构及其工作方法
CN110567874A (zh) * 2019-09-29 2019-12-13 深圳赛动生物自动化有限公司 细胞培养结果自动观察机构及其工作方法
CN111369515A (zh) * 2020-02-29 2020-07-03 上海交通大学 一种基于计算机视觉的隧道水渍检测系统及方法
CN111523483B (zh) * 2020-04-24 2023-10-03 北京邮电大学 中餐菜品图像识别方法及装置
CN111696082B (zh) * 2020-05-20 2024-07-02 平安科技（深圳）有限公司 图像分割方法、装置、电子设备及计算机可读存储介质
CN111768392B (zh) * 2020-06-30 2022-10-14 创新奇智(广州)科技有限公司 目标检测方法及装置、电子设备、存储介质
CN111881902B (zh) * 2020-07-28 2023-06-27 平安科技（深圳）有限公司 训练样本制作方法、装置、计算机设备及可读存储介质
CN111915585B (zh) * 2020-07-29 2025-02-28 深圳市商汤科技有限公司 图像标注方法、装置、设备以及存储介质
CN112101376B (zh) * 2020-08-14 2024-10-22 北京迈格威科技有限公司 图像处理方法、装置、电子设备和计算机可读介质
CN112329807A (zh) * 2020-09-08 2021-02-05 上海思路迪生物医学科技有限公司 医用图像诊断装置、图像处理装置和方法、判断单元以及存储介质
CN112330580B (zh) * 2020-10-30 2024-08-13 厦门窝赚科技有限公司 生成人体衣物融合图像的方法、装置、计算设备、介质
CN112381834B (zh) * 2021-01-08 2022-06-03 之江实验室 一种对图像交互式实例分割的标注方法
CN113705613B (zh) * 2021-07-27 2024-02-02 浙江工业大学 一种基于空间位置引导的x光片桡骨远端骨折分类方法
CN115918635B (zh) * 2022-09-17 2025-10-10 浙江红谱科技有限公司 一种基于无线感知技术的驱鸟系统和方法
CN118674682B (zh) * 2024-05-17 2025-10-14 北京大学第三医院（北京大学第三临床医学院） 超声造影输卵管图像处理方法、装置、设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9940539B2 (en) * 2015-05-08 2018-04-10 Samsung Electronics Co., Ltd. Object recognition apparatus and method
CN107967688A (zh) * 2017-12-21 2018-04-27 联想(北京)有限公司 对图像中的对象进行分割的方法及系统
CN109034210A (zh) * 2018-07-04 2018-12-18 国家新闻出版广电总局广播科学研究院 基于超特征融合与多尺度金字塔网络的目标检测方法
CN109300530A (zh) * 2018-08-08 2019-02-01 北京肿瘤医院 病理图片的识别方法及装置
CN110097564A (zh) * 2019-04-04 2019-08-06 平安科技（深圳）有限公司 基于多模型融合的图像标注方法、装置、计算机设备及存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108875811A (zh) * 2018-06-01 2018-11-23 平安科技（深圳）有限公司 动物种类的分类方法、装置、计算机设备和存储介质
CN109003260B (zh) * 2018-06-28 2021-02-09 深圳视见医疗科技有限公司 Ct图像肺结节检测方法、装置、设备及可读存储介质
CN109242869B (zh) * 2018-09-21 2021-02-02 安徽科大讯飞医疗信息技术有限公司 一种图像实例分割方法、装置、设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9940539B2 (en) * 2015-05-08 2018-04-10 Samsung Electronics Co., Ltd. Object recognition apparatus and method
CN107967688A (zh) * 2017-12-21 2018-04-27 联想(北京)有限公司 对图像中的对象进行分割的方法及系统
CN109034210A (zh) * 2018-07-04 2018-12-18 国家新闻出版广电总局广播科学研究院 基于超特征融合与多尺度金字塔网络的目标检测方法
CN109300530A (zh) * 2018-08-08 2019-02-01 北京肿瘤医院 病理图片的识别方法及装置
CN110097564A (zh) * 2019-04-04 2019-08-06 平安科技（深圳）有限公司 基于多模型融合的图像标注方法、装置、计算机设备及存储介质

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112215243A (zh) * 2020-10-30 2021-01-12 百度(中国)有限公司 图像特征提取方法、装置、设备及存储介质
CN114676759A (zh) * 2021-03-09 2022-06-28 北京航空航天大学 一种安检图像中违禁物品检测方法及装置
CN114782907A (zh) * 2022-03-29 2022-07-22 智道网联科技(北京)有限公司 无人车行驶环境识别方法、装置、设备和计算机可读存储介质
WO2024045826A1 (zh) * 2022-08-29 2024-03-07 上海智臻智能网络科技股份有限公司 建筑图纸的空间面积计算方法及系统、建筑图纸的处理方法、设备和存储介质
CN115861609A (zh) * 2022-11-15 2023-03-28 浙江大华技术股份有限公司 遥感图像的分割标注方法、电子设备及存储介质
CN116452599A (zh) * 2023-04-13 2023-07-18 江南大学 基于轮廓的图像实例分割方法及系统
CN116452599B (zh) * 2023-04-13 2025-08-05 江南大学 基于轮廓的图像实例分割方法及系统
CN116703941A (zh) * 2023-05-19 2023-09-05 依未科技(北京)有限公司 眼底渗出病灶提取方法及装置
CN116843899A (zh) * 2023-07-12 2023-10-03 中南大学深圳研究院 一种雾天交通场景图像语义分割方法及相关设备
CN117237203A (zh) * 2023-07-24 2023-12-15 西北工业大学 针对高动态条件下低信噪比模糊星图的智能化盲复原方法
CN117237203B (zh) * 2023-07-24 2025-08-29 西北工业大学 针对高动态条件下低信噪比模糊星图的智能化盲复原方法
CN121722253A (zh) * 2026-02-12 2026-03-24 厦门铠甲网络股份有限公司 基于人体姿态识别的云游戏交互方法和系统

Also Published As

Publication number Publication date
CN110097564B (zh) 2023-06-16
CN110097564A (zh) 2019-08-06
SG11202008418PA (en) 2020-10-29

Similar Documents

Publication Publication Date Title
WO2020199477A1 (zh) 基于多模型融合的图像标注方法、装置、计算机设备及存储介质
CN109493350B (zh) 人像分割方法及装置
CN108765278B (zh) 一种图像处理方法、移动终端及计算机可读存储介质
CN112308866B (zh) 图像处理方法、装置、电子设备及存储介质
CN112102340B (zh) 图像处理方法、装置、电子设备和计算机可读存储介质
CN111144242B (zh) 一种三维目标检测方法、装置及终端
CN111666905B (zh) 模型训练方法、行人属性识别方法和相关装置
US9384398B2 (en) Method and apparatus for roof type classification and reconstruction based on two dimensional aerial images
WO2021068618A1 (zh) 图像融合方法、装置、计算处理设备和存储介质
CN112668577B (zh) 大尺度图像中目标物体的检测方法、终端及装置
CN111080660A (zh) 一种图像分割方法、装置、终端设备及存储介质
CN112651953B (zh) 图片相似度计算方法、装置、计算机设备及存储介质
US8781187B2 (en) Methods, apparatuses, and computer program products for identifying a region of interest within a mammogram image
CN112508005B (zh) 用于处理图像的方法、装置、设备以及存储介质
CN117576405A (zh) 舌象语义分割方法、装置、设备及介质
CN112700460A (zh) 图像分割方法及系统
WO2024174726A1 (zh) 基于深度学习的手写及打印文本检测方法和装置
WO2020119058A1 (zh) 微表情描述方法、装置、计算机装置及可读存储介质
CN115439733A (zh) 图像处理方法、装置、终端设备及计算机可读存储介质
Wu et al. Atrous residual convolutional neural network based on U-Net for retinal vessel segmentation
CN108805838A (zh) 一种图像处理方法、移动终端及计算机可读存储介质
Makram et al. Robust detection and refinement of saliency identification
Lin et al. Efficient saliency map detection for low-light images based on image gradient
CN114118127B (zh) 一种视觉场景标志检测与识别方法及装置
CN115880493A (zh) 一种实例分割方法、装置及计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19923009

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19923009

Country of ref document: EP

Kind code of ref document: A1