CN115546466A - A weakly supervised image object localization method based on multi-scale salient feature fusion - Google Patents

Abstract

A weak supervision image target positioning method based on multi-scale salient feature fusion belongs to the field of computer vision. In order to solve the two problems of complicated work of ROI labeling of a small target image and insufficient CAM activation, the invention focuses on the research of outputting a class activation graph by a classification network under the condition of optimizing weak supervision. The invention relates to information fusion of two layers: (1) because the semantic information of the feature map at the bottommost layer in the convolutional neural network is weak but the position information is strong, the feature map can be fused with the feature map at the highest layer to obtain the final feature map of the classification network; (2) because the sensitivity of the classification network to the ROIs with different scales is different, the obtained class activation maps are also different, the complementary object information in the different activation maps is fused, so that the positioning of the target region in the image can be perfected, and a more accurate pseudo label is generated for a segmentation task.

Description

A weakly supervised image object localization method based on multi-scale salient feature fusion

technical field

The invention relates to a weakly supervised image target positioning method based on multi-scale salient feature fusion, which belongs to the field of computer vision.

Background technique

The positioning and segmentation of the region of interest (RegionOfInterest, ROI) in the image is a classic problem in computer vision research. At present, the research on the positioning and segmentation of ROI based on natural images has made great progress. However, for unnatural images in some specific fields (such as medical images, pollen grain images), their ROIs are smaller than natural images, so the ROI positioning and segmentation method based on natural images is not completely suitable for such images. Therefore, it is of great significance to study the localization and segmentation of small image objects based on specific fields.

At present, there are two types of mainstream deep learning-based small target segmentation methods: fully supervised learning and weakly supervised learning. Z. Ning et al. ^[1] proposed to use the saliency maps of the foreground and background of breast ultrasound images to guide the main network and the auxiliary network to learn the foreground saliency representation and the background saliency representation respectively, and finally fuse the two features to enhance the morphological learning ability of the segmentation network . However, this type of fully supervised deep learning method generally requires a large amount of labeled data sets, and obtaining pixel-level labels of images is a complicated and time-consuming task. Relatively speaking, it is easier to obtain data sets with only category information , so many works only use weakly supervised methods such as image-level labels to achieve target location segmentation. However, the class activation map (Class Activation Map, CAM) obtained by the classification network in weakly supervised learning can only cover the most significant part of the image, and cannot indicate the complete target area, that is, the positioning accuracy of CAM is low (insufficient activation), To this end, Li Y et al. ^[2] first used the prior knowledge of breast anatomy to constrain the search space of the classification network for breast lesion tissue, and then used the level set algorithm to correct the CAM. But it ignores an important information: for objects of different scales, the discriminative regions captured by the classification network are not consistent.

In order to solve the two problems of complicated ROI labeling for small target images and insufficient CAM activation, the present invention focuses on the research on optimizing the output class activation map of the classification network under weak supervision. The present invention involves information fusion at two levels: ①Since the semantic information of the feature map at the bottom layer in the convolutional neural network is weak but the position information is strong, it can be fused with the feature map at the top layer to obtain the final feature map of the classification network; ②Since the classification The sensitivity of the network to ROIs of different scales is different, and the class activation maps obtained are also different. Therefore, the fusion of complementary object information in different activation maps can improve the positioning of the target area in the image, and then generate more accurate pseudo-labels for segmentation tasks. .

references:

[1] Z.Ning, S.Zhong, Q.Feng, W.Chen and Y.Zhang,"SMU-Net:Saliency-GuidedMorphology-Aware U-Net for Breast Lesion Segmentation in Ultrasound Image,"inIEEE Transactions on Medical Imaging ,vol.41,no.2,pp.476-490,Feb.2022,doi:10.1109/TMI.2021.3116087.

[2] Li Y, Liu Y, Huang L, Wang Z, Luo J. Deep weakly-supervised breast tumor segmentation in ultrasound images with explicit anatomical constraints. Med Image Anal. 2022Feb; 76:102315.doi:10.1016/j.media.2021.102315 .Epub 2021Nov 28.PMID: 34902792.

Contents of the invention

Aiming at the problem that the existing research on the location and segmentation of image small objects based on fully supervised learning has complicated labeling work, and the research on location and segmentation of single-scale images based on weakly supervised learning has insufficient CAM activation, the present invention designs a method based on multi-scale saliency A Weakly-Supervised Image Object Localization Approach for Feature Fusion. Specifically, we obtain images of three different scales by constructing an image pyramid, and thus obtain a multi-scale CAM of the same image, then fuse them, and finally use the fused CAM as weakly supervised information to train a segmentation network.

The weakly supervised image target location method based on multi-scale salient feature fusion in the present invention consists of five stages: the first stage is image preprocessing, which mainly unifies the resolution of images in the data set. The second stage is the construction of the image pyramid. This stage mainly includes three parts: taking the input image as the source image to down-sample to construct the top layer of the image pyramid, up-sampling to construct the bottom layer of the image pyramid, and finally determining the number of layers of the image pyramid. The third stage is the acquisition of classifier feature maps. In this stage, a classifier is first trained for each layer of the image pyramid, and then for any layer of the classifier, the feature map of the highest layer is spliced with the feature map of the lowest layer to obtain a fused feature map. The fourth stage is the fusion of multi-scale CAM. In this stage, the multi-scale CAM of the same image is first obtained through the weighted sum of the feature maps of each layer, then all the CAMs are aligned, and finally they are fused to obtain the final CAM of the source image. The fifth stage is the prediction of the target area. This stage first converts the fused CAM into a pseudo-binary label, then uses the pseudo-label to train the segmentation network, and finally predicts the target region through the segmentation network.

Concrete scheme of the present invention is as shown in accompanying drawing 2.

Step 1: Image Preprocessing

The purpose of image preprocessing is to unify the dimensions of all images in the dataset. The data referred to in the present invention are mainly small target image data, such as the public mammary gland image data set and pollen image data set. If the image resolution in the data set is not uniform, the size of the feature map obtained by the subsequent classification network will be inconsistent, and the parameters of the fully connected layer in the classification network cannot adapt to feature maps of different sizes, so the size of all input images must be fixed as Uniform size.

Step 2: Image Pyramid Construction

In this step, the image in the dataset is used as the source image, and the three scale transformations of the input image are obtained by constructing a Gaussian pyramid. In order to obtain more global and finer-grained information than the original image at the same time, the Gaussian pyramid constructed by the present invention adopts a pyramid structure of down-sampling and up-sampling.

Step 2.1 Construction of the top layer of the image pyramid: take the input image as the source image, first use the template Gaussian kernel of 5*5 size to perform Gaussian smoothing on it, and then downsample the processed image by removing the even-numbered rows and columns in the image matrix , and finally get an image of 1/4 the size of the input image, and use it as the top layer of the image pyramid.

Step 2.2 Construction of the bottom layer of the image pyramid: take the input image as the source image, first expand the image to twice the original size in each direction, and fill the newly added rows and columns with the value 0; then the 5*5 size The template Gaussian kernel is multiplied by 4 and then convolved with the enlarged image to obtain the approximate value of the added pixels. Finally, an image that is 4 times the size of the input image is obtained, and this is used as the bottom layer of the image pyramid.

Step 2.3 Determining the number of layers in the image pyramid: determine the numbers for the images on different layers in the image pyramid, where the number of layers in the image pyramid starts from 0, and as the number of pyramid layers increases, the image resolution decreases accordingly. The image pyramid constructed by the present invention has three layers, wherein the original image is on the second layer, and the corresponding pyramid layer number is 1.

Step 3: Classifier feature map acquisition

In this step, a classifier is trained for images of three different scales in the image pyramid to obtain class activation maps of three different scales of the same image.

Step 3.1 Classification network training: the present invention selects the classic ResNet50 as the classification network for judging the category to which the input image belongs. Since there are three images of different scales in the image pyramid, it is finally necessary to train a classifier for each of the three image datasets of different scales.

Step 3.2 Fusion of high- and low-level feature maps: For each classification network, the shallow receptive field is small, and low-level geometric information such as texture and edge is extracted; while the high-level receptive field is large, more global and deeper semantic information is extracted . Therefore, the present invention aligns and splices the highest-level features and the lowest-level features in each classification network, prompting the network to enhance the low-level features of small target objects, so as to obtain the final fusion feature map of the network.

Step 4: Multi-scale CAM Fusion

This step obtains the CAMs of the three classification networks, aligns them and then fuses them, and finally obtains the fused CAM map corresponding to the image.

Step 4.1 Classification network CAM acquisition: For the final fusion feature map obtained in step 3.2, multiply it with the weight matrix of the fully connected layer in the classification network to obtain the CAM. Since the present invention uses three classification networks, for each source image, three CAMs of different scales will be finally obtained to form a CAM pyramid.

Step 4.2 Align multiple CAMs: align CAMs of different scales based on the size of the source image to facilitate subsequent fusion operations.

Step 4.3 Fusion of multiple CAMs: For any pixel in the fused CAM, the present invention adopts the following judging mechanism: if there are at least two independent CAMs whose activation values for a certain category at this point are greater than or equal to the threshold, the pixel is considered to belong to the category. If the pixel is not assigned to any category after the judgment mechanism, the pixel is ignored; if the pixel is assigned to multiple categories, the pixel is assigned to the maximum average activation of the three independent CAMs at that point The category to which the value corresponds.

Step 5: ROI Prediction

In this step, first convert the fused CAM obtained in step 4.3 into a pseudo-label, then train the image ROI positioning and segmentation network based on the pseudo-label, and finally use the network to predict the ROI.

Step 5.1 Fused CAM pseudo-label conversion: Convert the fused CAM to a pseudo-binary mask for segmentation network training. The present invention adopts the following judging mechanism: if any pixel point in the fused CAM belongs to the non-target category, assign the pixel value of this point as 0, otherwise assign it as 1.

Step 5.2 Segmentation network training prediction: Based on the pseudo-binary label training image segmentation network obtained in step 5.1, the segmentation network architecture selected by the present invention is U-Net, and finally use the trained network to perform ROI segmentation prediction on the test set.

Compared with the prior art, the present invention has the beneficial effects of:

1. The weakly supervised image target positioning method based on multi-scale salient feature fusion adopted by the present invention effectively avoids the pixel-level labeling of ROI required under fully supervised learning, which greatly reduces the workload of data labeling.

2. The weakly supervised image target positioning method based on multi-scale salient feature fusion adopted by the present invention splices the highest-level feature map and the lowest-level feature map obtained by each classification network, which strengthens the network's ability to detect low-level features of small target objects Learning, so that the network can focus on more features of small target objects.

3. The weakly supervised image target positioning method based on multi-scale salient feature fusion adopted by the present invention constructs a hybrid pyramid structure of down-sampling and up-sampling based on the source image, thereby obtaining a more global and finer-grained image than the original image at the same time features, and a more complete CAM is obtained by fusing these features.

Description of drawings

Fig. 1 is a schematic diagram of an image pyramid constructed in the present invention.

Fig. 2 is the overall flowchart of the method proposed by the present invention.

detailed description

Below in conjunction with accompanying drawing 2 of description, the embodiment of the present invention is described in detail:

The weakly supervised image target positioning method based on multi-scale salient feature fusion in the present invention consists of five stages: the first stage is image preprocessing, which mainly unifies the resolution of the data set. The second stage is the construction of the image pyramid. This stage mainly includes three parts: taking the input image as the source image to down-sample to construct the top layer of the image pyramid, up-sampling to construct the bottom layer of the image pyramid, and finally determining the number of layers of the image pyramid. The third stage is the acquisition of classifier feature maps. In this stage, a classifier is first trained for each layer of the image pyramid, and then for any layer of the classifier, the feature map of the highest layer is spliced with the feature map of the lowest layer to obtain a fused feature map. The fourth stage is the fusion of multi-scale CAM. In this stage, the multi-scale CAM of the same image is first obtained through the weighted sum of the feature maps of each layer, then all the CAMs are aligned, and finally they are fused to obtain the final CAM of the source image. The fifth stage is the prediction of ROI. This stage first converts the fused CAM into a pseudo-binary label, then uses the pseudo-label to train the segmentation network, and finally predicts the ROI through the segmentation network.

Specifically, the method includes the following steps:

Step 1: Image Preprocessing

The purpose of image preprocessing is to unify the dimensions of all images in the dataset. The data referred to in the present invention are mainly small target image data, such as the public mammary gland image data set and pollen image data set. If the image resolution in the data set is not unique, the size of the feature map obtained by the last convolutional layer of the subsequent classification network will be different, and the parameter dimension of the connection between the fully connected layer and the previous layer is fixed in advance, that is, the parameters of the fully connected layer It cannot adapt to different feature map sizes, so the size of all input images must be fixed. In order to minimize the loss of image information and facilitate the convolution operation in the subsequent classification network, we set the size of all images to 512*512.

Step 2: Image Pyramid Construction

This step obtains three scale transformations of the input image by constructing a Gaussian pyramid. In order to obtain more global and finer-grained information than the original image at the same time, the Gaussian pyramid constructed by the present invention adopts a pyramid structure of down-sampling and up-sampling. Specifically, the construction process includes two parts: first, the width and height of the input original image are down-sampled to 50% of the original image through the Gaussian pyramid, thereby obtaining a 256*256 resolution image as the top layer of the pyramid; Second, through the Gaussian pyramid, the width and height of the input original image are up-sampled to 200% of the original image, thereby obtaining a 1024*1024 resolution image as the bottom layer of the pyramid.

Step 2.1 Construction of the top layer of the image pyramid: For a given original image with a size of 512*512, we downsample to construct the top layer of the Gaussian pyramid with an image of 1/4 the size of the original image, and the corresponding resolution of the image is 256*256. The specific process is shown in formula (1): firstly, a Gaussian smoothing process is performed on the 512*512 original image, which is different from simple smoothing. When Gaussian smoothing calculates the weighted average of surrounding pixels, it gives more High weight value; then the processed image is down-sampled by removing even rows and columns in the image matrix to obtain a 256*256 resolution image.

(1≤l≤L,0≤x≤R _l ,0≤y≤C _l )

Among them, G _l is the l-th layer image of the Gaussian pyramid (the number of layers of the Gaussian pyramid starts from 0), L is the layer number of the top layer of the Gaussian pyramid, R _l and C _l are the number of rows and columns of the l-th layer image respectively, W( m, n) is the value of the mth row and the nth column of the Gaussian filter template, which is generally 5*5 in size. The present invention selects the two-dimensional separable 5*5 Gaussian kernel widely used in the unsharp mask algorithm to check the original image. Smoothing, its value is shown in (2).

Step 2.2 Construction of the bottom layer of the image pyramid: For a given 512*512 original image, we upsample an image 4 times the size of the original image to construct the lowest layer of the Gaussian pyramid, which corresponds to a resolution of 1024*1024. The specific process is as follows: First, the image is enlarged to twice the original image in each direction, and the newly added rows and columns are filled with the value 0; then the Gaussian kernel used in the downsampling is multiplied by 4, and then the It performs convolution operation with the enlarged image to obtain the approximate value of the newly added pixels, and finally obtains an image with a resolution of 1024*1024.

Step 2.3 Determination of the number of layers of the image pyramid: After the construction of the image pyramid is completed, the number of layers l in the Gaussian pyramid corresponding to the image with a resolution of w*h is determined by formula (3).

Where l ₀ is the number of layers of the original image of 512*512 in the image pyramid. Since the three scales of the image in the Gaussian pyramid are 1024, 512, and 256 respectively, the number of layers corresponding to the original image is l ₀ =1. From the formula (3), it can be obtained that the corresponding layer number of the 1024*1024 image is 0, that is, it is located at the lowest layer of the Gaussian pyramid; the corresponding layer number of the 256*256 image is 2, that is, it is located at the top layer of the Gaussian pyramid.

Step 3: Classifier feature map acquisition

In this step, a classifier is trained for images of three different scales in the image pyramid to obtain class activation maps of three different scales of the same image.

Step 3.1 Classification network training: The classification network is used to distinguish the category of the input image. The classification network selected by the present invention is ResNet50. The ResNet50 network includes 49 convolutional layers and 1 fully connected layer. Each residual block has three layers of convolution , the residual structure of the network can directly connect the input to the following network layer to avoid the loss of information. The present invention trains a classifier respectively for images with three resolutions of 256*256, 512*512, and 1024*1024 in the image pyramid, which are denoted as R ₁ , R ₂ , and R ₃ .

最低层特征图为

则分类器R₁最终的特征图f₁可由公式(4)得到。Step 3.2 Fusion of high-level and low-level feature maps: For each classification network, the feature map of the shallow layer has a small receptive field, and the local and general features such as image texture and edge are extracted, that is, low-level geometric information; As the number deepens, the high-level feature map has a larger receptive field, and deeper and more global features are extracted, that is, high-level semantic information. Therefore, the present invention splices the highest-level features and the lowest-level features in each classification network as the final feature map output by the network, prompting the network to enhance the low-level features of small target objects. For the classifier R ₁ , let the highest-level feature map obtained by its network be

The lowest layer feature map is

Then the final feature map f ₁ of classifier R ₁ can be obtained by formula (4).

为特征图的横向连接操作，即逐元素相加。同理，分类器R₂和分类器R₃最终的特征图f₂、f₃则可由公式(5)和(6)所得。Among them, UP is an upsampling operation, that is, upsampling the feature map of the highest layer to achieve the same size as the feature map of the lowest layer, which is convenient for subsequent processing.

is the horizontal connection operation of the feature map, that is, element-wise addition. Similarly, the final feature maps f ₂ and f ₃ of classifier R ₂ and classifier R ₃ can be obtained by formulas (5) and (6).

分别为分类器R₂得到的最高层特征图和最低层特征图，

则分别为分类器R₃得到的最高层特征图和最低层特征图，UP为上采样操作，

为特征图的横向连接操作，即逐元素相加。in

are the highest-level feature map and the lowest-level feature map obtained by the classifier R ₂ , respectively,

are respectively the highest-level feature map and the lowest-level feature map obtained by the classifier R ₃ , UP is an upsampling operation,

is the horizontal connection operation of the feature map, that is, element-wise addition.

Step 4: Multi-scale CAM Fusion

This step obtains the CAMs of the three classification networks, aligns them and then fuses them, and the final output is the fused CAM image corresponding to the image.

可由公式(7)获得。Step 4.1 Classification network CAM acquisition: For classifier R ₁ , the activation value of spatial pixel u(x,y) in the image with a resolution of 256*256 for class c

Can be obtained by formula (7).

为通道i中类别c对应的权重，f_i ¹(x,y)为分类器R₁最后融合特征图中通道i上位置 (x,y)的特征值。同理，对于分类器R₂和分类器R₃，图像上像素点u(x,y)关于类c的激活值

可分别由公式(8)和(9)获得。Where i is the channel number of the last convolutional layer of the classification network, K is the number of channels of the last convolutional layer of the classification network,

is the weight corresponding to category c in channel i, f _i ¹ (x, y) is the feature value of position (x, y) on channel i in the final fusion feature map of classifier R ₁ . Similarly, for classifier R ₂ and classifier R ₃ , the activation value of pixel u(x,y) on the image with respect to class c

can be obtained by formulas (8) and (9), respectively.

为通道i中类别c对应的权重，f_i ²(x,y)和f_i ³(x,y)分别为分类器R₂和R₃最后融合特征图中通道i上位置(x,y)的特征值。Where i is the channel number of the last convolutional layer of the classification network, K is the number of channels of the last convolutional layer of the classification network,

is the weight corresponding to category c in channel i, f _i ² (x, y) and f _i ³ (x, y) are the positions (x, y) on channel i in the final fusion feature map of classifiers R ₂ and R ₃ respectively eigenvalues of .

Step 4.2 Multiple CAM alignment: Since the input of the classifiers R ₁ , R ₂ , and R ₃ is a three-layer image pyramid, the dimensions of the three class activation maps obtained also constitute an activation map pyramid. In order to fuse three CAMs of different scales, they need to be aligned. In the present invention, all CAMs are set to the same size as the original input image, that is, 512*512 resolution.

Step 4.3 Multiple CAM fusion: The aligned three CAMs are fused into the final CAM. For the pixel point u(x,y) in the fusion class activation map _Magg , the fusion mechanism of the present invention is as follows: if there are at least two independent activation maps at this point, the activation value of the class c is greater than or equal to the threshold θ(θ ∈[0.5, 0.7]), then the pixel in _Magg is considered to belong to category c. If the pixel is not assigned to any category after the fusion mechanism, the pixel is ignored; if the pixel is assigned to multiple categories, the final category cla(x,y) is determined according to formula (10).

为像素点u(x,y)在第j层金字塔得到的特征图中关于类别c的激活值。

指像素点u(x,y)在P个特征图中关于背景类(类别编号为0)的平均激活值，

指像素点u(x,y)在P个特征图中关于类别c的平均激活值，

指像素点u(x,y)在P个特征图中关于类别N的平均激活值。index为取索引操作，即取数组中最大值对应的索引序号，在这里也指像素点所属类别，例如数组中第0 个平均激活值最大，则取出的索引值为0，也指其类别属于0。Where j is the number of pyramid layers, P is the total number of layers of the pyramid, where P=3, N is the number of categories divided by the data set (not including the background class),

is the activation value of category c in the feature map obtained for the pixel point u(x, y) at the jth layer of the pyramid.

Refers to the average activation value of the pixel point u(x,y) in the P feature map with respect to the background class (category number 0),

Refers to the average activation value of pixel u(x, y) in P feature maps with respect to category c,

Refers to the average activation value of pixel u(x, y) in P feature maps with respect to category N. index is an indexing operation, that is, taking the index number corresponding to the maximum value in the array. Here, it also refers to the category to which the pixel belongs. For example, the 0th in the array has the largest average activation value, and the extracted index value is 0, which also means that its category belongs to 0.

Step 5: ROI Area Prediction

This step first converts the fused CAM obtained in step 4.3 into a pseudo-label, then trains the segmentation network based on the pseudo-label, and finally uses the network to predict the ROI.

中像素点u(x,y)的取值由公式(11)确定。Step 5.1 Fused CAM pseudo-label conversion: Convert the fused CAM to a pseudo-binary mask for segmentation network training. pseudo binary mask

The value of the pixel point u(x, y) in is determined by the formula (11).

Among them, cla(x, y) refers to the category to which the pixel point u(x, y) belongs, and cla(x, y)=0 indicates that the pixel point belongs to a non-target category.

Step 5.2 Segmentation network training prediction: Based on the pseudo-binary label training image segmentation network obtained in step 5.1, the segmentation network architecture selected by the present invention is U-Net, and finally use the trained network to perform ROI segmentation prediction on the test set.

The present invention is mainly aimed at small target image data, such as medical image data and pollen image data with small lesion areas. By fusing the multi-scale salient features obtained from the image pyramid, the position information and contour information of small target objects can be enhanced, thereby Improve the performance of small object localization and segmentation tasks in weakly supervised scenarios. The steps described in specific examples of the invention may be modified without departing from the basic spirit of the invention in terms of system architecture. Accordingly, the present embodiments are to be considered as illustrative rather than restrictive, with the scope of the invention being defined by the appended claims rather than the foregoing description.

Claims (2)

1. A weak supervision image target positioning method based on multi-scale salient feature fusion is characterized by comprising the following steps:

step 1: image pre-processing

The purpose of image pre-processing is to unify the size of all images within a data set;

step 2: image pyramid construction

The method comprises the steps of taking an image in a data set as a source image, and obtaining three scale transformations of an input image by constructing a Gaussian pyramid; in order to obtain more global and finer-grained information than the original image, the constructed Gaussian pyramid adopts a pyramid structure with a mixture of down sampling and up sampling;

step 2.1, image pyramid top layer construction: taking an input image as a source image, firstly performing Gaussian smoothing processing on the input image by using a template Gaussian core of 5*5 size, then performing downsampling on the processed image by removing even rows and columns in an image matrix, and finally obtaining an image of 1/4 size of the input image, wherein the image is used as an image pyramid top layer;

step 2.2, constructing the pyramid bottom layer of the image: taking an input image as a source image, firstly, expanding the image to be 2 times of the original image in each direction, wherein newly added rows and columns are filled with a numerical value of 0; then, multiplying the template Gaussian kernel of 5*5 by 4, and then performing convolution operation on the multiplied image to obtain an approximate value of a newly added pixel; finally, obtaining an image of 4 times of the size of the input image, and taking the image as an image pyramid bottom layer;

step 2.3, determining the pyramid layer number of the image: determining numbers for images on different layers in an image pyramid, wherein the number of the pyramid layers of the image is from 0, and the resolution of the image is correspondingly reduced along with the increase of the pyramid layers; the constructed image pyramid is 3 layers, wherein the original image is positioned at the 2 nd layer, and the number of the corresponding pyramid layer is 1;

and step 3: classifier feature map acquisition

Respectively training a classifier aiming at three images with different scales in an image pyramid to obtain class activation graphs of the same image with three different scales;

step 3.1, training a classification network: selecting classical ResNet50 as a classification network for judging the class of the input image; because three images with different scales exist in the image pyramid, a classifier is required to be trained for three image data sets with different scales respectively;

step 3.2, fusing the high-low layer characteristic diagrams:

aligning and splicing the highest layer features and the lowest layer features in each classification network to promote the network to enhance the low-level features of the small target object so as to obtain a final fusion feature map of the network;

and 4, step 4: multi-scale CAM fusion

The step of obtaining the CAMs of the three classification networks, aligning the CAMs and then fusing the CAMs to finally obtain a fused CAM image corresponding to the image;

step 4.1, obtaining by the classification network CAM: obtaining a CAM by multiplying the final fusion characteristic diagram obtained in the step 3.2 by a weight matrix of a full connection layer in the classification network; because three classification networks are used, three CAMs with different scales are finally obtained for each source image to form a CAM pyramid;

step 4.2 multiple CAM alignment: aligning the CAMs with different scales based on the size of the source image so as to facilitate subsequent fusion operation;

step 4.3 multiple CAM fusions: for any pixel in the fused CAM, the following judgment mechanism is adopted: if the activation value of at least two independent CAMs at the point related to a certain category is larger than or equal to a threshold value, the pixel point is considered to belong to the category; if the pixel point is not allocated to any category after passing through the judgment mechanism, ignoring the pixel point; if the pixel point is allocated to a plurality of categories, the pixel point is allocated to the category corresponding to the maximum average activation value of the three independent CAMs at the point;

and 5: ROI prediction

Firstly, converting the fusion CAM obtained in the step 4.3 into a pseudo label, then training a positioning segmentation network of an image ROI based on the pseudo label, and finally predicting the ROI by using the network;

step 5.1, fusing CAM pseudo label conversion: converting the fused CAM into a pseudo-binary mask for segmenting network training; the following judgment mechanism is adopted: if any pixel point in the fusion CAM belongs to the non-target class, assigning the pixel value of the point to be 0, and otherwise assigning the pixel value to be 1;

step 5.2, training and predicting the segmentation network: and (5) training an image segmentation network based on the pseudo binary label obtained in the step (5.1), wherein the selected segmentation network architecture is U-Net, and finally, performing ROI segmentation prediction on the test set by using the trained network.

2. The weak supervised image target localization method based on multi-scale salient feature fusion as recited in claim 1, wherein:

step 1: image pre-processing

The purpose of image pre-processing is to unify the size of all images in the dataset; all images were sized 512 x 512;

step 2: image pyramid construction

The construction process includes two parts: firstly, the width and the height of an input original image are respectively down-sampled into 50% of an original image through a Gaussian pyramid, and an image with 256 × 256 resolution is obtained as the top layer of the pyramid; secondly, respectively up-sampling the width and the height of the input original image into 200% of the original image through a Gaussian pyramid, and thus obtaining an image with 1024 × 1024 resolution as the bottom layer of the pyramid; the method comprises the following specific steps:

step 2.1, image pyramid top layer construction:

for a given original image with a size of 512 × 512, downsampling to construct a top layer of a gaussian pyramid from an image with a size of 1/4 of the original image, wherein the corresponding resolution of the image is 256 × 256; the specific process is shown as formula (1): firstly, performing primary Gaussian smoothing on an original image of 512 by 512, wherein the primary Gaussian smoothing is different from simple smoothing, and when the weighted average value of surrounding pixels is calculated, pixels adjacent to a central point are endowed with higher weight values by Gaussian smoothing; the processed image is then down-sampled by removing even rows and columns from the image matrix to obtain a 256 x 256 resolution image;

1≤l≤L,0≤x≤R _l ,0≤y≤C _l

wherein G is _l The image of the first layer of the Gaussian pyramid is obtained, the number of the layers of the Gaussian pyramid is started from 0, L is the layer number of the top layer of the Gaussian pyramid, and R is _l And C _l Respectively the number of rows and columns of the image of the l layer, wherein W (m, n) is the value of the nth row and column of the mth row of the Gaussian filter template, the value is generally 5*5, and the original image is smoothed by selecting a Gaussian core of a two-dimensional separable 5*5 widely used in an anti-sharpening mask algorithm, and the value is shown as (2);

step 2.2, constructing the pyramid bottom layer of the image:

for a given 512 x 512 original image, upsampling to construct the lowest layer of a Gaussian pyramid by using an image with the size 4 times that of the original image, wherein the corresponding resolution is 1024 x 1024; the specific process is as follows: firstly, expanding the image to be 2 times of the original image in each direction, wherein the newly added rows and columns are filled with 0 values; then, multiplying the Gaussian kernel used in the down sampling by 4, and then performing convolution operation on the Gaussian kernel and the amplified image to obtain an approximate value of a newly added pixel, and finally obtaining an image with 1024 × 1024 resolution;

step 2.3, determining the pyramid layer number of the image:

after the image pyramid is constructed, determining the number l of layers in the Gaussian pyramid corresponding to the image with the resolution of w x h by a formula (3);

wherein l ₀ The number of layers l of the original image is 512 × 512, since the three scales of the image in the gaussian pyramid are 1024, 512, and 256, respectively, the number of layers l of the original image corresponds to ₀ =1; as can be seen from formula (3), the number of corresponding layers of 1024 × 1024 images is 0, i.e., the corresponding layers are located at the lowest layer of the gaussian pyramid; 256 by 256 images correspond to 2 layers, namely the images are positioned at the topmost layer of the Gaussian pyramid;

and step 3: classifier feature map acquisition

Respectively training a classifier aiming at three images with different scales in an image pyramid to obtain class activation graphs of the same image with three different scales, wherein the steps are as follows;

step 3.1, training a classification network: the classification network is used for judging the class of the input image, the selected classification network is ResNet50, the ResNet50 network comprises 49 convolution layers and 1 full-connection layer, and each residual block has three layers of convolution; respectively training a classifier for images with three resolutions of 256 × 256, 512 × 512 and 1024 × 1024 in the image pyramid, and marking as R ₁ 、R ₂ 、R ₃ ；

Step 3.2, fusing the high-low layer characteristic diagrams:

splicing the highest layer features and the lowest layer features in each classification network, and using the highest layer features and the lowest layer features as a final feature map output by the network to promote the network to enhance the low-level features of the small target objects; for the classifier R ₁ Let its network obtain the highest level feature diagram as

The lowest layer characteristic diagram is

Then the classifier R ₁ Final feature map f ₁ Can be obtained from formula (4);

wherein UP is an upsampling operation, i.e., upsampling is performed on the feature map of the highest layer to achieve the same size as the feature map of the lowest layer, so as to facilitate subsequent processing,

a cross-join operation for the feature map, i.e., element-by-element addition; in the same way, the classifier R ₂ And a classifier R ₃ Final feature map f ₂ 、f ₃ Then the results can be obtained from equations (5) and (6);

wherein

Are respectively a classifier R ₂ The obtained highest-level feature map and the lowest-level feature map,

are respectively the classifier R ₃ The obtained highest layer characteristic diagram and the lowest layer characteristic diagram, UP is the UP-sampling operation,

a cross-join operation for the feature map, i.e., element-by-element addition;

and 4, step 4: multi-scale CAM fusion

Acquiring CAMs of the three classification networks, aligning the CAMs, and fusing the CAMs, wherein the final output is a fused CAM image corresponding to the image, and the method specifically comprises the following steps:

step 4.1, obtaining by a classified network CAM: for the classifier R ₁ In particular, the activation value of a spatial pixel u (x, y) in an image with 256 × 256 resolution with respect to class c

Can be obtained from equation (7);

wherein i is the channel number of the last convolutional layer of the classification network, K is the channel number of the last convolutional layer of the classification network,

for the weight corresponding to class c in channel i, f _i ¹ (x, y) is a classifier R ₁ Finally, fusing the characteristic values of the positions (x, y) on the channel i in the characteristic diagram; for the same reason, for classifier R ₂ And a classifier R ₃ Activation value of a pixel point u (x, y) on an image with respect to class c

Can be obtained from equations (8) and (9), respectively;

wherein i is the channel number of the last convolutional layer of the classification network, and K is the channel number of the last convolutional layer of the classification network，

For the weight corresponding to class c in channel i, f _i ² (x, y) and f _i ³ (x, y) are classifiers R, respectively ₂ And R ₃ Finally, fusing the characteristic values of the positions (x, y) on the channel i in the characteristic diagram;

step 4.2 multiple CAM alignment: due to the classifier R ₁ 、R ₂ 、R ₃ The input of (2) is an image pyramid with three layers, so the sizes of the three obtained activation mapping graphs also form an activation graph pyramid; in order to fuse three differently scaled CAMs, it is necessary to align them, setting all CAMs to a size consistent with the original input image, i.e. 512 × 512 resolution;

step 4.3 multiple CAM fusions: fusing the three aligned CAMs into a final CAM; activation map M for fusion classes _agg The fusion mechanism of the pixel u (x, y) in (1) is as follows: if there are at least two independent activation graphs at the point where the activation value for class c is greater than or equal to the threshold value theta, theta epsilon [0.5,0.7]Then, consider M _agg The pixel belongs to the category c; if the pixel point is not allocated to any category after the fusion mechanism, ignoring the pixel; if the pixel point is allocated to a plurality of categories, judging the category cla (x, y) to which the pixel point belongs finally according to a formula (10);

where j is the number of pyramid levels, P is the total number of pyramid levels, where P =3,N is the number of classes of data set partitioning (excluding background classes),

the activation value of the pixel u (x, y) in the feature map obtained by the pyramid of the j layer about the category c is obtained;

finger pixelThe average activation value of the point u (x, y) in the P feature maps about the background class, and the background class is numbered as 0;

refers to the average activation value of pixel point u (x, y) in P feature maps for category c,

the average activation value of the pixel point u (x, y) in the P characteristic graphs about the category N is pointed; index is an index-taking operation, that is, an index sequence number corresponding to the maximum value in the array is taken, and here, the index sequence number also refers to the category to which the pixel belongs, for example, if the 0 th average activation value in the array is the maximum, the index value taken out is 0, and also refers to that the category belongs to 0;

and 5: ROI region prediction

Firstly, converting the fused CAM obtained in the step 4.3 into a pseudo label, training and segmenting a network based on the pseudo label, and finally predicting the ROI by using the network; the method comprises the following specific steps:

step 5.1, fusing CAM pseudo label conversion: converting the fused CAM into a pseudo-binary mask for segmenting network training; pseudo binary mask

The value of the middle pixel u (x, y) is determined by the formula (11);

wherein cla (x, y) refers to the category to which the pixel u (x, y) belongs, and cla (x, y) =0 indicates that the pixel belongs to the non-target category;

step 5.2, training and predicting the segmentation network: and (5) training an image segmentation network based on the pseudo binary label obtained in the step (5.1), wherein the selected segmentation network architecture is U-Net, and finally, performing ROI segmentation prediction on the test set by using the trained network.

Images