CN112101376A - Image processing method, apparatus, electronic device and computer readable medium - Google Patents

Abstract

The present invention provides an image processing method, device, electronic device and computer-readable medium, comprising: acquiring an image to be processed, and performing feature extraction on the image to be processed to obtain a multi-scale feature map; The corresponding part of the salient object is enhanced to obtain a multi-scale enhanced feature map; the image restoration is performed on the multi-scale enhanced feature map to obtain a salient object mask corresponding to the image to be processed. After the method of the present invention performs enhancement processing on the parts corresponding to the salient objects in the multi-scale feature map, in the obtained multi-scale enhanced feature map, the feature maps corresponding to the salient objects are more prominent, and finally the multi-scale enhancement is carried out. After the feature map is restored, the salient object mask obtained by segmentation is more accurate, which alleviates the technical problem of poor accuracy when processing the image by the existing salient object segmentation method.

Description

Image processing method, apparatus, electronic device and computer readable medium

technical field

The present invention relates to the technical field of image processing, and in particular, to an image processing method, apparatus, electronic device and computer-readable medium.

Background technique

Salient Object Segmentation is an important subject in Computer Vision. It has a very wide range of applications in the fields of mobile phone autofocus, unmanned driving, scene understanding, and image editing. The purpose of salient object segmentation is to distinguish the pixels of salient objects from other background pixels in an image. Unlike traditional semantic segmentation tasks, salient objects do not belong to the same class of objects and do not have semantically related labels. However, the salient objects are often in the middle of the image and have rich colors, as shown in Figure 1(a), and Figure 1(b) is a schematic diagram of the segmentation result of the salient objects corresponding to Figure 1(a).

Existing salient object segmentation methods are mainly divided into two categories. One of them is to analyze the texture of the image to determine the area with rich texture in the image, and then use the clustering method to distinguish the object from other areas with single texture. This method is limited by the clustering method, and it is difficult to obtain high accuracy; the other is to regard salient object segmentation as a standard object segmentation problem. However, standard object segmentation is to segment the preset types of objects in the image, for example, people, cars, and dogs in the image, but these objects may not be salient objects for a specific image, or Not all of them are salient objects, which leads to errors in the segmented salient objects.

To sum up, the existing saliency object segmentation methods have the technical problem of poor accuracy when processing images.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide an image processing method, apparatus, electronic device and computer-readable medium, so as to alleviate the technical problem of poor accuracy when processing an image in an existing saliency object segmentation method.

In a first aspect, an embodiment of the present invention provides an image processing method, including: acquiring an image to be processed, and performing feature extraction on the to-be-processed image to obtain a multi-scale feature map; The part corresponding to the salient object is enhanced to obtain a multi-scale enhanced feature map; the image restoration is performed on the multi-scale enhanced feature map to obtain a mask of the salient object corresponding to the to-be-processed image.

Further, performing feature extraction on the image to be processed includes: performing multi-layer downsampling processing on the image to be processed to obtain a multi-scale original feature map; performing optimization processing on the multi-scale original feature map to obtain the multi-scale original feature map. The multi-scale feature maps are described.

Further, performing the optimization process on the multi-scale original feature map includes: performing a first optimization process on the target original feature map in the multi-scale original feature map to obtain a first optimized feature map, wherein the The target original feature map is the feature map of the multi-scale original feature map, except the highest-dimensional original feature map; the second optimization process is performed on the highest-dimensional original feature map in the multi-scale original feature map to obtain the first Two optimized feature maps; the first optimized feature map and the second optimized feature map are used as the multi-scale feature maps.

Further, performing a first optimization process on the target original feature map in the multi-scale original feature map includes: using a first optimization module to perform an optimization process on the target original feature map to obtain a first initial optimized feature map, Wherein, the first optimization module includes: a preset number of first convolutional layers; summing the feature map of the first initial optimization and its corresponding target original feature map to obtain the first optimized feature map feature map.

Further, performing a second optimization process on the highest-dimensional original feature map in the multi-scale original feature map includes: using a second optimization module to perform an optimization process on the highest-dimensional original feature map to obtain an optimization weight, wherein the The second optimization module includes: a second convolution layer, a global pooling layer and a Sigmoid function processing layer; the optimization weight and the highest dimensional original feature map are multiplied to obtain a second initial optimized feature map; The second initial optimized feature map and the highest-dimensional original feature map are summed to obtain the second optimized feature map.

Further, performing enhancement processing on the part corresponding to the salient object in the multi-scale feature map includes: obtaining the initial position of the salient object according to the multi-scale feature map; At different expansion scales, crop the feature map of the highest dimension in the multi-scale feature map to obtain a plurality of cropped feature maps, and the plurality of cropped feature maps contain the feature information of the salient objects; One or more of the feature maps are used as target feature maps, each of the target feature maps is used as the current target feature map one by one, the correlation between the multiple cropped feature maps and the current target feature map is calculated, and the current target feature map is obtained. A plurality of correlation feature maps of the target feature map corresponding to the plurality of cropped feature maps one-to-one; according to the plurality of correlation feature maps and the current target feature map, the enhancement corresponding to the current target feature map is obtained feature map.

Further, obtaining the initial position of the salient object according to the multi-scale feature map includes: performing dimension reduction processing on the highest-dimensional feature map in the multi-scale feature map to obtain a single-channel feature map; The feature map of the channel is binarized to obtain a single-channel binarized feature map; the initial position of the salient object is determined according to the single-channel binarized feature map.

Further, according to the initial position, with at least two different expansion scales, cropping the feature map of the highest dimension in the multi-scale feature map includes: determining the pixel width and pixel height of the salient object according to the initial position. ; Determine the expanded pixel width and the expanded pixel height according to the expanded scale, the pixel width and the pixel height; In the feature map of the highest dimension, expand the expanded pixel width and all the expanded pixel widths along the initial position. Crop at the position after expanding the pixel height described above.

Further, calculating the correlation between the multiple cropping feature maps and the current target feature map includes: scaling the multiple cropping feature maps to a preset scale to obtain multiple cropping feature maps of the preset scale; The preset scale is that the sliding window slides on the current target feature map; after each sliding, the feature map included in the sliding window and the multiple cropped feature maps of the preset scale are respectively multiplied by the product operation, according to The result of the product operation obtains a plurality of the correlation feature maps of the current target feature map that correspond one-to-one with the multiple cropped feature maps.

Further, obtaining the enhanced feature map corresponding to the current target feature map according to the plurality of correlation feature maps and the current target feature map includes: combining each correlation feature in the plurality of correlation feature maps The product operation is performed on the current target feature map and the current target feature map to obtain a plurality of first enhanced feature maps corresponding to the current target feature map; the plurality of first enhanced feature maps and the current target feature map are connected in series to obtain the obtaining the second enhanced feature map corresponding to the current target feature map; obtaining the position enhanced feature map corresponding to the current target feature map, and connecting the second enhanced feature map and the position enhanced feature map in series to obtain a The enhanced feature map corresponding to the target feature map, wherein the scale of the position enhanced feature map is the same as the scale of the second enhanced feature map.

Further, acquiring the position enhancement feature map corresponding to the current target feature map includes: determining the center line of the salient object in the X direction and the center line in the Y direction based on the initial position of the salient object; The center line of the X direction is set as the first target value, and linearly transformed into the second target value along the X direction to obtain the position enhancement feature map in the X direction; the center line in the X direction is set as the first target value, along the X direction The Y direction is linearly transformed into the second target value to obtain the position enhancement feature map in the Y direction; the position enhancement feature map in the X direction and the position enhancement feature map in the Y direction are used as the position enhancement feature map.

Further, performing image restoration on the multi-scale enhanced feature map includes: up-sampling the multi-scale enhanced feature map to obtain a salient object mask corresponding to the to-be-processed image.

In a second aspect, an embodiment of the present invention further provides an image processing apparatus, including: a feature extraction unit, configured to acquire an image to be processed, and perform feature extraction on the to-be-processed image to obtain a multi-scale feature map; enhancement processing The unit is used for enhancing the part corresponding to the salient objects in the multi-scale feature map to obtain the multi-scale enhanced feature map; the image restoration unit is used for image restoration on the multi-scale enhanced feature map , to obtain the saliency object mask corresponding to the image to be processed.

In a third aspect, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, when the processor executes the computer program Implement the steps of the method according to any one of the above first aspects.

In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a non-volatile program code executable by a processor, the program code causing the processor to execute any one of the above-mentioned first aspect. steps of the method.

In the embodiment of the present invention, first, an image to be processed is acquired, and feature extraction is performed on the image to be processed to obtain a multi-scale feature map; then, parts corresponding to salient objects in the multi-scale feature map are enhanced to obtain Multi-scale enhanced feature map; finally, image restoration is performed on the multi-scale enhanced feature map to obtain a salient object mask corresponding to the image to be processed. It can be seen from the above description that after the enhancement processing is performed on the part corresponding to the salient objects in the multi-scale feature map, the multi-scale enhanced feature map obtained, the feature map corresponding to the salient objects is more prominent, and finally the multi-scale enhanced feature map is obtained. After image restoration is performed by enhancing the feature map, the salient object mask obtained by segmentation is more accurate, which alleviates the technical problem of poor accuracy of the existing salient object segmentation methods when processing images.

Description of drawings

In order to illustrate the specific embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the specific embodiments or the prior art. Obviously, the accompanying drawings in the following description The drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

1(a) is a schematic diagram of an image to be processed provided by an embodiment of the present invention;

FIG. 1(b) is a schematic diagram of a salient object segmentation result corresponding to FIG. 1(a) provided by an embodiment of the present invention;

2 is a schematic diagram of an electronic device according to an embodiment of the present invention;

3 is a flowchart of an image processing method provided by an embodiment of the present invention;

4 is an overall schematic diagram of an image processing method provided by an embodiment of the present invention;

5 is a schematic diagram of a first optimization process provided by an embodiment of the present invention;

6 is a schematic diagram of a second optimization process provided by an embodiment of the present invention;

FIG. 7 is a flowchart of enhancing processing for parts corresponding to salient objects in a multi-scale feature map provided by an embodiment of the present invention;

8 is a flowchart of determining an enhanced feature map corresponding to a current target feature map according to an embodiment of the present invention;

9 is a schematic diagram of a position enhancement feature map provided by an embodiment of the present invention;

FIG. 10 is a schematic diagram of performing enhancement processing on a part corresponding to a salient object in a multi-scale feature map provided by an embodiment of the present invention;

11 is a schematic diagram showing the results of training and testing the image processing method of the present invention and the existing salient object segmentation method on multiple public data sets provided by an embodiment of the present invention;

12 is a schematic diagram of a visualization result after processing an image to be processed by the image processing method of the present invention and the existing salient object segmentation method provided by the embodiment of the present invention;

FIG. 13 is a schematic diagram of an image processing apparatus according to an embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1:

First, an electronic device 100 for implementing an embodiment of the present invention is described with reference to FIG. 2 , and the electronic device can be used to execute the image processing methods of the embodiments of the present invention.

As shown in FIG. 2, electronic device 100 includes one or more processors 102, one or more memories 104, input devices 106, output devices 108, and cameras 110, these components being connected by a bus system 112 and/or other forms of connection mechanisms (not shown) interconnect. It should be noted that the components and structures of the electronic device 100 shown in FIG. 2 are only exemplary and not restrictive, and the electronic device may also have other components and structures as required.

The processor 102 may use a digital signal processor (DSP, Digital Signal Processing), a Field-Programmable Gate Array (FPGA, Field-Programmable Gate Array), a Programmable Logic Array (PLA, Programmable Logic Array) and an ASIC (Application Specific) Integrated Circuit) in at least one form of hardware, the processor 102 may be a central processing unit (CPU, Central Processing Unit) or other forms of processing units with data processing capability and/or instruction execution capability, and can control other components in the electronic device 100 to perform the desired functions.

The memory 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory, or the like. The non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of the present invention described below. and/or other desired functionality. Various application programs and various data, such as various data used and/or generated by the application program, etc. may also be stored in the computer-readable storage medium.

The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, mouse, microphone, touch screen, and the like.

The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

The camera 110 is used to collect the to-be-processed image, wherein the to-be-processed image collected by the camera is processed by the image processing method to obtain a salient object mask corresponding to the to-be-processed image. For example, the camera can photograph the user. A desired image (such as a photo, video, etc.), and then, the image is processed by the image processing method to obtain a salient object mask corresponding to the image to be processed, and the camera can also store the captured image in the in memory 104 for use by other components.

Exemplarily, the electronic device for implementing the image processing method according to the embodiment of the present invention may be implemented as a smart mobile terminal such as a smart phone, a tablet computer, or the like.

Example 2:

According to an embodiment of the present invention, an embodiment of an image processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although A logical order is shown in the flowcharts, but in some cases steps shown or described may be performed in an order different from that herein.

FIG. 3 is a flowchart of an image processing method according to an embodiment of the present invention. As shown in FIG. 3 , the method includes the following steps:

Step S302, acquiring the image to be processed, and performing feature extraction on the image to be processed to obtain a multi-scale feature map;

In the embodiment of the present invention, the above-mentioned multi-scale feature maps represent feature maps with different sizes (ie, different heights and widths). Feature extraction can be a multi-layer convolution downsampling process. Each time the image to be processed is downsampled by one layer, a feature map of one scale can be obtained. The feature map of this scale includes multiple sub-feature maps, which are actually multi-channel. Matrix (the matrix of each channel is a two-dimensional matrix), the matrix of each channel can correspond to a sub-feature map, where the number of elements in each row of the matrix represents the width of the sub-feature map corresponding to the channel, and the matrix The number of elements in each column represents the height of the sub-feature map corresponding to the channel.

Step S304, performing enhancement processing on the part corresponding to the salient object in the multi-scale feature map to obtain a multi-scale enhanced feature map;

The inventor considers that a salient object is only related to the pixel values of its neighborhood, and the more prominent the salient object is in the image, the higher the accuracy of the salient object mask obtained by segmentation. Therefore, after obtaining the multi-scale feature map, the parts corresponding to the salient objects in the multi-scale feature map are enhanced, and the features corresponding to the salient objects are more prominent after the part corresponding to the salient objects is enhanced. After enhancing the feature map for image restoration, the obtained salient object mask will be more accurate.

The process of the enhancement processing will be described in detail below, and will not be repeated here.

Step S306, performing image restoration on the multi-scale enhanced feature map to obtain a salient object mask corresponding to the image to be processed.

Specifically, the enhanced feature maps of various scales are up-sampled, and during the up-sampling process, the enhanced feature maps of different scales are fused to obtain a salient object mask corresponding to the image to be processed.

In the embodiment of the present invention, first, an image to be processed is acquired, and feature extraction is performed on the image to be processed to obtain a multi-scale feature map; then, parts corresponding to salient objects in the multi-scale feature map are enhanced to obtain Multi-scale enhanced feature map; finally, image restoration is performed on the multi-scale enhanced feature map to obtain a salient object mask corresponding to the image to be processed. It can be seen from the above description that after the enhancement processing is performed on the part corresponding to the salient objects in the multi-scale feature map, the multi-scale enhanced feature map obtained, the feature map corresponding to the salient objects is more prominent, and finally the multi-scale enhanced feature map is obtained. After image restoration is performed by enhancing the feature map, the salient object mask obtained by segmentation is more accurate, which alleviates the technical problem of poor accuracy of the existing salient object segmentation methods when processing images.

The above content briefly introduces the image processing method of the present invention, and the specific content involved is described in detail below.

In an optional embodiment of the present invention, in step S302, the step of performing feature extraction on the image to be processed includes the following processes (1)-(2):

(1) Multi-layer downsampling is performed on the image to be processed to obtain a multi-scale original feature map;

Specifically, referring to FIG. 4 , the multi-layer downsampling can be the downsampling of multi-layer convolutions. After each layer of downsampling, an original feature map of one scale can be obtained. In this way, an original feature map of multiple scales can be obtained.

It should be noted that, through experiments, it is found that the high-dimensional original feature map in the multi-scale original feature map (such as the original feature map obtained by downsampling above the third layer) has a greater impact on the accuracy of image processing, and the low-dimensional original feature map (For example, the original feature map obtained by the first layer downsampling and the second layer downsampling) has little effect on the accuracy of image processing. In order to improve the efficiency of image processing, it is not necessary to consider the first layer downsampling and the second layer downsampling. As a result, the subsequent process does not process the original feature maps obtained by the first layer downsampling and the second layer downsampling, as shown in Figure 4.

(2) The high-dimensional original feature map in the multi-scale original feature map is optimized to obtain a multi-scale feature map.

Specifically, it includes the following processes:

21) Perform a first optimization process on the target original feature map in the multi-scale original feature map to obtain the first optimized feature map, wherein the target original feature map is the multi-scale original feature map, except the highest-dimensional original feature map. High-dimensional original feature maps other than ;

Referring to FIG. 4 , the first optimization process, that is, the SRB process in FIG. 4 , is performed on all high-dimensional original feature maps except the highest-dimensional original feature map in the multi-scale original feature map. The specific process of the first optimization processing (SRB processing) is: using a first optimization module to perform optimization processing on the target original feature map to obtain a first initial optimized feature map, wherein the first optimization module includes: a preset number of first Convolutional layer; the first initial optimized feature map and its corresponding target original feature map are added to obtain the first optimized feature map.

In this embodiment of the present invention, the preset number of first convolutional layers may be two 3×3 convolutional layers. Referring to FIG. 5 , after the target original feature map passes through two 3×3 convolutional layers in series, the first initial The optimized feature map, and then adding the first initial optimized feature map and its corresponding target original feature map, the first optimized feature map can be obtained.

The fact that the feature map is actually a multi-dimensional matrix has been explained above, so the process of adding the first initial optimized feature map and its corresponding target original feature map is actually the process of adding the corresponding elements in the multi-dimensional matrix. .

22) performing a second optimization process on the highest-dimensional original feature map in the multi-scale original feature map to obtain a second optimized feature map;

Referring to FIG. 4 , a second optimization process, that is, the GRB process in FIG. 4 , is performed on the highest-dimensional original feature map in the multi-scale original feature map. The specific process of the second optimization processing (GRB processing) is: using the second optimization module to optimize the highest-dimensional original feature map to obtain the optimization weight, wherein the second optimization module includes: a second convolution layer, a global pooling layer and Sigmoid function processing layer; multiply the optimization weight and the highest-dimensional original feature map to obtain the second initial optimized feature map; add the second initial optimized feature map and the highest-dimensional original feature map to obtain the second initial optimized feature map Optimized feature map.

In this embodiment of the present invention, the second convolutional layer may be a 1×1 convolutional layer. Referring to FIG. 6 , the highest-dimensional original feature map sequentially passes through a 1×1 convolutional layer, a global pooling layer, a 1×1 convolutional layer, and the Sigmoid function. Then, the optimized weight is obtained, and the optimized weight is then multiplied with the original feature map of the highest dimension, and the result obtained by the operation is added to the original feature map of the highest dimension to obtain the second optimized feature map.

Similarly, the product operation is the product operation process of the optimization weight and the elements in the two-dimensional matrix, and the sum operation is the process of adding the corresponding elements in the two-dimensional matrix.

23) The first optimized feature map and the second optimized feature map are used as multi-scale feature maps.

The process of enhancing the part corresponding to the salient objects in the multi-scale feature map will be described in detail below.

In an optional embodiment of the present invention, referring to FIG. 7, step S304, the step of performing enhancement processing on the part corresponding to the salient object in the multi-scale feature map includes:

Step S701, obtaining the initial position of the salient object according to the multi-scale feature map;

Specifically, it includes the following processes: performing dimensionality reduction on the feature map with the highest dimension in the multi-scale feature map to obtain a single-channel feature map; binarizing the single-channel feature map to obtain a single-channel binarized feature map ; Determine the initial position of the salient object based on the single-channel binarized feature map.

Referring to FIG. 4 , the second optimized feature map (ie, the feature map with the highest dimension in the multi-scale feature map) after GRB processing (second optimization process) is subjected to dimensionality reduction processing through the convolution layer to obtain a single-channel feature map. (referring to N*M, the number of channels is 1), and then the single-channel feature map is binarized to obtain a single-channel binarized feature map (the image below the GRB in Figure 4), In the above single-channel binarization feature map, the part represented by 1 is the part corresponding to the salient object, and then the initial position of the salient object can be determined according to the position of 1. The initial position may be a position determined by the leftmost 1, the rightmost 1, the uppermost 1, and the lowermost 1 in the single-channel binarized feature map.

Step S702, according to the initial position, with at least two different expansion scales, crop the feature map of the highest dimension in the multi-scale feature map, and obtain a plurality of cropped feature maps, and the plurality of cropped feature maps contain feature information of salient objects;

Specifically, it includes the following processes: determine the pixel width and pixel height of the salient object according to the initial position; determine the extended pixel width and height according to the extended scale, pixel width and pixel height; in the feature map of the highest dimension, along the initial position The position after expanding the expanded pixel width and expanded pixel height is cropped. The cropped feature map is based on the initial position of the object and is expanded and cropped to a certain extent, so it contains salient objects, and also contains the feature information of salient objects.

For example, when the expansion scale is 10%, if the pixel width and pixel height of the salient object are 30*30, then the expanded pixel width (ie 10% of 30) is 3 pixels, and the expanded pixel height (ie 10 of 30) %) is 3 pixels, that is, in the feature map of the highest dimension, the initial position is expanded (that is, increased) by 3 background pixels in the left and right directions, and the initial position is expanded by 3 backgrounds in the upper and lower directions. After the pixels are cropped, the reason for such expansion is that the accuracy of the feature map corresponding to the salient objects obtained by direct binarization is poor, and it is not necessarily all the salient objects, so it needs to be expanded. to include all of the significant objects.

The above-mentioned expansion scale may also be 30%, 50%, etc., and the embodiment of the present invention does not specifically limit the above-mentioned expansion scale. Different cropped feature maps will be cropped according to different expansion scales, so the number of expanded scales is equal to the number of cropped feature maps.

Step S703, taking one or more of the multi-scale feature maps as the target feature map, using each target feature map as the current target feature map one by one, calculating the correlation between the multiple cropped feature maps and the current target feature map, and obtaining the current feature. A plurality of correlation feature maps corresponding to a plurality of cropped feature maps one-to-one in the graph;

The foregoing target feature maps may be one or more of multi-scale feature maps, which are not specifically limited in this embodiment of the present invention.

Specifically, it includes the following process: scaling multiple cropped feature maps to a preset scale to obtain multiple cropped feature maps of the preset scale; taking each target feature map as the current target feature map one by one, and using the preset scale as the sliding window to move the current target Sliding on the feature map; the feature map contained in the sliding window after each sliding is multiplied with each cropped feature map in multiple cropped feature maps of the preset scale, and the current target feature map is obtained according to the result of the multiplication operation. Multiple correlation feature maps corresponding to multiple cropped feature maps one-to-one.

In order to better understand the process, the above process is described below with a specific example: if the cropped feature map of the preset scale is the B map of 32*32*64, and the current target feature map is 64*64*128 A picture.

In order to simplify the description, the B1 picture of 32*32*1 and the A1 picture of 64*64*1 are used for illustration. On the A1 picture, the sliding window of 32*32 is used to slide in sequence according to the preset sliding step, and each slide is once. , you can get a 32*32 small block, the small block is multiplied with the 32*32 B1 map, and a new 32*32 small block is obtained. After all the sliding is completed, the correlation between the A1 map and the B1 map can be obtained. degree feature map, the size of the correlation feature map is 64*64*1;

When calculating the correlation feature map of the 32*32*1 B2 image and the 64*64*128 A2 image, on each channel of the A2 image, use 32*32 as the sliding window and follow the preset sliding step size. Sliding, every time you slide, you can get a 32*32 small block, which is multiplied with the 32*32 B2 map to get a new 32*32 small block. The correlation feature map of A2 and B2 can be obtained, and the size of the correlation feature map is 64*64*128;

When calculating the correlation feature map of 32*32*64 B picture and 64*64*128 A picture, each dimension of 32*32*64 is calculated with 64*64*128 according to the above process, and a 64*64*128 correlation feature map, and then add these 64 together to get a final 64*64*128 correlation feature map, which is the correlation feature map of A and B.

Step S704, obtaining an enhanced feature map corresponding to the current target feature map according to the plurality of correlation feature maps and the current target feature map.

Referring to Figure 8, it specifically includes the following process:

Step S801, performing a product operation on each correlation feature map in the plurality of correlation feature maps and the current target feature map to obtain a plurality of first enhanced feature maps corresponding to the current target feature map;

After the above product operation is performed, the part corresponding to the salient objects in the current target feature map is strengthened.

Step S802, connecting a plurality of first enhanced feature maps with the current target feature map in series to obtain a second enhanced feature map corresponding to the current target feature map;

For example: multiple first enhanced feature maps are two feature maps of 64*64*128 (corresponding to the width W*height H*channel number C of the map respectively), which are concatenated with the target feature map of 64*64*128 to obtain The second enhanced feature map of 64*64*384, the concatenation is the addition of the number of channels.

Step S803, obtaining the position enhancement feature map corresponding to the current target feature map, and connecting the second enhancement feature map and the position enhancement feature map in series to obtain an enhancement feature map corresponding to the target feature map, wherein the scale of the position enhancement feature map is the same as The scale of the second enhanced feature map is the same.

The specific process is as follows:

a) On the target matrix, determine the center line of the salient object in the X direction and the center line in the Y direction based on the initial position of the salient object. The target matrix is a single-channel matrix with the same scale as the second enhanced feature map. The value of an element can be 0;

b) setting the center line in the Y direction of the target matrix as the first target value, linearly transforming it into the second target value along the X direction, and obtaining a position enhancement feature map in the X direction;

c) setting the center line in the X direction of the target matrix as the first target value, linearly transforming it into the second target value along the Y direction, and obtaining the position enhancement feature map in the Y direction;

d) The position enhancement feature map in the X direction and the position enhancement feature map in the Y direction are used as the position enhancement feature map.

The above-mentioned first target value may be 1, and the second target value may be 0. A schematic diagram of the location-enhanced feature map obtained based on the initial location of the salient object is shown in Figure 9.

The above enhancement process (represented by LCB) can refer to Figure 10, according to the single-channel binarized feature map to determine the positioning information of the corresponding part of the salient object (including the initial position of the salient object, pixel width, pixel height, X The center line of the multi-scale feature map and the center line of the Y direction), and then according to the positioning information, with at least two different expansion scales, crop the feature map of the highest dimension in the multi-scale feature map, and obtain multiple cropped feature maps. At the same time, according to the positioning information Determine the position enhancement feature map, and then calculate the correlation feature map of the multiple cropped feature maps and the current target feature map in the multi-scale feature map, and multiply the correlation feature map and the current target feature map. The current target feature map and the position enhancement feature map corresponding to the current target feature map are connected in series, and finally the enhanced feature map corresponding to the current target feature map is obtained, thereby obtaining the enhanced feature map corresponding to each target feature map.

The enhanced feature map greatly strengthens the feature parts of the salient objects, so that the salient object masks obtained by segmentation have higher accuracy.

In an optional embodiment of the present invention, up-sampling the multi-scale enhanced feature maps to obtain a salient object mask corresponding to the to-be-processed image includes: fusing the enhanced feature maps of different scales to obtain a Process the salient object mask corresponding to the image.

In the embodiment of the present invention, the process of upsampling and fusion may be as follows: as shown in FIG. 4 , the enhanced feature map of the fifth layer (that is, the feature map obtained after downsampling of the fifth layer is processed by GRB, and then obtained after processing by LCB) The feature map of the , and then processed by SRB, and then upsampled after processing. After the upsampled feature map is added to the enhanced feature map of the third layer, and then processed by SRB, the salient object mask is finally obtained by 4 times magnification.

The inventor compares the image processing method of the present invention (represented by LCANet) and the existing saliency object segmentation method in multiple public datasets (DUTS-TE dataset, ECSSD dataset, HKU-IS dataset, PASCAL-S dataset respectively). Data set and DUT-OM data set) were trained and tested, and the results are shown in Figure 11, indicating that when the image processing method of the present invention is used to process the image, the obtained salient object mask has higher accuracy ( In FIG. 11 , for the maxF parameter, the larger the value, the higher the accuracy, and the smaller the value of the MAE parameter, the higher the accuracy). In addition, referring to FIG. 12 , it can also be seen from the visualization results that the image processing method of the present invention has better accuracy. In Figure 12, the GT column represents the segmentation result of salient objects marked manually, the LCANet column represents the salient object segmentation result of the present invention, and the other columns represent the salient objects obtained by other methods (the corresponding methods are marked below each column). It can be seen from the comparison that the segmentation result of the salient object obtained by the method of the present invention is closer to the segmentation result of the salient object marked manually, which means that the method of the present invention is more accurate than other existing methods.

Example 3:

An embodiment of the present invention further provides an image processing apparatus, which is mainly used to execute the image processing method provided by the above content of the embodiment of the present invention. The following describes the image processing apparatus provided by the embodiment of the present invention in detail.

13 is a schematic diagram of an image processing apparatus according to an embodiment of the present invention. As shown in FIG. 13 , the image processing apparatus mainly includes: a feature extraction unit 10, an enhancement processing unit 20 and an image restoration unit 30, wherein:

The feature extraction unit is used to obtain the image to be processed, and perform feature extraction on the image to be processed to obtain a multi-scale feature map;

The enhancement processing unit is used to enhance the part corresponding to the salient objects in the multi-scale feature map to obtain the multi-scale enhanced feature map;

The image restoration unit is used for image restoration of the multi-scale enhanced feature map to obtain a salient object mask corresponding to the image to be processed.

In the embodiment of the present invention, first, an image to be processed is acquired, and feature extraction is performed on the image to be processed to obtain a multi-scale feature map; then, parts corresponding to salient objects in the multi-scale feature map are enhanced to obtain Multi-scale enhanced feature map; finally, image restoration is performed on the multi-scale enhanced feature map to obtain a salient object mask corresponding to the image to be processed. It can be seen from the above description that after the enhancement processing is performed on the part corresponding to the salient objects in the multi-scale feature map, the multi-scale enhanced feature map obtained, the feature map corresponding to the salient objects is more prominent, and finally the multi-scale enhanced feature map is obtained. After image restoration is performed by enhancing the feature map, the salient object mask obtained by segmentation is more accurate, which alleviates the technical problem of poor accuracy of the existing salient object segmentation methods when processing images.

Optionally, the feature extraction unit is further configured to: perform multi-layer downsampling processing on the image to be processed to obtain a multi-scale original feature map; and perform optimization processing on the multi-scale original feature map to obtain a multi-scale feature map.

Optionally, the feature extraction unit is further configured to: perform a first optimization process on the target original feature map in the multi-scale original feature map to obtain a first optimized feature map, wherein the target original feature map is a multi-scale original feature In the figure, the feature maps except the highest-dimensional original feature map; the second optimization process is performed on the highest-dimensional original feature map in the multi-scale original feature map to obtain the second optimized feature map; the first optimized feature map and The second optimized feature map is used as a multi-scale feature map.

Optionally, the feature extraction unit is further configured to: use a first optimization module to perform optimization processing on the target original feature map to obtain a first initial optimized feature map, wherein the first optimization module includes: a preset number of first convolutions layer; the first initial optimized feature map and its corresponding target original feature map are added to obtain the first optimized feature map.

Optionally, the feature extraction unit is further configured to: use a second optimization module to perform optimization processing on the highest-dimensional original feature map to obtain optimization weights, wherein the second optimization module includes: a second convolution layer, a global pooling layer, and a Sigmoid Function processing layer; Multiply the optimization weight and the highest-dimensional original feature map to obtain the second initial optimized feature map; add the second initial optimized feature map and the highest-dimensional original feature map to obtain the second optimized feature map feature map.

Optionally, the enhancement processing unit is further configured to: obtain the initial position of the salient object according to the multi-scale feature map; according to the initial position, crop the feature map of the highest dimension in the multi-scale feature map with at least two different expansion scales, Obtain multiple cropped feature maps, which contain feature information of salient objects; use one or more of the multi-scale feature maps as target feature maps, and use each target feature map as the current target feature map one by one, Calculate the correlation between the multiple cropped feature maps and the current target feature map, and obtain multiple correlation feature maps of the current target feature map that correspond one-to-one with the multiple cropped feature maps; according to the multiple correlation feature maps and the current target feature map , to obtain the enhanced feature map corresponding to the current target feature map.

Optionally, the enhancement processing unit is further configured to: perform dimensionality reduction processing on the feature map with the highest dimension in the multi-scale feature map to obtain a single-channel feature map; perform binarization processing on the single-channel feature map to obtain a single-channel feature map The binarized feature map of ; determine the initial position of the salient object according to the single-channel binarized feature map.

Optionally, the enhancement processing unit is also used to: determine the pixel width and pixel height of the salient object according to the initial position; determine the expanded pixel width and the expanded pixel height according to the expanded scale, the pixel width and the pixel height; in the feature map of the highest dimension , and crop along the position where the original position is expanded by the expanded pixel width and expanded pixel height.

Optionally, the enhancement processing unit is further configured to: scale multiple cropped feature maps to a preset scale to obtain multiple cropped feature maps of the preset scale; use the preset scale as a sliding window to slide on the current target feature map; Multiply the feature map included in the sliding window after each sliding with multiple cropped feature maps of the preset scale, respectively, and obtain multiple cropped feature maps of the current target feature map one-to-one corresponding to the multiple cropped feature maps according to the result of the multiplication operation. Correlation feature map.

Optionally, the enhancement processing unit is further configured to: perform a product operation on each correlation feature map in the plurality of correlation feature maps and the current target feature map to obtain a plurality of first enhanced feature maps corresponding to the current target feature map; Connect a plurality of first enhanced feature maps and the current target feature map in series to obtain a second enhanced feature map corresponding to the current target feature map; obtain the position enhanced feature map corresponding to the current target feature map, and combine the second enhanced feature map and the location enhanced feature map The feature maps are concatenated to obtain an enhanced feature map corresponding to the current target feature map, wherein the scale of the position enhanced feature map is the same as the scale of the second enhanced feature map.

Optionally, the enhancement processing unit is further configured to: determine the center line in the X direction and the center line in the Y direction of the salient object based on the initial position of the salient object; set the center line in the Y direction as the first target value, along the X direction. The direction is linearly transformed into the second target value, and the position enhancement feature map in the X direction is obtained; the center line in the X direction is set as the first target value, and the second target value is linearly transformed along the Y direction to obtain the position enhancement feature in the Y direction. Figure; the position enhancement feature map in the X direction and the position enhancement feature map in the Y direction are used as the position enhancement feature map.

Optionally, the image restoration unit is further configured to: perform up-sampling on the multi-scale enhanced feature map to obtain a salient object mask corresponding to the image to be processed.

The image processing device provided by the embodiment of the present invention has the same implementation principle and technical effect as the method embodiment in the foregoing embodiment 2. For the sake of brief description, the parts not mentioned in the embodiment of the device can be implemented with reference to the foregoing method. corresponding content in the example.

In another embodiment, there is also provided a computer-readable medium having non-volatile program code executable by a processor, the program code causing the processor to execute any of the above-mentioned embodiments in Embodiment 2. steps of the method described.

In addition, in the description of the embodiments of the present invention, unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it may be a fixed connection or a detachable connection , or integrally connected; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or the internal communication between the two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in specific situations.

In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. The indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must have a specific orientation or a specific orientation. construction and operation, and therefore should not be construed as limiting the invention. Furthermore, the terms "first", "second", and "third" are used for descriptive purposes only and should not be construed to indicate or imply relative importance.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above may refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. The apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on such understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present invention, and are used to illustrate the technical solutions of the present invention, but not to limit them. The protection scope of the present invention is not limited thereto, although referring to the foregoing The embodiment has been described in detail the present invention, those of ordinary skill in the art should understand: any person skilled in the art who is familiar with the technical field within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments. Or can easily think of changes, or equivalently replace some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be covered in the present invention. within the scope of protection. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (15)

1. An image processing method, comprising:

acquiring an image to be processed, and performing feature extraction on the image to be processed to obtain a multi-scale feature map;

performing reinforcement processing on a part corresponding to the salient object in the multi-scale feature map to obtain a multi-scale reinforced feature map;

and carrying out image restoration on the multi-scale enhanced feature map to obtain a significant object mask corresponding to the image to be processed.

2. The method of claim 1, wherein feature extracting the image to be processed comprises:

carrying out multilayer downsampling processing on the image to be processed to obtain a multi-scale original characteristic diagram;

and optimizing the multi-scale original characteristic diagram to obtain the multi-scale characteristic diagram.

3. The method of claim 1 or 2, wherein optimizing the multi-scale raw feature map comprises:

performing first optimization processing on a target original feature map in the multi-scale original feature map to obtain a first optimized feature map, wherein the target original feature map is a feature map in the multi-scale original feature map except for a highest-dimension original feature map;

performing second optimization processing on the highest-dimensional original feature map in the multi-scale original feature map to obtain a second optimized feature map;

and taking the first optimized feature map and the second optimized feature map as the multi-scale feature map.

4. The method of claim 3, wherein performing the first optimization process on the target original feature map in the multi-scale original feature map comprises:

optimizing the target original feature map by using a first optimization module to obtain a first initially optimized feature map, wherein the first optimization module comprises: presetting a number of first winding layers;

and adding the first initially optimized feature map and the corresponding target original feature map to obtain the first optimized feature map.

5. The method according to claim 3 or 4, wherein performing the second optimization process on the highest-dimensional original feature map in the multi-scale original feature maps comprises:

optimizing the highest-dimensional original feature map by using a second optimization module to obtain an optimized weight, wherein the second optimization module comprises: the second convolution layer, the global pooling layer and the Sigmoid function processing layer;

performing product operation on the optimization weight and the highest-dimensional original feature map to obtain a second initially optimized feature map;

and adding the second initially optimized feature map and the highest-dimension original feature map to obtain the second optimized feature map.

6. The method according to any one of claims 1 to 5, wherein the enhancing the portion of the multi-scale feature map corresponding to the salient object comprises:

obtaining the initial position of the salient object according to the multi-scale feature map;

according to the initial position, cutting the feature map with the highest dimension in the multi-scale feature map with at least two different expansion scales to obtain a plurality of cut feature maps, wherein the plurality of cut feature maps comprise feature information of the salient object;

taking one or more of the multi-scale feature maps as target feature maps, taking each target feature map as a current target feature map one by one, and calculating the correlation degree between the plurality of cutting feature maps and the current target feature map to obtain a plurality of correlation degree feature maps of the current target feature map, which are in one-to-one correspondence with the plurality of cutting feature maps;

and obtaining a reinforced feature map corresponding to the current target feature map according to the plurality of correlation feature maps and the current target feature map.

7. The method of claim 6, wherein obtaining the initial position of the salient object from the multi-scale feature map comprises:

performing dimensionality reduction processing on the feature map with the highest dimension in the multi-scale feature map to obtain a single-channel feature map;

carrying out binarization processing on the single-channel feature map to obtain a single-channel binarization feature map;

and determining the initial position of the salient object according to the binarization feature map of the single channel.

8. The method of claim 6 or 7, wherein cropping the highest-dimensional feature map of the multi-scale feature map at least two different expansion scales according to the initial position comprises:

determining the pixel width and the pixel height of the salient object according to the initial position;

determining an extended pixel width and an extended pixel height according to the extended scale, the pixel width and the pixel height;

and in the feature map with the highest dimension, clipping is carried out along the position of the initial position after the expansion pixel width and the expansion pixel height are expanded.

9. The method of any of claims 6-8, wherein calculating the relevance of the plurality of cropped feature maps to the current target feature map comprises:

zooming the plurality of cutting feature maps to a preset scale to obtain a plurality of cutting feature maps with the preset scale;

sliding on the current target feature map by taking the preset scale as a sliding window;

and respectively carrying out product operation on the feature graph contained in the sliding window after each sliding and the plurality of cutting feature graphs with the preset scale, and obtaining a plurality of relevancy feature graphs, corresponding to the plurality of cutting feature graphs, of the current target feature graph according to the result of the product operation.

10. The method according to any one of claims 6 to 9, wherein obtaining the enhanced feature map corresponding to the current target feature map according to the plurality of correlation feature maps and the current target feature map comprises:

performing product operation on each correlation degree feature map in the plurality of correlation degree feature maps and the current target feature map to obtain a plurality of first strengthened feature maps corresponding to the current target feature map;

connecting the plurality of first enhanced feature maps with the current target feature map in series to obtain a second enhanced feature map corresponding to the current target feature map;

and acquiring a position strengthening feature map corresponding to the current target feature map, and connecting the second strengthening feature map and the position strengthening feature map in series to obtain a strengthening feature map corresponding to the current target feature map, wherein the scale of the position strengthening feature map is the same as that of the second strengthening feature map.

11. The method of claim 10, wherein obtaining the location-enhanced feature map corresponding to the current target feature map comprises:

determining a central line of the salient object in an X direction and a central line of the salient object in a Y direction based on the initial position of the salient object;

setting the central line in the Y direction as a first target value, and linearly converting the central line in the Y direction into a second target value along the X direction to obtain a position strengthening feature map in the X direction;

setting the central line in the X direction as the first target value, and linearly converting the central line in the X direction into the second target value along the Y direction to obtain a position strengthening feature map in the Y direction;

and taking the position strengthening characteristic diagram in the X direction and the position strengthening characteristic diagram in the Y direction as the position strengthening characteristic diagrams.

12. The method of any one of claims 1-11, wherein image restoring the multi-scale enhanced feature map comprises:

and performing up-sampling on the multi-scale enhanced feature map to obtain a significant object mask corresponding to the image to be processed.

13. An image processing apparatus characterized by comprising:

the device comprises a characteristic extraction unit, a feature extraction unit and a feature extraction unit, wherein the characteristic extraction unit is used for acquiring an image to be processed and extracting the characteristics of the image to be processed to obtain a multi-scale characteristic map;

the strengthening processing unit is used for strengthening the part corresponding to the salient object in the multi-scale feature map to obtain a multi-scale strengthening feature map;

and the image restoration unit is used for carrying out image restoration on the multi-scale enhanced characteristic image to obtain a significant object mask corresponding to the image to be processed.

14. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the method of any of the preceding claims 1 to 12 are implemented when the computer program is executed by the processor.

15. A computer-readable medium having non-volatile program code executable by a processor, characterized in that the program code causes the processor to perform the steps of the method of any of the preceding claims 1 to 12.

Images