WO2022111070A1

WO2022111070A1 - Image processing method and apparatus, and electronic device and storage medium

Info

Publication number: WO2022111070A1
Application number: PCT/CN2021/122901
Authority: WO
Inventors: 吴佳涛; 郭彦东; 李亚乾; 杨林
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-11-26
Filing date: 2021-10-09
Publication date: 2022-06-02
Also published as: CN112329889A

Abstract

Disclosed in embodiments of the present application are an image processing method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring a first image and a second image; acquiring a first global feature and a first local feature map, wherein the first global feature is a global feature corresponding to the first image, and the first local feature map comprises a local feature corresponding to the first image; acquiring a second global feature and a second local feature map, wherein the second global feature is a global feature corresponding to the second image, and the second local feature map comprises a local feature corresponding to the second image; and determining, on the basis of the first global feature, the first local feature map, the second global feature and the second local feature map, whether the first image is similar to the second image. Thus, combining global features and local features of a first image and a second image to perform similarity detection can realize more comprehensive detection in an image similarity detection process, and the accuracy of similarity detection is improved.

Description

Image processing method, device, electronic device and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Chinese Application No. 202011356038.4 filed on November 26, 2020, the entire contents of which are hereby incorporated by reference for all purposes.

technical field

The present application relates to the field of computer technology, and more particularly, to an image processing method, apparatus, electronic device, and storage medium.

Background technique

Usually images can have some features of their own, and then some similar images can be obtained by comparing some features of the images.

SUMMARY OF THE INVENTION

In view of the above problems, the present application proposes an image processing method, apparatus, electronic device and storage medium to improve the above problems.

In a first aspect, the present application provides an image processing method, the method includes: acquiring a first image and a second image; acquiring a first global feature and a first local feature map, where the first global feature is the first global feature. A global feature corresponding to an image, the first local feature map includes local features corresponding to the first image; a second global feature and a second local feature map are obtained, and the second global feature corresponds to the second image. The global features of the second image, the second local feature map includes the local features corresponding to the second image; Whether the first image is similar to the second image.

In a second aspect, the present application provides an image processing method, the method includes: acquiring a first image and a first image set; acquiring a first global feature and a first local feature map, where the first global feature is the The global feature corresponding to the first image, and the first local feature map includes the local feature corresponding to the first image; obtain the global feature and local feature map corresponding to each image in the first image set; a global feature and a global feature corresponding to each image in the first image set, determine a second image set from the first image set; based on the first local feature map and each image in the second image set A local feature map of each image, to determine an image in the second image set that is similar to the first image.

In a third aspect, the present application provides an image processing device, the device includes: an image acquisition unit for acquiring a first image and a second image; a feature acquisition unit for acquiring a first global feature and a first local feature The first global feature is a global feature corresponding to the first image, and the first local feature map includes local features corresponding to the first image; the feature acquiring unit is further configured to acquire a second global feature feature and a second local feature map, the second global feature is the global feature corresponding to the second image, and the second local feature map includes the local feature corresponding to the second image; the image comparison unit is used for Based on the first global feature, the first local feature map, the second global feature, and the second local feature map, it is determined whether the first image and the second image are similar.

In a fourth aspect, the present application provides an image processing device, the device comprising: an image acquisition unit for acquiring a first image and a first image set; a feature acquisition unit for acquiring a first global feature and a first local feature a feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes local features corresponding to the first image; the feature acquisition unit is further configured to acquire the A global feature and a local feature map corresponding to each image in the first image set; the first image determination unit is configured to, based on the first global feature and the global feature corresponding to each image in the first image set, from the determining a second image set from the first image set; a second image determining unit, configured to determine the second image based on the first local feature map and the local feature map of each image in the second image set An image in the collection that is similar to the first image.

In a fifth aspect, the present application provides an electronic device including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above method.

In a sixth aspect, the present application provides a computer-readable storage medium, where a program code is stored in the computer-readable storage medium, wherein the above-mentioned method is executed when the program code is executed by a startup controller.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 shows a schematic diagram of an application scenario of an image processing method proposed by an embodiment of the present application;

FIG. 2 shows a flowchart of an image processing method proposed by an embodiment of the present application;

Fig. 3 shows a flowchart included in an embodiment of S120 in Fig. 2;

FIG. 4 shows a schematic diagram of obtaining a global feature in an embodiment of the present application;

5 shows a schematic diagram of a feature set corresponding to a position in a first initial feature map in an embodiment of the present application;

FIG. 6 shows a schematic diagram of obtaining local features in an embodiment of the present application;

Fig. 7 shows a flowchart included in an embodiment of S130 in Fig. 2;

FIG. 8 shows a flowchart of an image processing method proposed by another embodiment of the present application;

9 shows a flowchart of determining whether the first image and the second image are similar based on the first global feature, the first local feature map, the second global feature, and the second local feature map in an embodiment of the present application;

FIG. 10 shows a schematic diagram of position correspondence in the embodiment of the present application;

FIG. 11 shows a flowchart of an image processing method proposed by still another embodiment of the present application;

FIG. 12 shows a structural block diagram of a data processing apparatus proposed by an embodiment of the present application;

FIG. 13 shows a structural block diagram of a data processing apparatus proposed by another embodiment of the present application;

FIG. 14 shows a structural block diagram of a data processing apparatus proposed by still another embodiment of the present application;

FIG. 15 shows a structural block diagram of the electronic device of the present application for executing the image processing method according to the embodiment of the present application;

FIG. 16 is a storage unit for storing or carrying a program code for implementing the image processing method according to the embodiment of the present application according to the embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

As the number of images increases, there are more applications for images. For example, it is possible to judge whether two images are similar by comparing the features included in the images, or to use a certain image as a benchmark to filter out an image that is similar to the benchmark image from a set of images.

However, the inventor found in the research that in the related process of acquiring similar images, there is still a problem that the similar images cannot be acquired more accurately.

Therefore, the inventor proposes an image processing method, device, electronic device and storage medium in the present application that can improve the above-mentioned problems. After acquiring the first image and the second image, the first global feature and the first local feature are further acquired. feature map, and obtain a second global feature and a second local feature map, and then determine the first image and the first image based on the first global feature, the first local feature map, the second global feature, and the second local feature map. Whether the two images are similar. Therefore, in the process of judging the similarity between the first image and the second image, the similarity detection can be carried out in combination with the global features and local features of the first image and the second image, so that it is possible to perform more similarity detection in the process of image similarity detection. For comprehensive detection, the accuracy of similar detection is improved.

It should be noted that, the image processing method provided by the embodiments of the present application may be executed in electronic devices such as mobile phones and tablet computers, and may also be executed in a server. Among them, the server may be a single physical server, or a server cluster or distributed system composed of multiple physical servers, or may provide cloud services, cloud computing, cloud storage, CDN (Content Delivery Network, Content Delivery Network), And cloud servers for basic cloud computing services such as artificial intelligence platforms. Wherein, in the case where the data processing method provided by the embodiment of the present application is executed by a server cluster or a distributed system composed of multiple physical servers, different steps in the data processing method may be executed by different physical servers respectively, or may be executed by Servers built by distributed systems execute in a distributed manner. For example, as shown in FIG. 1 , acquiring the first image and the second image, and acquiring the first global feature, the first local feature map, the second global feature, and the second local feature map may be performed by the server 110 alone, The subsequent steps can be performed by the server 120 .

Furthermore, the image processing method provided in this embodiment may also be completed by the electronic device and the server in cooperation. Optionally, the acquisition of the first image and the second image can be performed by the electronic device, and then the electronic device sends the acquired first image and the second image to the server, and then the server performs the subsequent acquisition of the first global feature, the first image and the second image. a local feature map, a second global feature, and a second local feature map, and determining the first image and the first image based on the first global feature, the first local feature map, the second global feature, and the second local feature map Whether the two images are similar, and after obtaining the similarity judgment result, the server can return the judgment result to the electronic device.

The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Referring to FIG. 2, an image processing method provided by an embodiment of the present application includes:

S110: Acquire the first image and the second image.

Wherein, the first image and the second image are images for similarity comparison. It should be noted that, the image processing method provided in this embodiment may operate in various scenarios, and correspondingly, the manners of acquiring the first image and the second image are different in different scenarios.

Optionally, the image processing method provided in this embodiment may be run in a network information search scenario. In this scenario, the user can input an image, and then search for images similar to the input image. Correspondingly, the image input by the user may be used as the first image, and the image currently being similarly compared with the first image may be used as the second image. Optionally, the image processing method provided in this embodiment may run in an image management program. In this scenario, the image selected by the user from the image management program may be used as the first image, and the images other than the selected image may be used as the first image. image as the second image.

S120: Acquire a first global feature and a first local feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes local features corresponding to the first image.

Among them, the global feature can be understood as the feature that characterizes the image content as a whole, and correspondingly, the local feature can be understood as the feature that characterizes the image content from the part of the image.

As one way, as shown in FIG. 3 , the obtaining of the first global feature and the first local feature map includes:

S121: Input the first image into a target neural network, and obtain a feature map output by the target neural network as a first initial feature map. Optionally, the target neural network can be a CNN (Convolutional Neural Networks) network. The target neural network in this embodiment may be a model that has been trained in other tasks, such as a classification model, a segmentation model, and the like.

S122: Perform a global average pooling process on the initial feature map to obtain a first global feature.

Optionally, global average pooling can be performed based on the following formula, which is:

Among them, F(i,j) represents the feature vector at position (i,j), w represents the width of the initial feature map, and h represents the height of the initial feature map. Take the initial feature map F of size w*h*d as an example, where w represents the width of the initial feature map, h represents the height of the initial feature map, and d represents the number of channels of the initial feature map. As shown in Figure 4, the initial feature map F of size w*h*d can be processed by global average pooling to obtain global image feature G of size 1*d as the first global feature.

S123: Acquire feature sets corresponding to multiple positions in the initial feature map, where the feature sets corresponding to each position include features of each position and features of adjacent positions of each position.

As one approach, the plurality of locations may be each location in the initial feature map. It should be noted that the initial feature map may correspond to a channel dimension, then the information of each position (i, j) includes the color information of each channel. Exemplarily, as shown in FIG. 5 , the initial feature map 10 shown in FIG. 5 , which is exemplarily shown as position a to position i (the initial feature map 10 may also include positions other than position a to position i) other positions), wherein, for position a, adjacent positions include position b, position e and position d, for example, for position e, adjacent positions include position a, position b, position c, position f , position i, position h, position g, position d.

S124: Perform an average process on the feature sets corresponding to each of the positions to obtain a local feature corresponding to each of the positions.

S125: Obtain the first local feature map based on the local features corresponding to each of the positions.

As shown in Figure 6, if the size of the first image is W*H*3, after processing by the target neural network, an initial feature map F with a size of w*h*d can be obtained, and then the position (i, j ) of the feature set, and then average the local domain of the feature set at the position (i, j) (that is, average processing), you can obtain a local feature of size 1*d, and then the size corresponding to each position is 1 By collecting the local features of *d, the first local feature map corresponding to the first image of size W*H*3 can be obtained.

It should be noted that the local feature corresponding to each position can also be understood as the local feature vector of each position. Optionally, the local feature vector of each position can be calculated by the following formula. The formula is: :

Among them, F(u, v) represents the feature vector at the position (u, v), and M represents the number of adjacent positions corresponding to the position (u, v).

S130: Acquire a second global feature and a second local feature map, where the second global feature is a global feature corresponding to the second image, and the second local feature map includes local features corresponding to the second image.

As one way, as shown in FIG. 7 , the obtaining of the second global feature and the second local feature map includes:

S131: Input the second image into a target neural network, and obtain a feature map output by the target neural network as a second initial feature map.

S132: Perform a global average pooling process on the second initial feature map to obtain a second global feature.

S133: Acquire feature sets corresponding to multiple positions in the second initial feature map, where the feature sets corresponding to each position include features of each position and features of adjacent positions of each position.

S134: Perform averaging processing on the feature sets corresponding to each of the positions to obtain local features corresponding to each of the positions.

S135: Obtain the second local feature map based on the local features corresponding to each of the positions.

It should be noted that the principle of obtaining the second global feature and the second local feature map from the input of the second image to the target neural network is the same as the above-mentioned principle of obtaining the first global feature and the first local feature map. No longer.

S140: Determine whether the first image and the second image are similar based on the first global feature, the first local feature map, the second global feature, and the second local feature map.

In this embodiment, it may be determined whether the first image and the second image are similar based on the first global feature, the first local feature map, the second global feature, and the second local feature map. Optionally, the first global feature and the second global feature can be compared, and the first local feature map and the second local feature map can be compared, and then the first global feature and the second global feature are satisfied. When certain conditions are met, and the first local feature map and the second local feature map also meet certain conditions, it is determined that the first image is similar to the second image, so that in the process of image similarity comparison, a more comprehensive Feature alignment to improve accuracy.

In the image processing method provided in this embodiment, after acquiring the first image and the second image, the first global feature and the first local feature map are further acquired, and the second global feature and the second local feature map are acquired, Further, it is determined whether the first image and the second image are similar based on the first global feature, the first local feature map, the second global feature and the second local feature map. Therefore, in the process of judging the similarity between the first image and the second image, the similarity detection can be carried out in combination with the global features and local features of the first image and the second image, so that it is possible to perform more similarity detection in the process of image similarity detection. For comprehensive detection, the accuracy of similar detection is improved.

Referring to FIG. 8, an image processing method provided by an embodiment of the present application includes:

S210: Acquire the first image and the second image.

S220: Obtain a first global feature and a first local feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes local features corresponding to the first image.

S230: Obtain a second global feature and a second local feature map, where the second global feature is a global feature corresponding to the second image, and the second local feature map includes local features corresponding to the second image.

S240: If the first global feature and the second global feature satisfy a first similarity condition, and the first local feature map and the second local feature map satisfy a second similarity condition, determine that the first image and the second The two images are similar.

In one way, after the first global feature, the first local feature map, the second global feature, and the second local feature map are acquired, the first image and the second image can be determined based on the process shown in the following figure Is it similar, as shown in Figure 9, the process includes:

S250: Detect whether the first global feature and the second global feature satisfy a first similarity condition.

S251: If the first global feature and the second global feature do not satisfy the first similarity condition, determine that the first image and the second image are not similar.

It should be noted that, in this embodiment, the global feature can be understood as a vector representing the overall feature of the image. In this embodiment, the local feature map will include respective local features of multiple regions of the image, that is, the local feature map will include multiple local features. Then the corresponding first global feature can be understood as a vector representing the overall feature of the first image, and the first local feature map includes a plurality of local features representing respective corresponding positions of multiple positions in the first image. Correspondingly, the second global feature can be understood as a vector representing the overall feature of the second image, and the second local feature map includes a plurality of local features representing respective corresponding positions of multiple positions in the second image.

Then, in the process of similarity comparison based on global features, it can be understood as comparing the distance between two vectors. In the process of similarity comparison based on local feature maps, it will be based on multiple feature vectors. For comparison, it can be determined that because there are more features to be compared, similarity comparison based on local features consumes more time and computing resources than similarity comparison based on global features. Then, in order to avoid waste of resources, in the process of comparing the similarity between the first image and the second image, the first global feature can be compared with the second global feature, that is, the first global feature can be determined first. and whether the second global feature satisfies the first similarity condition.

If it is determined that the first global feature and the second global feature satisfy the first similarity condition, it is preliminarily determined that the first image and the second image are likely to be similar, and further to confirm whether they are really similar, then The alignment may be performed based on the first local feature map and the second local feature map. Correspondingly, if it is determined that the first global feature and the second global feature do not satisfy the first similarity condition, it may be directly determined that the first image and the second image are not similar.

S252: If the first global feature and the second global feature satisfy the first similarity condition, detect whether the first local feature map and the second local feature map satisfy the second similarity condition;

S253: If the first local feature map and the second local feature map satisfy a second similarity condition, determine that the first image is similar to the second image.

S254: If the first local feature map and the second local feature map do not satisfy the second similarity condition, determine that the first image and the second image are not similar.

Wherein, optionally, if the distance between the first global feature and the second global feature is less than a first distance threshold, it is determined that the first global feature and the second global feature satisfy a first similarity condition. Wherein, as an approach, the distance between the first global feature and the second global feature may be calculated based on the Euclidean distance or the cosine distance. In this embodiment, the distance between the first global feature and the second global feature can be calculated based on the following formula, where the formula is:

D _G =||G ¹ -G ² || ₂

Wherein, D _G represents the calculated distance between the first global feature and the second global feature, G ¹ represents the first global feature, and G ² represents the second global feature.

Optionally, calculating whether the first local feature map and the second local feature map satisfy the second similarity condition may include:

Calculate the feature distance between the corresponding first local feature and the second local feature, the first local feature is the local feature included in the first local feature map, and the second local feature is the second local feature The local features included in the feature map.

It should be noted that the first local feature map is obtained by combining local features corresponding to multiple positions in the aforementioned first initial feature map. Correspondingly, the second local feature map is obtained by combining local features corresponding to multiple positions in the aforementioned second initial feature map. Optionally, the positions in the first initial feature map and the second initial feature map may be identified based on coordinates, and then positions with the same corresponding coordinates may be used as mutually corresponding positions. For example, the coordinates of the position (i ₁ , j ₁ ) of the first local feature map and the position (i ₁ , j ₁ ) of the second local feature map are both (i ₁ , j ₁ ), then the first local The position (i ₁ , j ₁ ) in the feature map and the position (i ₁ , j ₁ ) in the second local feature map are two positions whose positions correspond to each other. Correspondingly, the local features corresponding to the two positions corresponding to each other are the corresponding local features. Then, the first local feature and the second local feature corresponding to each other can be understood as the position corresponding to the first local feature, and the position corresponding to the second local feature is corresponding to each other.

As shown in FIG. 10 , the first initial feature map 20 and the second initial feature map 30 are shown, wherein the coordinates of the position 21 of the first initial feature map 20 are (1, 1), and the coordinates of the second initial feature map 30 are The coordinates corresponding to the position 31 are also (1, 1), because the coordinates of the position 21 in the first initial feature map 20 and the coordinates of the position 31 in the second initial feature map 30 are the same, then the position 21 and the position feature 31 is the corresponding position, and the first local feature corresponding to the position 21 and the first local feature corresponding to the position 31 are the corresponding first local feature and the second local feature.

Then, as a way, in the case where the sizes of the first image and the second image are the same, the sizes of the first initial feature map and the second initial feature map are also the same. Then, for each position in the first initial feature map, there will be a corresponding position in the second initial feature map.

As a way, the feature distance between the first local feature and the second local feature corresponding to each other can be calculated based on the following formula, which is:

D _L (i, j)=||L ¹ (i, j)-L ² (i, j)|| ₂

Wherein, L ¹ (i, j) represents the local feature corresponding to the position (i, j) in the first initial feature map. L ² (i, j) represents the local feature corresponding to the position (i, j) in the second initial feature map.

The first local feature whose corresponding feature distance is smaller than the second distance threshold is used as the target local feature.

As shown in the foregoing, in the case where each position in the first initial feature map has a corresponding position in the second initial feature map, the first partial corresponding to each position in the first initial feature map can be obtained by calculation The feature distance corresponding to the feature. The feature distance corresponding to the first local feature may be understood as the feature distance between the first local feature and the second local feature corresponding to the position.

If the ratio of the number of target local features to the number of local features included in the first local feature map is greater than a scale threshold, it is determined that the first local feature map and the second local feature map satisfy a second similarity condition.

where the ratio can be calculated based on the following formula, which is:

Among them, _NL represents the number of target local features, w represents the width of the first initial feature map, and h is the height of the first initial feature map. Optionally, the R can be 0.6.

It should be noted that the product of the width of the first initial feature map and the height of the first initial feature map represents the number of positions in the first initial feature map. In the case that each position corresponds to a local feature, The number of local features included in the first local feature map is the number of locations in the first initial feature map.

It should be noted that, as a method, before acquiring the first global feature and the first local feature map, the method further includes: if the sizes of the first image and the second image are different, comparing the size of the first image and the second image. The dimensions are updated to be the same.

In an image processing method provided by the present application, after acquiring the first image and the second image, it further acquires the first global feature and the first local feature map, and acquires the second global feature and the second local feature map, and further acquires the second global feature and the second local feature map. Based on the first global feature, the first local feature map, the second global feature, and the second local feature map, it is determined whether the first image and the second image are similar. Therefore, in the process of judging the similarity between the first image and the second image, the similarity detection can be carried out in combination with the global features and local features of the first image and the second image, so that it is possible to perform more similarity detection in the process of image similarity detection. For comprehensive detection, the accuracy of similar detection is improved. Moreover, in this embodiment, the possibility that the first image and the second image are similar may be preliminarily determined based on the first global feature and the second global feature, and then the first global feature and the second global feature satisfy After the first similarity condition is met, it is then detected whether the first local feature map and the second local feature map satisfy the second similarity condition, so as to improve the effective utilization of computing resources.

Referring to FIG. 11 , an image processing method provided by an embodiment of the present application includes:

S310: Acquire a first image and a first set of images.

Wherein, the first image set is a set of images for which image similarity matching is performed. In this embodiment, the images in the first image set may be matched with the first image to obtain images in the first image set that are similar to the first image.

S320: Obtain a first global feature and a first local feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes local features corresponding to the first image.

S330: Acquire a global feature and a local feature map corresponding to each image in the first image set.

Optionally, the global feature and the local feature map corresponding to each image in the first image set may be obtained based on the foregoing method of obtaining the first global feature and the first local feature map.

S340: Determine a second image set from the first image set based on the first global feature and the global feature corresponding to each image in the first image set.

It should be noted that, in this embodiment, the global feature corresponding to each image in the first image set may be compared with the first global feature of the first image, and then the corresponding comparison result satisfies the aforementioned The images corresponding to the global features of the first similarity condition are combined into a second image set. Exemplarily, the first image set includes image A, image B, and image C, wherein the global feature corresponding to image A is global feature t1, the global feature corresponding to image B is global feature t2, and the global feature corresponding to image C is is the global feature t3. Then, the global feature t1 is compared with the first global feature, then the global feature corresponding to the comparison result is the global feature t1, and the image corresponding to the global feature t1 is the image A.

S350: Based on the first local feature map and the local feature map of each image in the second image set, determine an image in the second image set that is similar to the first image.

After the second image set is obtained, the local feature map of each image in the second image set can be compared with the first local feature map, and then the corresponding comparison result satisfies the second similarity condition of the local feature map. The corresponding image is an image similar to the first image. Exemplarily, the first image set includes image A, image B, and image C, and the screened second image set includes image B and image C, then if the corresponding comparison result satisfies the aforementioned second similarity condition. If the image corresponding to the local feature map is image B, it can be determined that image B is an image similar to the first image.

In an image processing method provided by the present application, after acquiring the first image and the second image, it further acquires the first global feature and the first local feature map, and acquires the corresponding image of each image in the first image set. global feature and local feature map, and then based on the first global feature and the global feature corresponding to each image in the first image set, determine a second image set from the first image set, and then based on the first image set The first local feature map and the local feature map of each image in the second image set determine images in the second image set that are similar to the first image.

Therefore, in the process of screening images similar to the first image from the first image set based on the first image, preliminary screening can be performed based on the first global feature and the global feature corresponding to each image in the first image set, Obtain the second image set, so that the images in the first image set that are actually not similar to the first image can be excluded more quickly, and then based on the first local feature map and the local part of each image in the second image set feature to determine images in the second set of images that are similar to the first image, so that actually similar images can be determined based on local features of the images in a relatively smaller range.

Referring to FIG. 12, an image processing apparatus 400 provided by an embodiment of the present application, the apparatus 400 includes:

The image acquisition unit 410 is used for acquiring the first image and the second image.

A feature acquisition unit 420 is configured to acquire a first global feature and a first local feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes a corresponding global feature of the first image. local features.

The feature acquisition unit 420 is further configured to acquire a second global feature and a second local feature map, where the second global feature is a global feature corresponding to the second image, and the second local feature map includes the first The local features corresponding to the two images.

In one way, the feature acquisition unit 420 is specifically configured to input the first image into the target neural network, and obtain a feature map output by the target neural network as an initial feature map; perform global average pooling on the initial feature map process to obtain the first global feature; obtain the feature sets corresponding to multiple positions in the initial feature map, and the feature set corresponding to each position includes the feature of each position, and the adjacent positions of each position. feature; perform average processing on the feature sets corresponding to each of the positions to obtain local features corresponding to each of the positions; and obtain the first local feature map based on the local features corresponding to each of the positions.

In one way, the feature acquisition unit 420 is specifically configured to input the second image into the target neural network, and obtain the feature map output by the target neural network as the second initial feature map; for the second initial feature map Perform global average pooling processing to obtain a second global feature; obtain feature sets corresponding to multiple positions in the second initial feature map, and the feature set corresponding to each position includes the feature of each position, and each The features of the adjacent positions of the positions; the average processing of the corresponding feature sets for each of the positions to obtain the local features corresponding to each of the positions; based on the local features corresponding to each of the positions to obtain the first Two local feature maps.

The image comparison unit 430 is configured to determine whether the first image and the second image are similar based on the first global feature, the first local feature map, the second global feature and the second local feature map.

Wherein, optionally, the image comparison unit 430 is specifically configured to, if the first global feature and the second global feature satisfy the first similarity condition, and the first local feature map and the second local feature map satisfy The second similarity condition determines that the first image is similar to the second image.

Optionally, the image comparison unit 430 is specifically configured to detect whether the first local feature map and the second local feature map satisfy the first similarity condition if the first global feature and the second global feature satisfy the first similarity condition. Two similarity conditions; if the first global feature and the second global feature do not satisfy the first similarity condition, it is determined that the first image and the second image are not similar.

In one way, the image comparison unit 430 is specifically configured to determine the first global feature and the second global feature if the distance between the first global feature and the second global feature is less than a first distance threshold The first similarity condition is satisfied.

In one way, the image comparison unit 430 is specifically configured to calculate the feature distance between the corresponding first local features and the second local features, where the first local features are the local features included in the first local feature map feature, the second local feature is the local feature included in the second local feature map; the first local feature whose corresponding feature distance is less than the second distance threshold is used as the target local feature; if the target local feature is If the ratio of the number to the number of local features included in the first local feature map is greater than a scale threshold, it is determined that the first local feature and the second local feature satisfy a first similarity condition.

As shown in Figure 13, the device 400 further includes:

The size processing unit 440 is configured to update the size of the first image and the second image to be the same if the sizes of the first image and the second image are different.

Referring to FIG. 14, an image processing apparatus 500 provided by an embodiment of the present application, the apparatus 500 includes:

an image acquisition unit 510, configured to acquire a first image and a first image set;

A feature acquisition unit 520 is configured to acquire a first global feature and a first local feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes the corresponding first image. local features;

The feature obtaining unit 520 is further configured to obtain a global feature and a local feature map corresponding to each image in the first image set;

a first image determination unit 530, configured to determine a second image set from the first image set based on the first global feature and the global feature corresponding to each image in the first image set;

A second image determining unit 540, configured to determine an image in the second image set that is similar to the first image based on the first local feature map and the local feature map of each image in the second image set .

An image processing apparatus provided by the present application, after acquiring the first image and the second image, further acquires the first global feature and the first local feature map, and acquires the second global feature and the second local feature map, and further Based on the first global feature, the first local feature map, the second global feature, and the second local feature map, it is determined whether the first image and the second image are similar. Therefore, in the process of judging the similarity between the first image and the second image, the similarity detection can be carried out in combination with the global features and local features of the first image and the second image, so that the image similarity detection process can be more accurate. For comprehensive detection, the accuracy of similar detection is improved.

It should be noted that the apparatus embodiments in the present application correspond to the foregoing method embodiments, and the specific principles in the apparatus embodiments may refer to the content in the foregoing method embodiments, which will not be repeated here.

An electronic device provided by the present application will be described below with reference to FIG. 15 .

Referring to FIG. 15 , based on the above-mentioned image processing method and apparatus, an embodiment of the present application further provides another electronic device 200 that can execute the above-mentioned image processing method. The electronic device 200 includes one or more (only one shown in the figure) a processor 102, a memory 104, and a network module 106 that are coupled to each other. Wherein, the memory 104 stores programs that can execute the contents of the foregoing embodiments, and the processor 102 can execute the programs stored in the memory 104.

The processor 102 may include one or more cores for processing data. The processor 102 uses various interfaces and lines to connect various parts of the entire electronic device 200, and executes by running or executing the instructions, programs, code sets or instruction sets stored in the memory 104, and calling the data stored in the memory 104. Various functions of the electronic device 200 and processing data. Optionally, the processor 102 may adopt at least one of digital signal processing (Digital Signal Processing, DSP), field-programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). A hardware form is implemented. The processor 102 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly handles the operating system, user interface and application programs, etc.; the GPU is used for rendering and drawing of the display content; the modem is used to handle wireless communication. It can be understood that, the above-mentioned modem may not be integrated into the processor 102, and is implemented by a communication chip alone.

The memory 104 may include a random access memory (Random Access Memory, RAM), or may include a read-only memory (Read-Only Memory, ROM). Memory 104 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 104 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing the operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing the following method embodiments, and the like. For example, the memory 104 may store means for image processing. The apparatus for image processing may be the aforementioned apparatus 400 or the aforementioned apparatus 500 . The storage data area may also store data (such as phone book, audio and video data, chat record data) created by the electronic device 100 during use.

The network module 106 is used for receiving and sending electromagnetic waves, realizing mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices, for example, communicate with an audio playback device. The network module 106 may include various existing circuit elements for performing these functions, eg, antennas, radio frequency transceivers, digital signal processors, encryption/decryption chips, subscriber identity module (SIM) cards, memory, etc. . The network module 106 can communicate with various networks such as the Internet, an intranet, a wireless network, or communicate with other devices through a wireless network. The aforementioned wireless network may include a cellular telephone network, a wireless local area network, or a metropolitan area network. For example, the network module 106 may interact with the base station for information.

Please refer to FIG. 16 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. The computer-readable storage medium 1100 stores program codes, and the program codes can be invoked by the processor to execute the methods described in the above method embodiments.

The computer-readable storage medium 1100 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 1100 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1100 has storage space for program code 1110 that performs any of the method steps in the above-described methods. The program codes can be read from or written to one or more computer program products. Program code 1110 may be compressed, for example, in a suitable form.

To sum up, an image processing method, device, electronic device and storage medium provided by this application, after acquiring the first image and the second image, will further acquire the first global feature and the first local feature map, and Obtain a second global feature and a second local feature map, and then determine whether the first image and the second image are similar based on the first global feature, the first local feature map, the second global feature, and the second local feature map . Therefore, in the process of judging the similarity between the first image and the second image, the similarity detection can be carried out in combination with the global features and local features of the first image and the second image, so that the image similarity detection process can be more accurate. For comprehensive detection, the accuracy of similar detection is improved.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: it can still be Modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements to some of the technical feature diagrams; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of the present application .

Claims

An image processing method, characterized in that the method comprises:

obtain the first image and the second image;

acquiring a first global feature and a first local feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes a local feature corresponding to the first image;

acquiring a second global feature and a second local feature map, where the second global feature is a global feature corresponding to the second image, and the second local feature map includes a local feature corresponding to the second image;

Based on the first global feature, the first local feature map, the second global feature, and the second local feature map, it is determined whether the first image and the second image are similar.
The method according to claim 1, wherein the first image and the second image are determined based on the first global feature, the first local feature map, the second global feature and the second local feature map Are they similar, including:

If the first global feature and the second global feature satisfy the first similarity condition, and the first local feature map and the second local feature map satisfy the second similarity condition, determine the first image and the second image resemblance.
The method according to claim 2, wherein the method further comprises:

If the first global feature and the second global feature satisfy the first similarity condition, detecting whether the first local feature map and the second local feature map satisfy the second similarity condition;

If the first global feature and the second global feature do not satisfy the first similarity condition, it is determined that the first image and the second image are not similar.
The method according to claim 2 or 3, wherein the method further comprises:

If the distance between the first global feature and the second global feature is less than a first distance threshold, it is determined that the first global feature and the second global feature satisfy a first similarity condition.
The method according to claim 4, wherein, if the distance between the first global feature and the second global feature is less than a first distance threshold, determining the first global feature and the second global feature Before the feature satisfies the first similarity condition, it further includes:

The distance between the first global feature and the second global feature is calculated based on the Euclidean distance or the cosine distance.
The method according to claim 2 or 3, wherein the method further comprises:

Calculate the feature distance between the corresponding first local feature and the second local feature, the first local feature is the local feature included in the first local feature map, and the second local feature is the second local feature The local features included in the feature map;

Taking the first local feature whose corresponding feature distance is less than the second distance threshold as the target local feature;

If the ratio of the number of target local features to the number of local features included in the first local feature map is greater than a scale threshold, it is determined that the first local feature map and the second local feature map satisfy a second similarity condition.
The method according to claim 6, wherein the number of local features included in the first local feature map is the number of locations in the first initial feature map, and the number of locations in the first initial feature map is all The product of the width of the first initial feature map and the height of the first initial feature map.
The method according to any one of claims 1-7, wherein the acquiring the first global feature and the first local feature map comprises:

Inputting the first image to the target neural network, and obtaining the feature map output by the target neural network as the first initial feature map;

Perform a global average pooling process on the first initial feature map to obtain a first global feature;

Obtaining feature sets corresponding to multiple positions in the first initial feature map, where the feature sets corresponding to each position include features of each position and features of adjacent positions of each position;

Average processing is performed on the respective feature sets corresponding to each of the positions to obtain local features corresponding to each of the positions;

The first local feature map is obtained based on the local features corresponding to each of the positions.
The method according to any one of claims 1-8, wherein the acquiring the second global feature and the second local feature map comprises:

The second image is input into the target neural network, and the feature map output by the target neural network is obtained as the second initial feature map;

Perform a global average pooling process on the second initial feature map to obtain a second global feature;

Obtaining feature sets corresponding to multiple positions in the second initial feature map, where the feature sets corresponding to each position include features of each position and features of adjacent positions of each position;

Average processing is performed on the respective feature sets corresponding to each of the positions to obtain local features corresponding to each of the positions;

The second local feature map is obtained based on the local features corresponding to each of the positions.
The method according to any one of claims 1-9, wherein before acquiring the first global feature and the first local feature map, the method further comprises:

If the sizes of the first image and the second image are different, the sizes of the first image and the second image are updated to be the same.
The method according to any one of claims 1-10, wherein the first global feature is a vector representing the overall feature of the first image; the second global feature is a vector representing the second image A vector of overall features.
The method according to any one of claims 1-10, wherein the first local feature map comprises a plurality of local features representing respective corresponding positions of a plurality of positions in the first image; the second local feature map Including a plurality of local features corresponding to a plurality of positions in the second image.
An image processing method, characterized in that the method comprises:

obtaining a first image and a first set of images;

acquiring a first global feature and a first local feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes a local feature corresponding to the first image;

Obtain the global feature and local feature map corresponding to each image in the first image set;

determining a second set of images from the first set of images based on the first global feature and a global feature corresponding to each image in the first set of images;

Based on the first local feature map and the local feature map of each image in the second set of images, an image in the second set of images that is similar to the first image is determined.
The method according to claim 13, wherein the second image is determined from the first image set based on the first global feature and the global feature corresponding to each image in the first image set Collection, including:

Comparing the global feature corresponding to each image in the first image set with the first global feature of the first image respectively;

The images corresponding to the global features whose corresponding comparison results satisfy the first similarity condition in the first image set are combined into a second image set.
The method according to claim 13, characterized in that, based on the first local feature map and the local feature map of each image in the second image set, determining the relationship between the second image set and the second image set Images similar to the first image, including:

comparing the local feature map of each image in the second image set with the first local feature map;

The image corresponding to the local feature map whose corresponding comparison result satisfies the second similarity condition in the second image set is regarded as an image similar to the first image.
An image processing device, characterized in that the device comprises:

an image acquisition unit for acquiring the first image and the second image;

A feature acquisition unit, configured to acquire a first global feature and a first local feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes the first local feature map corresponding to the first image. local features;

The feature acquisition unit is further configured to acquire a second global feature and a second local feature map, where the second global feature is a global feature corresponding to the second image, and the second local feature map includes the second Local features corresponding to the image;

An image comparison unit, configured to determine whether the first image and the second image are similar based on the first global feature, the first local feature map, the second global feature, and the second local feature map.
The apparatus of claim 16, wherein the apparatus comprises:

A size processing unit, configured to update the size of the first image and the second image to be the same if the sizes of the first image and the second image are different.
An image processing device, characterized in that the device comprises:

an image acquisition unit for acquiring a first image and a first image set;

A feature acquisition unit, configured to acquire a first global feature and a first local feature map, where the first global feature is a global feature corresponding to the first image, and the first local feature map includes the first local feature map corresponding to the first image. local features;

The feature obtaining unit is further configured to obtain a global feature and a local feature map corresponding to each image in the first image set;

a first image determining unit, configured to determine a second image set from the first image set based on the first global feature and the global feature corresponding to each image in the first image set;

A second image determining unit, configured to determine an image in the second image set that is similar to the first image based on the first local feature map and the local feature map of each image in the second image set.
An electronic device, comprising a processor and a memory;

One or more programs are stored in the memory and configured to be executed by the processor to implement the method of any of claims 1-12 or the method of any of claims 13-15.
A computer-readable storage medium, wherein a program code is stored in the computer-readable storage medium, wherein, when the program code is executed by a processor, the method according to any one of claims 1-12 or The method of any of claims 13-15.