CN112766421A - Structure-aware-based face clustering method and device - Google Patents

Abstract

The present application proposes a structure-aware-based face clustering method and device, wherein the method includes: acquiring multiple face images to be processed, and extracting each of the to-be-processed face images based on a pre-trained convolutional neural network model face features of the face image, and construct a K-nearest neighbor graph according to the face features of each of the face images to be processed; input the K-nearest neighbor graph into a pre-trained edge score prediction model, and obtain the K-nearest neighbor graph The score of each edge in ; wherein, the edge score prediction model is obtained by using the structure-preserving subgraph sampling strategy to sample the K-nearest neighbor graph, and using the sampled subgraph to train the graph convolutional neural network; According to the score of each edge in the K-nearest neighbor graph, a first pruning operation is performed on the K-nearest neighbor graph to obtain face clusters for the plurality of face images to be processed. Solve the technical problem of insufficient accuracy of face clustering in related technologies.

Description

Structure-aware-based face clustering method and device

technical field

The present application relates to the technical field of artificial intelligence and deep learning in the technical field of image processing, and in particular, to a method and device for face clustering based on structure perception.

Background technique

The development of face recognition technology relies on the introduction of large-scale face datasets. In recent years, with the development of face recognition technology, the scale of face datasets has become larger and larger. However, with the increase in the scale of datasets, the cost of labeling has become more and more expensive.

The face clustering algorithm is an effective method to reduce the cost of labeling. However, in the related art, when faced with large-scale real face data, the clustering accuracy of the clustering model needs to be improved.

SUMMARY OF THE INVENTION

The present application aims to solve one of the technical problems in the related art at least to a certain extent.

Therefore, the first purpose of the present application is to propose a face clustering method based on structure perception, so as to improve the accuracy of face clustering.

The second object of the present application is to propose a device.

The third object of the present application is to propose an electronic device.

A fourth object of the present application is to propose a non-transitory computer-readable storage medium.

A fifth object of the present application is to propose a computer program product.

In order to achieve the above purpose, the embodiment of the first aspect of the present application proposes a face clustering method, including:

Obtain a plurality of face images to be processed, and extract the face features of each of the face images to be processed based on a pre-trained convolutional neural network model, and construct a face feature based on the face features of each of the face images to be processed K-nearest neighbor graph;

The K-nearest neighbor graph is input into the pre-trained edge score prediction model, and the score of each edge in the K-nearest neighbor graph is obtained; wherein, the edge score prediction model utilizes the structure-preserving subgraph sampling strategy to perform the K-nearest neighbor graph. Sampling, and using the sampled subgraphs to train the graph convolutional neural network;

According to the score of each edge in the K-nearest neighbor graph, a first pruning operation is performed on the K-nearest neighbor graph to obtain face clusters for the plurality of face images to be processed.

In order to achieve the above purpose, an embodiment of the second aspect of the present application proposes a device, including:

a first acquisition module, used for acquiring a plurality of face images to be processed;

a feature extraction module for extracting the face features of each of the to-be-processed face images based on a pre-trained convolutional neural network model;

a building module for constructing a K-nearest neighbor graph according to the facial features of each of the to-be-processed facial images;

A prediction module for inputting the K-nearest neighbor graph into a pre-trained edge score prediction model to obtain the score of each edge in the K-nearest neighbor graph; wherein, the edge score prediction model utilizes a structure-preserving subgraph sampling strategy It is obtained by sampling the K-nearest neighbor graph and training the graph convolutional neural network with the sampled subgraph;

A pruning operation module, configured to perform a first pruning operation on the K-nearest neighbor graph according to the score of each edge in the K-nearest neighbor graph to obtain face clusters for the plurality of face images to be processed .

According to a third aspect of the present application, an electronic device is provided, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the structure-aware-based face clustering methods.

According to a fourth aspect of the present application, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the structure-awareness-based storage medium of the first aspect of the present application. face clustering methods.

According to a fifth aspect of the present application, there is provided a computer program product, comprising a computer program that, when executed by a processor, implements the structure-aware-based face clustering method according to the first aspect.

According to the technical solutions of the embodiments of the present application, the features of a face image are extracted through a convolutional neural network, accurate face features are obtained, and a K-nearest neighbor graph is generated. Through the structure-preserving subgraph sampling strategy, a subgraph is obtained from the K-nearest neighbor graph, which can represent the relationships within and between clusters in the K-nearest neighbor graph. The first pruning operation is performed on the graph to make the face clustering more accurate.

Additional aspects and advantages of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a flowchart of a structure-aware-based face clustering method according to an embodiment of the present application;

2 is a flowchart of a structure-aware-based face clustering method according to another embodiment of the present application;

3 is a schematic diagram of a second pruning operation according to an embodiment of the present application;

4 is a flowchart of a structure-aware-based face clustering method according to yet another embodiment of the present application;

5 is a flowchart of a structure-aware-based face clustering method according to an embodiment of the present application;

6 is a structural block diagram of a structure-aware-based face clustering apparatus according to an embodiment of the present application;

7 is a structural block diagram of a structure-aware-based face clustering apparatus according to another embodiment of the present application;

FIG. 8 is a block diagram of an electronic device used to implement the structure-aware-based face clustering method according to the embodiment of the present application.

Detailed ways

The following describes in detail the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to be used to explain the present application, but should not be construed as a limitation to the present application.

The following describes the structure-aware-based face clustering method and apparatus according to the embodiments of the present application with reference to the accompanying drawings.

FIG. 1 is a flowchart of a structure-aware-based face clustering method according to an embodiment of the present application. It should be noted that the structure-awareness-based face clustering method in the embodiment of the present application can be applied to the structure-awareness-based face clustering apparatus in the embodiment of the present application, and the structure-awareness-based face clustering apparatus can be configured on the electronic device of the embodiments of the present application.

As shown in Figure 1, the structure-aware-based face clustering method may include:

Step 101: Obtain a plurality of face images to be processed, and extract the face features of each face image to be processed based on a pre-trained convolutional neural network model, and construct K according to the face features of each face image to be processed. Neighbor graph.

In some embodiments of the present application, face images belonging to the same type can be connected together through the K-nearest neighbor graph, where the nodes in the K-nearest neighbor graph represent face images, and the edges of the nodes in the K-nearest neighbor graph represent the corresponding Face images belong to the same class. In the process of constructing the K-nearest neighbor graph, two parameters are required, and the two parameters are: the face feature of each face image to be processed and the K value. Among them, the face features can be extracted from the face images to be processed through a pre-trained convolutional neural network model; the K value can be set based on experience and/or the number of face images to be processed, and the K value can determine K The classification effect of the nearest neighbor graph.

Understandably, due to the limitations of the pre-trained convolutional neural network model and the selection of the K value, in the K nearest neighbor graph, not only face images belonging to the same class will be connected together, but also face images that do not belong to the same class. A certain probability is connected together. Therefore, the accuracy of face image classification of K-nearest neighbor graph at this time cannot meet the requirements. In order to obtain accurate face clustering, it is necessary to further operate on the K-nearest neighbor graph to remove the connection relationship of wrongly connected face images.

Step 102, input the K-nearest neighbor graph into the pre-trained edge score prediction model, and obtain the score of each edge in the K-nearest neighbor graph; wherein, the edge score prediction model utilizes the structure-preserving subgraph sampling strategy to sample the K-nearest neighbor graph, and It is obtained by training a graph convolutional neural network using the sampled subgraphs.

It is understandable that the score of each edge can be obtained by scoring each edge in the K-nearest neighbor graph, so that the pruning operation of the K-nearest neighbor graph can be realized by screening the scores, so as to obtain more accurate face clustering.

In some embodiments of the present application, in order to score each edge in the K-nearest neighbor graph, an edge score prediction model may be pre-trained. In some cases, due to the limitation of hardware computing power, the structure-preserving sub-graph sampling strategy can be used to sample the K-nearest neighbor graph, and obtain the score prediction model through the sub-graph after sampling. In this way, an edge score prediction model suitable for K-nearest neighbor graphs can be obtained under the premise of saving computing power. Generally speaking, in the K-nearest neighbor graph, nodes with close connections can be regarded as a cluster, and there can be connections between different clusters, but the connections between different clusters are usually not close.

For example, the structure-preserving subgraph sampling strategy may be to randomly sample the same number of nodes from each cluster in the K-nearest neighbor graph, and retain the connection relationship between these nodes to obtain a subgraph. The subgraph can retain the connection relationship between nodes in each cluster, and also retain the connection relationship between clusters. It is understandable that each edge in this subgraph will connect two nodes, and two nodes connected by one edge can be used as A node pair. An edge score prediction model can be obtained through the subgraph, and the initial model of the edge score prediction model includes, but is not limited to, any one of a graph convolutional neural network and a graph convolutional neural network connected to a multi-layer perceptron. When training this edge score prediction model, the model input is the entire subgraph, and the model output is the edge score corresponding to each edge in the subgraph. Understandably, when the node pair belongs to the same class, the edge score corresponding to the node pair is 1; when the node pair does not belong to the same class, the edge score corresponding to the node pair is 0. The training can also be supervised using a loss function including, but not limited to, any of an exponential loss function and a cross-entropy loss function.

Step 103 , perform a first pruning operation on the K-nearest neighbor graph according to the scores of each edge in the K-nearest neighbor graph, and obtain face clusters for multiple face images to be processed.

In some embodiments of the present application, in order to make the face clustering in the K-nearest neighbor graph more accurate, a first pruning operation may be performed on the K-nearest neighbor graph. The first pruning operation can be completed by a pre-trained edge score prediction model. The pre-trained edge score prediction model can score each edge in the K-nearest neighbor graph. A threshold can be preset, and the scoring result and the threshold can be calculated. Compare. When the score is greater than or equal to the threshold, it means that the two nodes connected by the edge belong to the same class, and the edge can be retained; when the score is less than the threshold, it means that the two nodes connected by the edge do not belong to the same class, the edge Edges will be removed. Understandably, edges connecting nodes belonging to the same class can be called true edges; edges connecting nodes not belonging to the same class can be called false edges. After the first pruning operation, face clusters for multiple face images to be processed can be obtained.

According to the structure-aware-based face clustering method according to the embodiment of the present application, the face image to be processed is clustered according to the face features extracted by the convolutional neural network model, and a K-nearest neighbor graph is constructed. Through deep learning technology, more accurate face features can be obtained, so that the face clustering of the constructed K-nearest neighbor graph is more accurate. Using the subgraph of the K-nearest neighbor graph as a training set, the graph convolutional neural network can be trained to obtain a score prediction model. The subgraph of the K-nearest neighbor graph can represent the relationship between each face image in a cluster in the K-nearest neighbor graph, It can also represent the relationship between different clusters in the K-nearest neighbor graph. After the graph convolutional neural network is trained, a score prediction model can be obtained. Through the score prediction model, the pruning operation of the K-nearest neighbor graph can be realized, false edges can be removed, and more accurate face clustering can be obtained.

In the second embodiment of the present application, based on the first embodiment, in order to make the face clustering more accurate, a second pruning operation may be performed on the face clustering after the first pruning. The method can be specifically described using Embodiment 2 based on the structure-aware-based face clustering method in FIG. 1 . Optionally, after the first pruning operation is performed on the K-nearest neighbor graph according to the score of each edge in the K-nearest neighbor graph, steps 201-202 are further included.

FIG. 2 is a flowchart of a structure-aware-based face clustering method according to another embodiment of the present application. In order to explain more clearly how to perform the second pruning operation, it can be described in detail with reference to FIG. 2 , which is a flowchart of a structure-aware-based face clustering method according to an embodiment of the present application, which specifically includes:

Step 201: Calculate the intimacy between two nodes in the K-nearest neighbor graph that has undergone the first pruning operation.

In some embodiments of the present application, there may be an affinity between two nodes in the K-nearest neighbor graph. The intimacy reflects whether two nodes belong to the same category. The higher the intimacy, the higher the probability that the pair of nodes belong to the same category; the lower the intimacy, the lower the probability that the pair of nodes belong to the same category. In general, the probability that a node connected by an edge between clusters is a class is low; the probability that a node connected by an edge within a cluster is a class is high.

For example, the calculation method of the intimacy may be to determine the range diameter according to the number of nodes in the cluster, with the center of the cluster as the center of the circle, the nodes within the range diameter are considered to have sufficient intimacy, and the points not within the range diameter are considered intimate. Insufficient degree.

In some embodiments of the present application, the intimacy between two nodes can also be calculated by the following formula:

Among them, n ₁ represents the number of edges connected to node N1, n ₂ represents the number of edges connected to node N2, and k represents the number of neighbor nodes shared by node N1 and node N2.

C、D的亲密度为

3 is a schematic diagram of a second pruning operation according to an embodiment of the present application. As shown in the figure, node A has nine edges, node B has eight edges, and node A and node B share one neighbor node. Node C has seven edges, and node D has ten edges. Among them, the common neighbor nodes of node C and node D are six. It can be concluded that the intimacy of A and B is

The intimacy of C and D is

中的每个元素

表示节点N_i和N_j的共有邻居数。那亲密度的值为

其中，sum₀＝vec(∑_ja_·j ^-1),sum₁＝vec(∑_ia_i· ^-1)。其中，vec()表示向量化，sum₀表示对矩阵A按行求和之后对各元素取倒数再向量化，sum₁表示对矩阵A按列求和之后对各元素取倒数再向量化。In some embodiments of this application, matrix operations can also be used to calculate intimacy. The adjacency matrix corresponding to the K-nearest neighbor graph is A∈RN×N, then the number of common neighbor nodes of all pairwise nodes is

each element in

represents the number of shared neighbors of nodes N _i and N _j . The value of intimacy is

Wherein, sum ₀ =vec(∑ _j a _·j ^-1 ), sum ₁ =vec(∑ _i a _{i ·} ^-1 ). Among them, vec() means vectorization, sum ₀ means taking the reciprocal of each element after summing the matrix A in rows and then vectorizing it, sum ₁ means taking the inverse of each element after summing the matrix A in columns and then vectorizing.

Step 202, according to the intimacy between the two nodes, perform a second pruning operation on the K-nearest neighbor graph that has undergone the first pruning operation.

In some embodiments of the present application, based on the K-nearest neighbor graph of the first pruning operation, a second pruning operation may be performed, and this operation may be implemented by presetting an intimacy threshold. When the intimacy between the two nodes is greater than or equal to the threshold, the edges corresponding to the two nodes are retained; when the intimacy between the two nodes is less than the threshold, the edges corresponding to the two nodes are removed.

According to the structure-aware-based face clustering method according to the embodiment of the present application, after the first pruning operation, there are still some false edges. Since the false edges are wrongly connected to different categories, the accuracy of face clustering will be affected. Therefore, the second pruning operation can be performed on the K-nearest neighbor graph after the first pruning operation through the intimacy between the two nodes. This operation makes the obtained face clustering more accurate, and the structure of the K-nearest neighbor graph is clearer. . After the second pruning, most of the wrong edges in the K-nearest neighbor graph have been removed, and the face clustering information read from the K-nearest neighbor graph will be more accurate.

In the third embodiment of the present application, based on the above-mentioned embodiment, the edge score prediction model can be obtained by training the following steps in advance.

More clearly, it can be described in detail with reference to FIG. 4 , which is a flowchart of a structure-aware-based face clustering method according to another embodiment of the present application, which specifically includes:

Step 401: Obtain a training sample set, where the training sample set includes multiple face sample images.

Understandably, a training sample set is usually required when training a deep learning model. The purpose of the present application is to obtain face clusters. Therefore, in some embodiments of the present application, the training sample set may include multiple face sample images.

Step 402, extracting face features of each face sample image based on the convolutional neural network model.

Understandably, the subgraphs used for training the graph convolutional neural network can be obtained by sampling according to the K-nearest neighbor graph. When constructing the K-nearest neighbor graph, the face features of the face sample images need to be used. In some embodiments of the present application, a convolutional neural network model may be used to extract a face feature of a face sample image, and the face feature may be a vector.

Step 403 , construct K-nearest neighbor graph samples according to the face features of each face sample image.

Understandably, each face sample image has its corresponding face feature, and K-nearest neighbor image samples can be constructed according to this face feature.

Step 404: Sampling the K-nearest neighbor graph samples based on the structure-preserving subgraph sampling strategy to obtain the subgraph obtained after sampling, and use the subgraph obtained after sampling to train the graph convolutional neural network to obtain network parameters, and according to the network The parameters generate the edge score prediction model.

In some embodiments of the present application, the K-nearest neighbor graph samples are sampled based on the structure-preserving subgraph sampling strategy to obtain a subgraph obtained after sampling, and the subgraph may be obtained by randomly selecting from each cluster of the K-nearest neighbor graph. Select a certain number of face images to obtain sub-images, which can also be obtained by the following steps:

Step 1, randomly select M clusters from the K-nearest neighbor graph samples as sampling seeds.

It can be understood that, in order to model the edges between face images in a cluster, M clusters can be randomly selected from the K-nearest neighbor image samples as sampling seeds in a cluster unit. Wherein, there may be one or more sampling seeds.

Step 2: For each seed cluster, select N nearest neighbor clusters of each seed cluster, and take the graph composed of M clusters and N nearest neighbor clusters as the first subgraph S1.

In some embodiments of the present application, in order to model the relationship between clusters, for each seed cluster, N nearest neighbor clusters can be selected, and the calculation method of the nearest neighbor clusters can be realized by similarity calculation. The degree calculation can be the cosine similarity of cluster centers. In order to model the relationships within and between clusters at the same time, the first subgraph S1 can be composed of M clusters and N nearest neighbor clusters.

Step 3, randomly select K1 clusters from the first subgraph S1 to construct a second subgraph S2.

In some embodiments of the present application, in order to perform a generalization operation on the selected clusters, K1 clusters may be randomly selected from the first subgraph S1 to construct the second subgraph S2.

Step 4, randomly select K2 nodes from the second subgraph S2 as the subgraph obtained after sampling.

In some embodiments of the present application, in order to generalize the nodes in the cluster, K2 nodes may be randomly selected from the second subgraph S2 as the sampled subgraph.

The K-nearest neighbor graph obtained by the structure-preserving subgraph sampling strategy can train the graph convolutional neural network, and the training method can include the following steps:

Step 1: Input the subgraph obtained after sampling into the graph convolutional neural network, and obtain the predicted scores of each edge in the subgraph.

In some embodiments of the present application, when the nodes connected by each edge belong to the same class, the predicted score value may be 1; when the face images connected by each edge belong to different classes, the predicted score value may be 0.

Step 2: Calculate the loss value between the predicted score value of each edge and the real score value of each corresponding edge based on the cross-entropy loss function.

In some embodiments of the present application, there is a loss value between the predicted score value of each edge and the actual score value of its corresponding edge, and the loss value can be calculated by using a cross-entropy loss function.

Step 3: Train the graph convolutional neural network according to the loss value.

Understandably, a graph convolutional neural network can be trained based on the loss value.

Through training, network parameters can be obtained, and an edge score prediction model can be generated according to the obtained network parameters. The output of the edge score prediction model is shown in FIG. 5 , which is the output result of the edge score prediction model according to an embodiment of the present application, where the horizontal axis represents the output of the edge score prediction model, and the closer to 0, the connected The lower the probability that two face images belong to the same type; the closer to 1, the lower the probability that the two face images connected by the edge belong to the same type.

According to the structure-aware face clustering method according to the embodiment of the present application, a K-nearest neighbor graph is constructed, and a structure-preserving subgraph sampling strategy is used to process the K-nearest neighbor graph to obtain a subgraph, thereby obtaining an edge score prediction model. In this method, the subgraph retains the important structural information of the entire K-nearest neighbor graph, not only retaining the connections with high correlation within the cluster, but also retaining the connections with low correlation between clusters. And for generalization, randomness is introduced, which can enhance the performance of the model.

According to an embodiment of the present application, the present application further provides a structure-aware-based face clustering apparatus.

FIG. 6 is a structural block diagram of a structure-aware-based face clustering apparatus according to an embodiment of the present application. As shown in FIG. 6 , the structure-aware-based face clustering apparatus 600 may include: a first acquisition module 601 , a construction module 602 , a prediction module 603 , and a pruning operation module 604 .

Specifically, the first acquisition module 601 is used to acquire a plurality of face images to be processed;

A feature extraction module, which is used to extract the face features of each face image to be processed based on the pre-trained convolutional neural network model;

Building module 602, for constructing a K-nearest neighbor graph according to the facial features of each to-be-processed facial image;

The prediction module 603 is used to input the K-nearest neighbor graph into the pre-trained edge score prediction model, and obtain the score of each edge in the K-nearest neighbor graph; wherein, the edge score prediction model utilizes the structure-preserving subgraph sampling strategy to perform the K-nearest neighbor graph. Sampling, and using the sampled subgraphs to train the graph convolutional neural network;

The pruning operation module 604 is configured to perform a first pruning operation on the K-nearest neighbor graph according to the score of each edge in the K-nearest neighbor graph, to obtain face clusters for a plurality of face images to be processed.

FIG. 7 is a structural block diagram of an apparatus for face clustering based on structure perception according to another embodiment of the present application. In some embodiments of the present application, as shown in FIG. 7 , the structure-aware-based face clustering apparatus further includes: a pre-training module 705 .

Specifically, the pre-training module 705 is used for pre-training the edge score prediction model; wherein, the pre-training module is specifically used for: acquiring a training sample set, the training sample set includes a plurality of face sample images; extracting each person based on the convolutional neural network model face features of face sample images; construct K-nearest neighbor map samples according to the face features of each face sample image; sample K-nearest neighbor map samples based on the structure-preserving sub-graph sampling strategy to obtain the sub-graph obtained after sampling, and use The subgraph obtained after sampling trains the graph convolutional neural network, obtains the network parameters, and generates an edge score prediction model according to the network parameters.

701-704 in FIG. 7 and 601-604 in FIG. 6 have the same function and structure.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

According to the embodiments of the present application, the present application further provides an electronic device, a readable storage medium, and a computer program product.

FIG. 8 shows a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the application described and/or claimed herein.

As shown in FIG. 8 , the device 800 includes a computing unit 801 that can be executed according to a computer program stored in a read only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803 Various appropriate actions and handling. In the RAM 803, various programs and data necessary for the operation of the device 800 can also be stored. The computing unit 801 , the ROM 802 , and the RAM 803 are connected to each other through a bus 804 . An input/output (I/O) interface 805 is also connected to bus 804 .

Various components in the device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc. ; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 809 allows the device 800 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 801 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 801 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the information sorting method. For example, in some embodiments, the information sorting method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 808 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 800 via ROM 802 and/or communication unit 809 . When a computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the information sorting method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the information sorting method by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of this application, a machine-readable medium may be a tangible medium that may contain or store the program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), the Internet, and blockchain networks.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or a cloud host. It is a host product in the cloud computing service system to solve the traditional physical host and VPS service ("Virtual Private Server", or "VPS" for short). , there are the defects of difficult management and weak business expansion. The server can also be a server of a distributed system, or a server combined with a blockchain.

According to the structure-aware-based face clustering method according to the embodiment of the present application, the face image to be processed is clustered according to the face features extracted by the convolutional neural network model, and a K-nearest neighbor graph is constructed. Through deep learning technology, more accurate face features can be obtained, so that the face clustering of the constructed K-nearest neighbor graph is more accurate. Using the subgraph of the K-nearest neighbor graph as a training set, the graph convolutional neural network can be trained to obtain a score prediction model. The subgraph of the K-nearest neighbor graph can represent the relationship between each face image in a cluster in the K-nearest neighbor graph, It can also represent the relationship between different clusters in the K-nearest neighbor graph. After the graph convolutional neural network is trained, a score prediction model can be obtained, and the first pruning operation of the K-nearest neighbor graph can be realized through the score prediction model, and the wrongly connected edges can be removed to obtain more accurate face clustering.

After the first pruning operation, the second pruning operation can also be performed on the K-nearest neighbor graph. This operation removes false edges through intimacy calculation, so that the obtained face clustering is more accurate. The structure of the K-nearest neighbor graph Clearer. After the second pruning, most of the false edges in the K-nearest neighbor graph have been removed, and the face clustering information read from the K-nearest neighbor graph will be more accurate.

In the process of constructing the K-nearest neighbor graph, the structure-preserving subgraph sampling strategy can be used to process the K-nearest neighbor graph to obtain a subgraph, thereby obtaining an edge score prediction model. In this method, the subgraph retains the important structural information of the entire K-nearest neighbor graph, not only retaining the connections with high correlation within the cluster, but also retaining the connections with low correlation between clusters. And for generalization, randomness is introduced, which can enhance the performance of the model.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (10)

1. a face clustering method based on structure perception, is characterized in that, comprises: Obtain a plurality of face images to be processed, and extract the face features of each of the face images to be processed based on a pre-trained convolutional neural network model, and construct a face feature based on the face features of each of the face images to be processed K-nearest neighbor graph; The K-nearest neighbor graph is input into the pre-trained edge score prediction model, and the score of each edge in the K-nearest neighbor graph is obtained; wherein, the edge score prediction model utilizes the structure-preserving subgraph sampling strategy to perform the K-nearest neighbor graph. Sampling, and using the sampled subgraphs to train the graph convolutional neural network; According to the score of each edge in the K-nearest neighbor graph, a first pruning operation is performed on the K-nearest neighbor graph to obtain face clusters for the plurality of face images to be processed. 2. The method according to claim 1, wherein after the first pruning operation is performed on the K-nearest neighbor graph according to the score of each edge in the K-nearest neighbor graph, the method further comprises: include: Calculate the intimacy between two nodes in the K-nearest neighbor graph that has undergone the first pruning operation; According to the intimacy between the two nodes, a second pruning operation is performed on the K-nearest neighbor graph that has undergone the first pruning operation. 3. The method according to claim 2, wherein the intimacy between the two nodes is calculated by the following formula:

Among them, n ₁ represents the number of edges connected to node N1, n ₂ represents the number of edges connected to node N2, and k represents the number of neighbor nodes shared by node N1 and node N2. 4. The method according to any one of claims 1 to 3, wherein the edge score prediction model is obtained by training the following steps in advance: Obtain a training sample set, the training sample set includes a plurality of face sample images; Extract the face feature of each of the face sample images based on the convolutional neural network model; According to the face features of each of the face sample images, construct a K-nearest neighbor graph sample; Based on the structure-preserving subgraph sampling strategy, samples of K-nearest neighbor graphs are sampled to obtain subgraphs obtained after sampling, and the graph convolutional neural network is trained by using the subgraphs obtained after sampling to obtain network parameters, and according to The network parameters generate the edge score prediction model. 5. The method according to claim 4, wherein the sampling of K-nearest neighbor graph samples based on the structure-preserving subgraph sampling strategy to obtain the subgraph obtained after sampling, comprising: Randomly select M clusters from the K-nearest neighbor graph samples as sampling seeds; For each seed cluster, select the N nearest neighbor clusters of each seed cluster, and take the graph composed of the M clusters and the N nearest neighbor clusters as the first subgraph S1; Randomly select K1 clusters from the first subgraph S1 to construct a second subgraph S2; K2 nodes are randomly selected from the second subgraph S2 as the subgraph obtained after the sampling. 6. The method according to claim 4, wherein the training of the graph convolutional neural network using the subgraph obtained after the sampling comprises: inputting the subgraph obtained after the sampling into the graph convolutional neural network to obtain the predicted score values of each edge in the subgraph; Calculate the loss value between the predicted score value of each edge and the actual score value of the corresponding edge based on the cross entropy loss function; The graph convolutional neural network is trained according to the loss value. 7. A face clustering device based on structure perception, characterized in that, comprising: a first acquisition module, used for acquiring a plurality of face images to be processed; a feature extraction module for extracting the face features of each of the to-be-processed face images based on a pre-trained convolutional neural network model; a building module for constructing a K-nearest neighbor graph according to the facial features of each of the to-be-processed facial images; A prediction module for inputting the K-nearest neighbor graph into a pre-trained edge score prediction model to obtain the score of each edge in the K-nearest neighbor graph; wherein, the edge score prediction model utilizes a structure-preserving subgraph sampling strategy It is obtained by sampling the K-nearest neighbor graph and training the graph convolutional neural network with the sampled subgraph; A pruning operation module, configured to perform a first pruning operation on the K-nearest neighbor graph according to the score of each edge in the K-nearest neighbor graph to obtain face clusters for the plurality of face images to be processed . 8. The device according to claim 7, wherein the pruning operation module is further used for: After performing the first pruning operation on the K-nearest neighbor graph according to the score of each edge in the K-nearest neighbor graph, calculate the difference between two nodes in the K-nearest neighbor graph that has undergone the first pruning operation. intimacy; According to the intimacy between the two nodes, a second pruning operation is performed on the K-nearest neighbor graph that has undergone the first pruning operation. 9. The device according to claim 7 or 8, characterized in that, further comprising: A pre-training module for pre-training the edge score prediction model; wherein, the pre-training module is specifically used for: Obtain a training sample set, the training sample set includes a plurality of face sample images; Extract the face feature of each of the face sample images based on the convolutional neural network model; According to the face features of each of the face sample images, construct a K-nearest neighbor graph sample; Based on the structure-preserving subgraph sampling strategy, samples of K-nearest neighbor graphs are sampled to obtain subgraphs obtained after sampling, and the graph convolutional neural network is trained by using the subgraphs obtained after sampling to obtain network parameters, and according to The network parameters generate the edge score prediction model. 10. The apparatus according to claim 9, wherein the pre-training module is specifically used for: Randomly select M clusters from the K-nearest neighbor graph samples as sampling seeds; For each seed cluster, select the N nearest neighbor clusters of each seed cluster, and take the graph composed of the M clusters and the N nearest neighbor clusters as the first subgraph S1; Randomly select K1 clusters from the first subgraph S1 to construct a second subgraph S2; K2 nodes are randomly selected from the second subgraph S2 as the subgraph obtained after the sampling.

Images