CN115828109A - Cross-social network virtual identity association method and device based on multi-modal fusion and representation alignment - Google Patents

Abstract

The invention discloses a cross-social network virtual identity association method and a device based on multi-mode fusion and representation alignment, wherein the method comprises the following steps: extracting characteristics of user names of social networks of different platforms, texts published by users and social relations of the users to respectively obtain characteristic information of the users in different modes; performing multi-mode fusion by using an attention mechanism according to the characteristic information to obtain a first user representation fused with multi-dimensional characteristics; aligning the first user representation through representation to strengthen the user representation, and finally obtaining a second user representation with the same distribution on different platforms; and calculating cosine similarity between the second user representations to obtain similarity scores between the users, and taking the user pair with the highest score as an identity correlation result. The method solves the problems that a single modal model cannot completely describe a user and the distribution difference exists in different social platforms through a multi-modal fusion and representation alignment method.

Description

Cross-social network virtual identity association method based on multimodal fusion and representation alignment and device

technical field

The invention belongs to the technical field of social network virtual identity association, and in particular relates to a cross-social network virtual identity association method and device for multi-modal fusion and representation alignment.

Background technique

Nowadays, social network has become an indispensable part of people's life with its high degree of convenience. Usually, people like to join multiple social platforms to enjoy different services, such as using WeChat to communicate, using Weibo to read news or check in. Therefore, many scholars are committed to research related to social networks, and cross-social network virtual identity association is an important part of it. The purpose is to identify the social accounts of the same natural person on different platforms. and other fields have attracted great attention.

In fact, many methods have been proposed for user identity linking. The current methods can be divided into three categories: methods based on user attributes, methods based on user-generated content, and methods based on user social relations. But these methods all have certain flaws. For user attributes, for privacy reasons, users selectively disclose personal data attributes and keep some sensitive information (such as age or contact information) private, and may even forge or imitate information, increasing the uncertainty and ambiguity of information sex. Due to the richness of social networks, users post in various forms (text, pictures, etc.), and if only a single content is used, it will cause information loss. Based on the social relationship between users, existing methods place too much emphasis on structured information, but the characteristics of users’ friends in social networks are also very helpful for identifying users. After all, sometimes the characteristics of friends may be better than the user’s own. The characteristics are more unique. If the characteristics of his friends are taken into account, the accuracy rate will be greatly improved. Therefore, multi-modal user information should be utilized rather than limited to single-modal information. On the other hand, the confidence level that characterizes the user varies from modality to modality. Sometimes the user's text conveys more information than other modals, and sometimes images play a key role. Therefore, adaptively representing different modalities is the key to solving this problem.

Secondly, although the same user may publish similar information on different social platforms, due to the inconsistent data distribution between platforms, these similar information may also have different representations. But existing methods often directly link user identities based on their representations, without considering the semantic gap between them. Therefore, how to make the representation of the same user close to each other on different platforms is another big challenge.

Contents of the invention

The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, provide a cross-social network virtual identity association method and device based on multi-modal fusion and representation alignment, and solve the problem of single model through multi-modal fusion and representation alignment It is not possible to fully describe the problems of users and the problems of distribution differences among different social platforms.

In order to achieve the above object, the present invention adopts the following technical solutions:

In the first aspect, the present invention provides a cross-social network virtual identity association method based on multimodal fusion and representation alignment, including the following steps:

Feature extraction is performed on the user names, texts published by users and social relations of users of different social network users, and the characteristic information of user names, text characteristic information published by users and user social relation characteristic information are respectively obtained;

According to the user name feature information obtained, the text feature information published by the user, and the user social relationship feature information, the attention mechanism is used to perform multimodal fusion to obtain a first user representation that combines multi-dimensional features;

performing user representation enhancement processing on the first user representation through a representation alignment method, and finally obtaining second user representations with the same distribution space on different platforms;

Calculate the cosine similarity between the second user representations to obtain the similarity score between users, and use the user pair with the highest score as the identity association result.

As a preferred technical solution, the feature extraction of the user name is specifically:

将得到的所有用户名向量依次拼接得到用户名计数矩阵

由于C0是一个稀疏矩阵，为此使用一个自动编码器将其进行转换，转换的公式的具体为：For the username of a given user, the character-level Bag-of-Words model is used for feature extraction, and the number of occurrences of each character in each username is counted to obtain a vector

Concatenate all the obtained username vectors in sequence to obtain a username count matrix

Since C0 is a sparse matrix, an autoencoder is used to convert it. The conversion formula is as follows:

和

分别为用户名向量，通过梯度下降不断训练损失函数L_c，得到最优的W_e和b_e，最终得到维度为d的用户名嵌入矩阵

Among them, W _e , be _e are the weights and biases of the encoder, W _d , b _d are the weights and biases of the decoder, C ¹ is the decoder username vector matrix,

and

They are the username vectors, and the loss function L _c is continuously trained through gradient descent to obtain the optimal W _e and be _e , and finally a username embedding matrix with dimension d

As a preferred technical solution, the feature extraction of the text published by the user is specifically:

Input the text published by the user into the Word2Vec model to obtain the embedding vector of each text, and then average the embedding vector of each user’s published text as the representation of the user’s published text, and splice the text embedding vectors of all users in turn, Get the text embedding matrix with dimension d

As a preferred technical solution, the feature extraction of the user social relationship is specifically:

The n×m adjacency matrix obtained from the social relations composed of n users on platform N ¹ and m users on platform N ² is obtained through the DeepWalk model to obtain the embedding vector of each user's social relations, and the social relation embedding vectors of all users are concatenated in sequence , get the user social relationship embedding matrix with dimension d

As a preferred technical solution, the multimodal fusion is to use the attention mechanism to perform multimodal fusion of the obtained three kinds of user feature information embedding matrices, and assign different weights to each modality to reflect the difference between different modalities. After multi-modal fusion, the first user representation matrix Z _f is obtained; the calculation formula is:

Among them, α _c , α _T , and α _V are the weights of user name, text, and social relationship embedding matrix respectively; f(.) is the attention network.

As a preferred technical solution, the specific steps of said representation alignment strengthening user representation are:

First, put the first user representation into a fully connected layer to map the user representations of the two platforms into the same space to obtain the second user representation. The calculation formula of the second user representation is:

为平台N多模态融合得到的第一用户表示，Z为第二用户表示；Among them, W _l , b _l are the weight and bias of the fully connected layer respectively,

is the first user representation obtained by multi-modal fusion of platform N, and Z is the second user representation;

Secondly, in order to train all weights and offsets in this method, the minimum EMD distance is used as the first optimization goal, and the calculation formula of the first optimization goal is:

的第二用户表示

和用户

的第二用户表示

的距离，F_ij为用户

和用户

之间的关联概率，

表示F范数的平方；Among them, L _E is the first optimization objective, d _ij is the user

The second user of

and user

The second user of

distance, F _ij is the user

and user

The correlation probability between

Indicates the square of the F norm;

In addition, by reducing the representation distance between user pairs and the difference between P _ij and F _ij , a second optimization objective is set to better guide the learning of the second user representation, the calculation formula of the second optimization objective is:

Among them, _LR is the second optimization objective, n _p is the number of associated user sample pairs, λ ₁ and λ ₂ are hyperparameters, and for associated user sample pairs, the true association probability P _ij =1;

Realizing the final optimization objective L is the sum of the first optimization objective and the second optimization objective, namely:

L＝L _E +L _R

Finally, L is continuously optimized by the gradient descent method to obtain the optimal weight and bias, and finally the second user representation Z is obtained according to the optimal W _l and b _l .

As a preferred technical solution, the identity association result is calculated by calculating the cosine similarity between the representations of the second users, and the calculation formula is as follows:

为平台N¹的用户

的第二用户表示和

为平台N²的用户

的第二用户表示，S_ij为用户

和用户

的余弦相似性。in,

For users of platform N ¹

A second user representation of and

For users of platform N ²

The second user of , S _ij is the user

and users

cosine similarity of .

In the second aspect, the present invention also provides a cross-social network virtual identity association system based on multi-modal fusion and representation alignment, applying the multi-modal fusion and representation alignment cross-social network virtual identity association method, including features Extraction module, multimodal fusion module, representation alignment module and identity association module;

The feature extraction module is used to perform feature extraction on user names of social networks on different platforms, texts published by users, and user social relations, and respectively obtain user name feature information, text feature information published by users, and user social relationship feature information;

The multi-modal fusion module is used to perform multi-modal fusion by using an attention mechanism according to the three kinds of user feature information, and obtain a first user representation that integrates multi-dimensional features;

The representation alignment module is configured to strengthen the user representation through representation alignment of the first user representation, and finally obtain a second user representation with the same distribution on different platforms;

The identity association module is used to calculate the cosine similarity between the second user representations, obtain the similarity score between users, and use the user pair with the highest score as the identity association result.

In a third aspect, the present invention also provides an electronic device, the electronic device comprising:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores computer program instructions executable by the at least one processor, the computer program instructions are executed by the at least one processor, so that the at least one processor can perform the multimodal-based Fusion and representation alignment approach to virtual identity association across social networks.

In the fourth aspect, the present invention also provides a computer-readable storage medium, which stores a program, and when the program is executed by a processor, implements the cross-social network virtual identity association method based on multimodal fusion and representation alignment .

Compared with the prior art, the present invention has the following advantages and beneficial effects:

1. The present invention fully mines three modal information of user name, user published content and social relationship. Based on the character-level Bag-of-Words model to extract the specific composition features of each user name; extract text and social relationship features from Word2Vec and DeepWalk models, and on this basis, use the attention mechanism to automatically learn feature weights for multi-modal fusion , to a certain extent, it solves the problem that the existing methods cannot fully describe the user using single-modal information, or cannot perfectly integrate multi-modal information;

2. After the user representation is obtained, the present invention further strengthens the user representation through representation alignment, so that user representations belonging to the same natural person on different platforms are as close as possible, and solve the problem of data distribution differences between different social platforms.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 is a frame diagram of a cross-social network virtual identity association method based on multimodal fusion and representation alignment according to an embodiment of the present invention;

FIG. 2 is a block diagram of a cross-social network virtual identity association system based on multimodal fusion and representation alignment according to an embodiment of the present invention;

FIG. 3 is a structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Apparently, the described embodiments are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of this application.

Reference in this application to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrences of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

Multimodal fusion: Multimodal fusion refers to the process of combining information from two or more modalities to make predictions. In the process of prediction, a single modality usually cannot contain all the effective information required to produce accurate prediction results. The multimodal fusion process combines information from two or more modalities to achieve information supplementation and broaden the input data. The coverage of information improves the accuracy of prediction results and the robustness of prediction models.

Please refer to FIG. 1 , in one embodiment of the present application, a cross-social network virtual identity association method for multi-modal fusion and representation alignment is provided, including the following steps:

S1. Feature extraction is performed on user names, texts published by users, and social relationships of users in different social networks, and feature information of user names, text feature information published by users, and user social relationship feature information are respectively obtained.

S11. Feature extraction of the user name, specifically:

如用户名“abza12”中a：2，b：1，z：1，“1”：1，“2”：1，因此，得到向量

将得到的所有用户名向量依次拼接得到用户名计数矩阵

由于C0是一个稀疏矩阵，为此使用一个自动编码器将其进行转换，转换的公式的具体为：For the username "abza12" of a given user, the character-level Bag-of-Words model is used for feature extraction, and the number of occurrences of each character in each username is counted to obtain a vector

For example, in the user name "abza12", a: 2, b: 1, z: 1, "1": 1, "2": 1, therefore, get the vector

Concatenate all the obtained username vectors in sequence to obtain a username count matrix

Since C0 is a sparse matrix, an autoencoder is used to convert it. The conversion formula is as follows:

和

分别为用户名向量，通过梯度下降不断训练L_c，得到最优的W_e和b_e，最终得到维度为d的用户名嵌入矩阵

Among them, W _e , be _e are the weights and biases of the encoder, W _d , b _d are the weights and biases of the decoder,

and

They are the username vectors respectively, and L _c is continuously trained through gradient descent to obtain the optimal We _and be _e , and finally obtain the username embedding matrix with dimension d

S12. Feature extraction of the text published by the user, specifically:

For the text published by the user, such as "Today is a sunny day.", after removing the stop words, input the resulting text into the Word2Vec model to obtain the vector representation of each word, and further add the word vectors of each text Get the embedding vector of each text, and then average the embedding vectors of all the texts published by each user as the representation of the user’s published text, concatenate the text embedding vectors of all users in turn to obtain a text embedding matrix with dimension d

S13. Feature extraction of the user social relationship, specifically:

For user social relations, if the i-th user on platform ^N1 has a friend relationship with the j-th user on platform ^N2 , the ijth position of the adjacency matrix is set to 1, and the n users on platform ^N1 and the mth user on platform ^N2 The n×m adjacency matrix obtained by the social relationship composed of four users is obtained through the DeepWalk model to obtain the embedding vector of each user's social relationship, and the social relationship embedding vectors of all users are sequentially spliced to obtain the user social relationship embedding matrix with dimension d

S2. According to the feature information, use the attention mechanism to perform multi-modal fusion to obtain a first user representation that combines multi-dimensional features.

S21. The multimodal fusion is an embedding matrix of the obtained three types of user feature information; the attention mechanism is used to perform multimodal fusion, and different weights are assigned to each modality to reflect the importance between different modalities, After multimodal fusion, the first user representation matrix Z _f is obtained; the calculation formula is:

Among them, α _C , α _T , and α _V are the weights of user name, text, and social relationship embedding matrix respectively; f(.) is the attention network.

S3. Strengthen the user representation by aligning the first user representation, and finally obtain a second user representation with the same distribution on different platforms.

S31. First, put the first user representation into a fully connected layer to map the user representations of the two platforms into the same space to obtain the second user representation. The calculation formula of the second user representation is:

为平台N多模态融合得到的第一用户表示，Z为第二用户表示。Among them, W _l , b _l are the weight and bias of the fully connected layer respectively,

is the first user representation obtained by multimodal fusion of platform N, and Z is the second user representation.

S32, secondly, using the minimum EMD distance as the first optimization objective, the calculation formula of the first optimization objective is:

的第二用户表示

和用户

的第二用户表示

的距离，F_ij为用户

和用户

之间的关联概率，

表示F范数的平方。Among them, L _E is the first optimization objective, d _ij is the user

The second user of

and user

The second user of

distance, F _ij is the user

and user

The correlation probability between

Indicates the square of the F-norm.

S33. In this method, by reducing the representation distance between pairs of users and the difference between P _ij and F _ij , a second optimization goal is set to better guide the learning of the second user representation. The calculation formula of the second optimization goal is: :

Among them, _LR is the second optimization objective, n _p is the number of associated user sample pairs, λ ₁ and λ ₂ are hyperparameters, and for associated user sample pairs, the true association probability P _ij =1;

Realizing the final optimization objective L is the sum of the first optimization objective and the second optimization objective, namely:

L＝L _E +L _R

Finally, L is continuously optimized by the gradient descent method to obtain the optimal weight and bias, and finally the second user representation Z is obtained according to the optimal W _l and b _l .

S4. Calculate the cosine similarity between the second user representations to obtain a similarity score between users, and use the user pair with the highest score as an identity association result.

S41. Computing the second user representation cosine similarity between the platform N ¹ and platform N ² users:

为平台N¹的用户

的第二用户表示和

为平台N²的用户

的第二用户表示，S_ij为用户

和用户

的余弦相似性；最终根据用户之间的相似性得分，并将得分最高的用户对作为身份关联结果。in,

For users of platform N ¹

A second user representation of and

For users of platform N ²

The second user of , S _ij is the user

and user

The cosine similarity of ; Finally, according to the similarity score between users, the user pair with the highest score is used as the identity association result.

It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because Certain steps may be performed in other orders or simultaneously in accordance with the present invention.

Based on the same idea as the cross-social network virtual identity association method based on multimodal fusion and representation alignment in the above-mentioned embodiments, the present invention also provides a cross-social network virtual identity association system based on multimodal fusion and representation alignment. The system can be used to implement the above-mentioned cross-social network virtual identity association method based on multimodal fusion and representation alignment. For the sake of illustration, in the schematic structural diagram of the embodiment of the cross-social network virtual identity association system based on multimodal fusion and representation alignment, only the parts related to the embodiment of the present invention are shown. Those skilled in the art can understand that the illustrated structure The device is not intended to be limited and may include more or fewer components than shown, or combine certain components, or have different arrangements of components.

Please refer to Fig. 2, in another embodiment of the present application, a cross-social network virtual identity association system 100 based on multimodal fusion and representation alignment is provided, the system includes a feature extraction module 101, a multimodal fusion module 102. Representation alignment module 103 and identity association module 104

The feature extraction module 101 is used to perform feature extraction on user names of social networks on different platforms, texts published by users, and user social relations, and respectively obtain user name feature information, text feature information published by users, and user social relationship feature information ;

The multimodal fusion module 102 is configured to perform multimodal fusion using an attention mechanism according to the three types of user feature information, and obtain a first user representation that combines multidimensional features;

The representation alignment module 103 is configured to strengthen the user representation through representation alignment of the first user representation, and finally obtain a second user representation with the same distribution on different platforms;

The identity association module 104 is configured to calculate the cosine similarity between the second user representations, obtain the similarity score between users, and use the user pair with the highest score as the identity association result.

It should be noted that the cross-social network virtual identity association system based on multimodal fusion and representation alignment of the present invention corresponds to the cross-social network virtual identity association method based on multimodal fusion and representation alignment of the present invention. The technical features and beneficial effects described in the embodiment of the cross-social network virtual identity association method based on multi-modal fusion and representation alignment are applicable to the embodiment of the cross-social network virtual identity association system based on multi-modal fusion and representation alignment, For specific content, please refer to the description in the method embodiment of the present invention, which will not be repeated here, and is hereby declared.

In addition, in the implementation of the cross-social network virtual identity association system of multi-modal fusion and representation alignment in the above-mentioned embodiment, the logical division of each program module is only an example. Considering the configuration requirements or the convenience of software implementation, the above-mentioned function allocation is completed by different program modules, that is, the internal structure of the cross-social network virtual identity association system based on multi-modal fusion and representation alignment is divided into different program modules, To complete all or part of the functions described above.

Referring to FIG. 3 , in one embodiment, an electronic device for implementing a cross-social network virtual identity association method based on multimodal fusion and representation alignment is provided, and the electronic device 200 may include a first processor 201, a second A memory 202 and a bus may also include a computer program stored in the first memory 202 and operable on the first processor 201, such as a cross-social network virtual identity association program for multimodal fusion and representation alignment 203.

Wherein, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example: SD or DX memory, etc.), a magnetic memory, Disk, CD, etc. The first storage 202 may be an internal storage unit of the electronic device 200 in some embodiments, such as a mobile hard disk of the electronic device 200 . The first memory 202 may also be an external storage device of the electronic device 200 in other embodiments, such as a plug-in mobile hard disk equipped on the electronic device 200, a smart memory card (Smart Media Card, SMC), a secure digital ( SecureDigital, SD) card, flash memory card (Flash Card), etc. Further, the first memory 202 may also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various data installed in the electronic device 200, such as the code of the cross-social network virtual identity association program 203 for multimodal fusion and representation alignment, but also can be used for temporary Store the data that has been output or will be output accurately.

In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same function or different functions packaged, including one or A combination of multiple central processing units (Central Processing unit, CPU), microprocessors, digital processing chips, graphics processors and various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect various components of the entire electronic device, and runs or executes programs stored in the first memory 202 or modules, and call data stored in the first memory 202 to execute various functions of the electronic device 200 and process data.

FIG. 3 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation to the electronic device 200, and may include fewer or more components, or combinations of certain components, or different arrangements of components.

The cross-social network virtual identity association program 203 stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, can be realised:

Feature extraction is performed on user names, texts published by users, and social relationships of users in different social networks, and feature information of user names, text features published by users, and user social relationship feature information are obtained respectively;

Using the attention mechanism to perform multi-modal fusion according to the feature information, to obtain a first user representation that combines multi-dimensional features;

Strengthening the user representation by aligning the first user representation, and finally obtaining a second user representation with the same distribution on different platforms;

Calculate the cosine similarity between the second user representations to obtain the similarity score between users, and use the user pair with the highest score as the identity association result.

Further, if the integrated modules/units of the electronic device 200 are realized in the form of software function units and sold or used as independent products, they may be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM, Read-Only Memory) .

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be realized through computer programs to instruct related hardware, and the programs can be stored in a non-volatile computer-readable storage medium When the program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiment is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited by the above-mentioned embodiment, and any other changes, modifications, substitutions, combinations, Simplifications should be equivalent replacement methods, and all are included in the protection scope of the present invention.

Claims (10)

1. A cross-social network virtual identity association method based on multimodal fusion and representation alignment, characterized in that it comprises the following steps: Feature extraction is performed on the user names, texts published by users and social relations of users of different social network users, and the characteristic information of user names, text characteristic information published by users and user social relation characteristic information are respectively obtained; According to the user name feature information obtained, the text feature information published by the user, and the user social relationship feature information, the attention mechanism is used to perform multimodal fusion to obtain a first user representation that combines multi-dimensional features; performing user representation enhancement processing on the first user representation through a representation alignment method, and finally obtaining second user representations with the same distribution space on different platforms; Calculate the cosine similarity between the second user representations to obtain the similarity score between users, and use the user pair with the highest score as the identity association result. 2. according to claim 1, based on the cross-social network virtual identity association method of multimodal fusion and representation alignment, it is characterized in that the feature extraction of the user name is specifically: For the username of a given user, the character-level Bag-of-Words model is used for feature extraction, and the number of occurrences of each character in each username is counted to obtain a vector

Concatenate all the obtained username vectors in sequence to obtain a username count matrix

Since C ⁰ is a sparse matrix, an autoencoder is used to convert it, and the conversion formula is as follows:

Among them, W _e , be _e are the weights and biases of the encoder, W _d , b _d are the weights and biases of the decoder, C ¹ is the decoder username vector matrix,

and

They are the username vectors, and the loss function L _c is continuously trained through gradient descent to obtain the optimal W _e and be _e , and finally a username embedding matrix with dimension d

3. according to claim 1, based on the cross-social network virtual identity association method of multimodal fusion and representation alignment, it is characterized in that the feature extraction of the text published by the user is specifically: Input the text published by the user into the Word2Vec model to obtain the embedding vector of each text, and then average the embedding vector of each user’s published text as the representation of the user’s published text, and splice the text embedding vectors of all users in turn, Get the text embedding matrix with dimension d

4. according to claim 1, based on the cross-social network virtual identity association method of multimodal fusion and representation alignment, it is characterized in that the feature extraction of the user social relationship is specifically: The n×m adjacency matrix obtained from the social relations composed of n users on platform N ¹ and m users on platform N ² is obtained through the DeepWalk model to obtain the embedding vector of each user's social relations, and the social relation embedding vectors of all users are concatenated in sequence , get the user social relationship embedding matrix with dimension d

5. The cross-social network virtual identity association method based on multimodal fusion and representation alignment according to claim 1, wherein the multimodal fusion is an embedded matrix of three kinds of user characteristic information obtained, and the attention is used to The force mechanism performs multi-modal fusion, and assigns different weights to each mode to reflect the importance of different modes. After multi-modal fusion, the first user representation matrix Z _f is obtained; the calculation formula is:

Among them, α _C , α _T , and α _V are the weights of user name, text, and social relationship embedding matrix respectively; f(.) is the attention network. 6. According to the cross-social network virtual identity association method based on multi-modal fusion and representation alignment according to claim 1, it is characterized in that, the specific steps of said representation alignment strengthening user representation are: First, put the first user representation into a fully connected layer to map the user representations of the two platforms into the same space to obtain the second user representation. The calculation formula of the second user representation is:

Among them, W _l , b _l are the weight and bias of the fully connected layer respectively,

is the first user representation obtained by multi-modal fusion of platform N, and Z is the second user representation; Secondly, in order to train all weights and offsets in this method, the minimum EMD distance is used as the first optimization goal, and the calculation formula of the first optimization goal is:

Among them, L _E is the first optimization objective, d _ij is the user

The second user of

and user

The second user of

distance, F _ij is the user

and user

The correlation probability between

Indicates the square of the F norm; In addition, by reducing the representation distance between user pairs and the difference between P _ij and F _ij , a second optimization objective is set to better guide the learning of the second user representation, the calculation formula of the second optimization objective is:

Among them, _LR is the second optimization objective, n _p is the number of associated user sample pairs, λ ₁ and λ ₂ are hyperparameters, and for associated user sample pairs, the true association probability P _ij =1; Realizing the final optimization goal L is the sum of the first optimization goal and the second optimization goal, namely: L＝L _E +L _R Finally, L is continuously optimized by the gradient descent method to obtain the optimal weight and bias, and finally the second user representation Z is obtained according to the optimal W _l and b _l . 7. The cross-social network virtual identity association method based on multimodal fusion and representation alignment according to claim 1, wherein the identity association result is calculated by calculating the cosine similarity between the second user representations, and the calculation formula as follows:

in,

For users of platform N ¹

A second user representation of and

For users of platform N ²

The second user of , S _ij is the user

and user

cosine similarity of . 8. A cross-social network virtual identity association system based on multi-modal fusion and representation alignment, characterized in that it is applied to the cross-social network virtual identity of multi-modal fusion and representation alignment described in any one of claims 1-7 Association methods, including feature extraction modules, multimodal fusion modules, representation alignment modules, and identity association modules; The feature extraction module is used to perform feature extraction on user names of social networks on different platforms, texts published by users, and user social relations, and respectively obtain user name feature information, text feature information published by users, and user social relationship feature information; The multimodal fusion module is used to perform multimodal fusion using an attention mechanism according to the three types of user feature information, to obtain a first user representation that integrates multi-dimensional features; The representation alignment module is configured to strengthen the user representation through representation alignment of the first user representation, and finally obtain a second user representation with the same distribution on different platforms; The identity association module is used to calculate the cosine similarity between the second user representations, obtain the similarity score between users, and use the user pair with the highest score as the identity association result. 9. An electronic device, characterized in that the electronic device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; wherein, Said memory stores computer program instructions executable by said at least one processor, said computer program instructions being executed by said at least one processor, to enable said at least one processor to perform the Any one of the cross-social network virtual identity association methods based on multimodal fusion and representation alignment. 10. A computer-readable storage medium, storing a program, characterized in that, when the program is executed by a processor, the cross-social interaction based on multimodal fusion and representation alignment described in any one of claims 1-7 is realized. Network virtual identity association method.

Images