CN114169515B - Social relationship identification method based on high-order graph neural network - Google Patents
Social relationship identification method based on high-order graph neural network Download PDFInfo
- Publication number
- CN114169515B CN114169515B CN202010845651.6A CN202010845651A CN114169515B CN 114169515 B CN114169515 B CN 114169515B CN 202010845651 A CN202010845651 A CN 202010845651A CN 114169515 B CN114169515 B CN 114169515B
- Authority
- CN
- China
- Prior art keywords
- features
- model
- graph
- neural network
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a social relation recognition method based on a high-order graph neural network, which mainly relates to the problems that related features are firstly extracted to construct a graph structure in the field of deep learning, then the high-order graph neural network is utilized to conduct graph reasoning, and finally a classifier is used for classifying social relation. The method comprises the following steps: respectively extracting two faces, a person pair joint area, a scene and a person pair space position feature through four pre-training models and a full connection layer; then, the extracted features are used as nodes to construct a fully-connected undirected graph, and the undirected graph is sent into a graph neural network for graph reasoning; and finally, classifying social relations by using 8 two classifiers in combination with graph reasoning results. According to the method, the pre-training model and one full-connection layer are utilized to fully extract all levels of features, social relation information among all levels of features is effectively inferred and hidden through the high-order graph neural network, the effect of the classifier is improved, and social relation identification with higher accuracy is achieved.
Description
Technical Field
The invention relates to a social relationship identification problem in the field of deep learning, in particular to a social relationship identification method based on a high-order graph neural network.
Background
In the field of computer vision, social relationships are important clues for understanding the behavior of people in images, etc., while social relationship identification based on images is a key method for understanding people's daily interactive communication. Most of the existing researches are used for identifying social relations by extracting relevant features of faces, bodies and the whole image, so that higher accuracy is achieved. In addition, the graph neural network is used as a brand new network, can simulate human thinking to infer the graph, and provides a new tool and method for social relationship identification. Currently, social relationship identification plays an important role in related applications such as social robots, social media, urban public spaces and the like.
Social relationship identification is an important research content in the field of computer vision, and is widely paid attention to researchers at home and abroad. The related research method is basically limited to carrying out social relation classification by directly splicing and fusing the extracted related features, ignoring the correlation among the features and failing to mine social relation information therein. The graph neural network, in particular the higher-order graph neural network, can just perform the task, and the correlation reasoning among the features is completed. Therefore, the method and the device respectively extract the characteristics of two faces of the character pairs, the characteristics of the character pair combining area, the scene characteristics of the whole picture and the relative spatial position characteristics among the faces in the image through the pre-training model and a full-connection layer, construct a full-connection undirected graph by taking the characteristics as nodes, then take the undirected graph as the input of a graph neural network to carry out graph reasoning, and finally further classify reasoning results through 8 classifiers, thereby improving the recognition accuracy of social relations.
Disclosure of Invention
The invention aims to provide a social relation recognition method based on a high-order graph neural network, which introduces the high-order graph neural network to infer graphs formed by various layers of features, fully learns related information among the features, and effectively solves the problem that social relation information among the features cannot be utilized in social relation recognition.
For ease of description, the following concepts are first introduced:
pre-training model: the training of neural network models requires a lot of data and time and sufficient computing resources, and in order to avoid repeated training of the network, model parameters with good effects trained by other researchers are migrated to the model in a specific task and fine-tuned to adapt to the requirements of the task.
Graph (Graph): as shown in fig. 1, the graph in the graph theory is a graph in a non-euclidean space, and is composed of nodes (nodes) and edges (edges) connecting the nodes.
Fig. neural network (Graph Neural Network, GNN): the neural network structure directly calculated on the graph learns the expression of the nodes in a message transmission mode, and updates the information of the current node by the adjacent nodes until the whole graph converges to a stable state.
High-order graph neural network (Higher-Order Graph Neural Network, k-GNN): expanding on the basis of the graph neural network, not only focusing on each node in the graph, but also taking into consideration subsets of the nodes in the graph, and carrying out message transmission among the subsets so as to obtain structural information provided by the node subsets.
The invention adopts the following technical scheme:
the social relation recognition method based on the high-order graph neural network is mainly characterized by comprising the following steps of:
a. the method comprises the steps of extracting characteristics of different layers in a picture through different pre-training models in a targeted manner, wherein the characteristics comprise characteristics of two faces of a person pair, characteristics of a person pair joint area and scene characteristics of the whole picture;
b. the function of 8 classification is realized by constructing 8 classifiers, and the reasoning result is effectively utilized to accurately divide 8 types of social relations such as dominant (domino), competitive (competitive), trust (trust), warmth (warmm), friendly (friendly), intimate (confidential), tank integrity (remonstant), encouragement (assured) and the like;
c. utilizing the extracted features of the pre-training model and the calculated spatial position features as nodes to construct an undirected graph, and carrying out reasoning on the graph by simulating human thinking through a high-order graph neural network;
the method mainly comprises the following steps:
(1) Data processing and enhancement: uniformly cutting two faces serving as input character pairs, a character pair joint region and the whole picture into 224 multiplied by 224 images, and carrying out data enhancement processing such as random modification brightness, contrast, random affine transformation, normalization and the like, wherein the whole picture is additionally subjected to random horizontal overturning processing; in addition, the spatial position and area information of two faces of the person pair are normalized and then used as one path of input;
(2) Pre-training model selection: the pre-training models obtained by training on VGGFACE2 and ImageNet, places365 data sets are respectively used as feature extraction networks of two FACEs of a person pair, a person pair joint region and a whole picture, wherein the features of the FACEs are extracted by two VGG-FACE (RESNET-50) models, the features of the person pair joint region are extracted by a RESNET-101 model, the scene features are extracted by a RESNET-50 model applicable to scene classification tasks, and the last full-connection layer is deleted by each network;
(3) Model construction: the model consists of three parts, wherein the first part consists of 4 pre-trained RESNET models and 1 full-connection layer, and two face features, character pair combination region features, scene features and face space position features are extracted respectively; the second part consists of two first-order graph neural networks and two second-order or third-order graph neural networks in cascade connection; the third part is 8 classifiers composed of two full-connection layers respectively;
(4) Model training and storage: the 5 paths of data processed in the step (1) are respectively used as the input of a pre-training model and a full-connection layer, the characteristics extracted from the first part of the model are used as nodes, an undirected graph is constructed in a full-connection mode to be used as the input of a graph neural network to conduct graph reasoning, then the reasoning results are respectively sent to 8 two classifiers to conduct social relationship classification, finally model loss is calculated according to the corresponding labels in the data set and the classification results, model parameters are reversely propagated and updated, and the step is repeated until a model with the highest accuracy is trained and stored;
(5) Social relationship identification: and (3) identifying social relationship of the picture with the face boundary box by utilizing the model obtained by training in the step (4).
The beneficial effects of the invention are as follows:
(1) The pre-training model is fully utilized for feature extraction, so that a great amount of training time and calculation resources are saved.
(2) A method for constructing a graph structure using hierarchical features as nodes is provided.
(3) The graph neural network is introduced to simulate human thinking to conduct graph reasoning on the graph formed by the features of each layer, and social relation information among the features is effectively mined.
(4) And decomposing the task of 8 categories into 8 tasks of two categories, and classifying social relations by maximally utilizing the result of graph reasoning.
Drawings
Fig. 1 is a diagram structure in a non-european space.
Fig. 2 is a diagram showing the overall structure of the model.
Detailed description of the preferred embodiments
The present invention will be described in further detail with reference to the drawings and examples, which are only for further illustration and are not to be construed as limiting the scope of the invention, and some insubstantial modifications and adaptations of the invention to one of ordinary skill in the art based on the foregoing disclosure should and are intended to be within the scope of the invention.
The social relation recognition method based on the high-order graph neural network specifically comprises the following steps:
(1) Data processing and enhancement
The data set provides related information of each picture, including picture file names, boundary boxes of two faces and 8 kinds of labels, and the pictures can be cut through the information to obtain pictures of figures on the two faces and pictures of the combined areas of the figures.
Uniformly cutting two faces serving as input character pairs, a character pair joint region and the whole picture into 224 multiplied by 224 images, and carrying out data enhancement processing such as random modification brightness, contrast, random affine transformation, normalization and the like, wherein the whole picture is additionally subjected to random horizontal overturning processing; in addition, the spatial position and area information of the two faces of the person pair are normalized and then are input as one path. The spatial position and area information of the single face are shown in formula (1):
b pos ＝{x min ,y min ,x max ,y max ,area} (1)
wherein x is min ,y min ,x max ,y max The minimum value of the horizontal coordinate, the minimum value of the vertical coordinate, the maximum value of the horizontal coordinate and the maximum value of the vertical coordinate of the human face boundary box are respectively shown, and area represents the area of the boundary box.
(2) Pre-training model selection
The pre-training models obtained by training on VGGFACE2 and ImageNet, places365 data sets are respectively used as feature extraction networks of two FACEs of a person pair, a person pair joint region and a whole picture, wherein the features of the FACEs are extracted by two VGG-FACE (RESNET-50) models, the features of the person pair joint region are extracted by a RESNET-101 model, the scene features are extracted by a RESNET-50 model applicable to scene classification tasks, and the last full-connection layer is deleted by each network.
(3) Model construction
As shown in fig. 2, the model is composed of three parts, the first part is composed of 4 pre-trained rest models and 1 full connection layer, and two face features, character pair combination region features, scene features and face space position features are extracted respectively; the second part consists of two first-order graph neural networks and two second-order or third-order graph neural networks in cascade connection; the third part is 8 classifiers composed of two full-connection layers respectively.
(4) Model training and preservation
And (3) respectively taking 5 paths of data processed in the step (1) as input of a pre-training model and a full-connection layer, taking the characteristics extracted from the first part of the model as nodes, constructing an undirected graph in a full-connection mode as input of a graph neural network to perform graph reasoning, respectively sending reasoning results into 8 two classifiers to perform social relationship classification, finally calculating model loss according to the corresponding labels and classification results in the data set, reversely transmitting and updating model parameters, and repeating the step until a model with the highest accuracy is trained and stored.
(5) Social relationship identification
And (3) identifying social relationship of the picture with the face boundary box information by utilizing the model obtained by training in the step (4).
Claims (3)
1. A social relation recognition method based on a high-order graph neural network is characterized by comprising the following steps of:
a. the method comprises the steps of extracting characteristics of different layers in a picture through different pre-training models in a targeted manner, wherein the characteristics comprise characteristics of two faces of a person pair, characteristics of a person pair joint area and scene characteristics of the whole picture;
b. the function of 8 classification is realized by constructing 8 classifier, and the reasoning result is effectively utilized to accurately divide the 8 social relations of leading, competing, trusting, warmth, friendliness, intimacy, tank integrity and encouragement;
c. utilizing the extracted features of the pre-training model and the calculated spatial position features as nodes to construct an undirected graph, and carrying out reasoning on the graph by simulating human thinking through a high-order graph neural network;
the method mainly comprises the following steps:
(1) Data processing and enhancement: uniformly cutting two faces serving as input character pairs, a character pair joint region and the whole picture into 224 multiplied by 224 images, and carrying out random brightness modification, contrast, random affine transformation and normalized data enhancement processing, wherein the whole picture is additionally subjected to random horizontal overturning processing; in addition, the spatial position and area information of two faces of the person pair are normalized and then used as one path of input;
(2) Pre-training model selection: the pre-training models obtained by training on VGGFACE2 and ImageNet, places365 data sets are respectively used as feature extraction networks of two faces of a person pair, a person pair joint area and a whole picture, and the last full-connection layer of each network is deleted;
(3) Model construction: the model consists of three parts, wherein the first part consists of 4 pre-trained RESNET models and 1 full-connection layer, and two face features, character pair combination region features, scene features and face space position features are extracted respectively; the second part consists of two first-order graph neural networks and two second-order or third-order graph neural networks in cascade connection; the third part is 8 classifiers composed of two full-connection layers respectively;
(4) Model training and storage: the 5 paths of data processed in the step (1) are respectively used as the input of a pre-training model and a full-connection layer, a full-connection undirected graph is constructed through the features extracted from the first part of the model to be used as the input of a graph neural network, the reasoning and the acquisition of node level knowledge and structural information are realized, the reasoning results are respectively sent to 8 two classifiers to carry out social relationship classification, finally model loss is calculated according to the corresponding labels and classification results in the data set, model parameters are reversely propagated and updated, and the step is repeated until the model with the highest accuracy is trained and stored;
(5) Social relationship identification: and (3) identifying social relationship of the picture with the face boundary box information by utilizing the model obtained by training in the step (4).
2. The social relationship identifying method based on the higher-order graph neural network as claimed in claim 1, wherein in the step (2), corresponding task networks are selected for different levels of features, wherein features of FACEs are extracted by two VGG-FACE models, features of a person pair joint region are extracted by a rest-101 model, and scene features are extracted by a rest-50 model suitable for scene classification tasks.
3. The social relationship identifying method based on the higher-order graph neural network as claimed in claim 1, wherein in the step (3), the 8-classification task is decomposed into 8-classification tasks, so that the model predicts the social relationship more accurately.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010845651.6A CN114169515B (en) | 2020-08-20 | 2020-08-20 | Social relationship identification method based on high-order graph neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010845651.6A CN114169515B (en) | 2020-08-20 | 2020-08-20 | Social relationship identification method based on high-order graph neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114169515A CN114169515A (en) | 2022-03-11 |
| CN114169515B true CN114169515B (en) | 2023-04-28 |
Family
ID=80475345
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010845651.6A Active CN114169515B (en) | 2020-08-20 | 2020-08-20 | Social relationship identification method based on high-order graph neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114169515B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109471959A (en) * | 2018-06-15 | 2019-03-15 | 中山大学 | Personage's social relationships discrimination method and system in image based on figure inference pattern |
| CN109948447A (en) * | 2019-02-21 | 2019-06-28 | 山东科技大学 | The discovery of personage's cyberrelationship and evolution rendering method based on video image identification |
| WO2020114122A1 (en) * | 2018-12-07 | 2020-06-11 | 阿里巴巴集团控股有限公司 | Neural network system and method for analyzing relationship network graph |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017000300A1 (en) * | 2015-07-02 | 2017-01-05 | Xiaoou Tang | Methods and systems for social relation identification |
-
2020
- 2020-08-20 CN CN202010845651.6A patent/CN114169515B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109471959A (en) * | 2018-06-15 | 2019-03-15 | 中山大学 | Personage's social relationships discrimination method and system in image based on figure inference pattern |
| WO2020114122A1 (en) * | 2018-12-07 | 2020-06-11 | 阿里巴巴集团控股有限公司 | Neural network system and method for analyzing relationship network graph |
| CN109948447A (en) * | 2019-02-21 | 2019-06-28 | 山东科技大学 | The discovery of personage's cyberrelationship and evolution rendering method based on video image identification |
Non-Patent Citations (1)
| Title |
|---|
| 何昊晨 ; 张丹红 ; .基于多维社交关系嵌入的深层图神经网络推荐方法.计算机应用.2020,(第10期),13-21. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114169515A (en) | 2022-03-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ray et al. | Sunny and dark outside?! improving answer consistency in vqa through entailed question generation | |
| US20190303768A1 (en) | Community Question Answering-Based Article Recommendation Method, System, and User Device | |
| Cowell | Introduction to inference for Bayesian networks | |
| CN114064918B (en) | Multi-modal event knowledge graph construction method | |
| CN111400452B (en) | Text information classification processing method, electronic device and computer readable storage medium | |
| CN112016601B (en) | Network model construction method based on knowledge graph enhanced small sample visual classification | |
| CN112364161B (en) | Microblog theme mining method based on dynamic behaviors of heterogeneous social media users | |
| CN105631749A (en) | User portrait calculation method based on statistical data | |
| CN113065974A (en) | Link prediction method based on dynamic network representation learning | |
| Zhang et al. | Dynamical analysis of Hyper-SIR rumor spreading model | |
| CN113641797A (en) | Data processing method, device, equipment, storage medium and computer program product | |
| CN114610910A (en) | Group perception-oriented teacher-student group privacy protection method and system | |
| CN113254652A (en) | Social media posting authenticity detection method based on hypergraph attention network | |
| CN116166910A (en) | Social media account vermicelli water army detection method, system, equipment and medium | |
| CN111598252A (en) | University computer basic knowledge problem solving method based on deep learning | |
| CN113191144B (en) | Network rumor recognition system and method based on propagation influence | |
| CN117556802A (en) | User portrait method, device, equipment and medium based on large language model | |
| CN114169515B (en) | Social relationship identification method based on high-order graph neural network | |
| CN113360664A (en) | Knowledge graph complementing method | |
| Manna et al. | Exploring topic models on short texts: a case study with crisis data | |
| CN113010772B (en) | Data processing method, related equipment and computer readable storage medium | |
| CN115858943A (en) | Social relationship identification method based on Transformer | |
| CN115168678A (en) | Time sequence perception heterogeneous graph nerve rumor detection model | |
| CN114493905B (en) | Social relationship identification method based on multilevel feature fusion | |
| CN114168769A (en) | Visual question-answering method based on GAT (generic object transform) relational reasoning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |