CN114169515A - Social relationship identification method based on high-order graph neural network - Google Patents

Social relationship identification method based on high-order graph neural network Download PDF

Info

Publication number
CN114169515A
CN114169515A CN202010845651.6A CN202010845651A CN114169515A CN 114169515 A CN114169515 A CN 114169515A CN 202010845651 A CN202010845651 A CN 202010845651A CN 114169515 A CN114169515 A CN 114169515A
Authority
CN
China
Prior art keywords
features
graph
neural network
model
graph neural
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010845651.6A
Other languages
Chinese (zh)
Other versions
CN114169515B (en
Inventor
卿粼波
高建军
李林东
吴晓红
陈洪刚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan University
Original Assignee
Sichuan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sichuan University filed Critical Sichuan University
Priority to CN202010845651.6A priority Critical patent/CN114169515B/en
Publication of CN114169515A publication Critical patent/CN114169515A/en
Application granted granted Critical
Publication of CN114169515B publication Critical patent/CN114169515B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a social relationship recognition method based on a high-order graph neural network, and mainly relates to the problems that relevant features are extracted firstly to construct a graph structure, then graph reasoning is carried out by using the high-order graph neural network, and finally a classifier is used for carrying out social relationship classification in the field of deep learning. The method comprises the following steps: respectively extracting two human faces, a character pair combined area, a scene and character pair space position characteristics through four pre-training models and a full connection layer; then, the extracted features are used as nodes to construct a fully-connected undirected graph and are sent to a graph neural network for graph reasoning; and finally, classifying the social relationship by using 8 secondary classifiers and combining graph reasoning results. According to the method, the pre-training model and a full-connection layer are utilized to fully extract the characteristics of each layer, the social relationship information hidden among the characteristics of each layer is effectively inferred through the high-order graph neural network, the effect of the classifier is improved, and the social relationship recognition with high accuracy is achieved.

Description

Social relationship identification method based on high-order graph neural network
Technical Field
The invention relates to the problem of social relationship identification in the field of deep learning, in particular to a social relationship identification method based on a high-order graph neural network.
Background
In the field of computer vision, social relationships are important clues for understanding the behavior of people in images, and image-based social relationship identification is a key method for understanding daily interactive communication of people. Most of the existing researches carry out social relationship recognition by extracting relevant features of human faces, human bodies and the whole image, and higher accuracy is obtained. In addition, the graph neural network is a brand-new network, can simulate human thinking to carry out reasoning on the graph, and provides a new tool and a new method for social relationship recognition. Currently, social relationship identification plays an important role in social robots, social media, urban public spaces, and other related applications.
Social relationship recognition has received wide attention from researchers at home and abroad as an important research content in the field of computer vision. The related research method is basically limited to the classification of social relations by directly splicing and fusing the extracted related features, the correlation among the features is ignored, and the social relation information in the features cannot be mined. The graph neural network, especially the high-order graph neural network, is just competent for the task, and the relevance reasoning of the characteristics is completed. Therefore, the method comprises the steps of respectively extracting two face features of a character pair, the feature of a character pair combined area, the scene feature of a whole picture and the relative spatial position feature between faces in an image through a pre-training model and a full connection layer, constructing a full connection undirected graph by taking the features as nodes, then performing graph reasoning by taking the undirected graph as input of a graph neural network, and finally further classifying reasoning results by 8 secondary classifiers, so that the identification accuracy of social relations is improved.
Disclosure of Invention
The invention aims to provide a social relationship identification method based on a high-order graph neural network, which introduces the high-order graph neural network to reason about a graph formed by characteristics of each level, sufficiently learns related information among the characteristics and effectively solves the problem that the social relationship information among the characteristics cannot be utilized in the social relationship identification.
For convenience of explanation, the following concepts are first introduced:
pre-training the model: the training of the neural network model requires a large amount of data and time and sufficient computing resources, and in order to avoid repeated training of the network, model parameters trained by other researchers and having a good effect are migrated to the model in a specific task and fine-tuned to meet the requirements of the task.
FIG. (Graph): as shown in fig. 1, the graph in the graph theory is a graph in a non-euclidean space, and is composed of nodes (nodes) and edges (edges) connecting the nodes.
Graph Neural Network (GNN): the neural network structure directly calculated on the graph learns the expression of the nodes in a message transmission mode, and the information of the current node is updated by the adjacent nodes until the whole graph converges to a stable state.
High-level Graph Neural networks (high-Order Graph Neural networks, k-GNN): the method is developed on the basis of a graph neural network, only each node in the graph is not focused, and the subsets of the nodes in the graph are taken into consideration, and message transmission is carried out among the subsets, so that structural information provided by the node subsets is obtained.
The invention specifically adopts the following technical scheme:
a social relationship identification method based on a high-order graph neural network is provided, and the method is mainly characterized in that:
a. the method comprises the steps that features of different layers in a picture are extracted in a targeted mode through different pre-training models, wherein the features comprise the features of two human faces of a character pair, the features of a character pair combined area and the scene features of the whole picture;
b. 8 classification functions are realized by constructing 8 classifiers, the reasoning result is effectively utilized, and 8 types of social relationships such as leading (dominant), competitive, trusting (trusting), warm (warm), friendly (friendly), intimate (innolved), honest (devoting), encouragement (approved) and the like are accurately divided;
c. constructing an undirected graph by using the features extracted by the pre-training model and the spatial position features obtained by calculation as nodes, and reasoning the graph by simulating human thinking through a high-order graph neural network;
the method mainly comprises the following steps:
(1) data processing and enhancement: uniformly cutting two human faces serving as an input human pair, a human pair combined area and the whole picture into 224 x 224 images, and performing data enhancement processing such as random brightness modification, contrast modification, random affine transformation, normalization and the like, wherein random horizontal turning processing is additionally performed on the whole picture; in addition, the spatial position and area information of two human faces of the human figure pair are normalized and then are used as one path of input;
(2) pre-training model selection: pre-training models obtained by training on VGGFACE2, ImageNet and Places365 data sets are respectively used as feature extraction networks of two FACEs of a person pair, a person pair combined area and a whole picture, wherein the features of the FACEs are extracted by two VGG-FACE (RESNET-50) models, the features of the person pair combined area are extracted by a RESNET-101 model, scene features are extracted by a RESNET-50 model suitable for a scene classification task, and the last full connection layer is deleted by each network;
(3) constructing a model: the model consists of three parts, wherein the first part consists of 4 pre-trained RESNET models and 1 full connection layer, and two human face features, character pair combined region features, scene features and human face spatial position features are respectively extracted; the second part is composed of two first-order graph neural networks and two second-order or third-order graph neural network cascades; the third part is 8 classifiers composed of two full-connection layers;
(4) model training and saving: respectively taking the 5 paths of data processed in the step (1) as the input of a pre-training model and a full connection layer, taking the characteristics extracted from the first part of the model as nodes, constructing an undirected graph in a full connection mode as the input of a graph neural network to carry out graph reasoning, respectively sending the reasoning results into 8 secondary classifiers to carry out social relationship classification, finally calculating model loss according to corresponding labels and classification results in a data set and reversely propagating and updating model parameters, and repeating the step until a model with the highest accuracy is trained and stored;
(5) and (3) identifying social relations: and (4) carrying out social relationship recognition on the picture with the face boundary box by using the model obtained by training in the step (4).
The invention has the beneficial effects that:
(1) the pre-training model is fully utilized to extract the features, so that a large amount of training time and computing resources are saved.
(2) A method for constructing graph structures using hierarchical features as nodes is presented.
(3) The graph neural network is introduced to simulate human thinking to carry out graph reasoning on the graph formed by the features of each level, and the social relationship information among the features is effectively mined.
(4) And decomposing the 8 classified tasks into 8 two classified tasks, and performing social relationship classification by using the result of graph reasoning to the maximum extent.
Drawings
Fig. 1 is a diagram structure in a non-euclidean space.
Fig. 2 is a model overall structure.
Detailed description of the invention
The present invention is further described in detail with reference to the drawings and examples, it should be noted that the following examples are only for illustrating the present invention and should not be construed as limiting the scope of the present invention, and those skilled in the art should be able to make certain insubstantial modifications and adaptations to the present invention based on the above disclosure and should still fall within the scope of the present invention.
The social relationship identification method based on the high-order graph neural network specifically comprises the following steps:
(1) data processing and enhancement
The data set provides relevant information of each picture, including a picture file name, a bounding box of two faces and 8 types of labels, and the pictures can be cut through the information to obtain pictures of two faces of people and pictures of a joint area of people.
Uniformly cutting two human faces serving as an input human pair, a human pair combined area and the whole picture into 224 x 224 images, and performing data enhancement processing such as random brightness modification, contrast modification, random affine transformation, normalization and the like, wherein random horizontal turning processing is additionally performed on the whole picture; in addition, the spatial position and area information of the two faces of the person pair are normalized and then are used as one path of input. The spatial position and area information of a single face is shown in formula (1):
bpos＝{xmin,ymin,xmax,ymax,area} (1)
wherein x ismin,ymin,xmax,ymaxRespectively representing the minimum value of the abscissa, the minimum value of the ordinate, the maximum value of the abscissa and the maximum value of the ordinate of the human face boundary box, and area represents the area of the boundary box.
(2) Pre-training model selection
Pre-training models obtained by training on VGGFACE2, ImageNet and Places365 data sets are respectively used as feature extraction networks of two FACEs of a person pair, a person pair combined area and a whole picture, wherein the features of the FACEs are extracted by two VGG-FACE (RESNET-50) models, the features of the person pair combined area are extracted by a RESNET-101 model, scene features are extracted by a RESNET-50 model suitable for a scene classification task, and the last full connection layer is deleted in each network.
(3) Model construction
As shown in fig. 2, the model consists of three parts, the first part consists of 4 pre-trained RESNET models and 1 full-link layer, and two face features, a character pair union region feature, a scene feature and a face spatial position feature are respectively extracted; the second part is composed of two first-order graph neural networks and two second-order or third-order graph neural network cascades; the third part is 8 classifiers composed of two full-connection layers.
(4) Model training and preservation
And (2) respectively taking the 5 paths of data processed in the step (1) as the input of a pre-training model and a full connection layer, taking the characteristics extracted from the first part of the model as nodes, constructing an undirected graph in a full connection mode to be used as the input of a graph neural network for graph reasoning, then respectively sending the reasoning results into 8 secondary classifiers for social relationship classification, finally calculating model loss according to corresponding labels and classification results in a data set and reversely propagating and updating model parameters, and repeating the step until a model with the highest accuracy is trained and stored.
(5) Social relationship identification
And (4) carrying out social relationship recognition on the picture with the face boundary box information by using the model obtained by training in the step (4).

Claims (4)

1. A social relationship identification method based on a high-order graph neural network is characterized by comprising the following steps:
a. the features of different layers in the picture are extracted in a targeted manner through different pre-training models, wherein the features comprise the features of two human faces of a character pair, the features of a character pair combined area and the scene features of the whole picture;
b. 8 classification functions are realized by constructing 8 classifiers, the reasoning result is effectively utilized, and 8 types of social relationships such as leading (dominant), competitive, trusting (trusting), warm (warm), friendly (friendly), intimate (innolved), honest (devoting), encouragement (approved) and the like are accurately divided;
c. constructing an undirected graph by using the features extracted by the pre-training model and the spatial position features obtained by calculation as nodes, and reasoning the graph by simulating human thinking through a high-order graph neural network;
the method mainly comprises the following steps:
(1) data processing and enhancement: uniformly cutting two human faces serving as an input human pair, a human pair combined area and the whole picture into 224 x 224 images, and performing data enhancement processing such as random brightness modification, contrast modification, random affine transformation, normalization and the like, wherein random horizontal turning processing is additionally performed on the whole picture; in addition, the spatial position and area information of two human faces of the human figure pair are normalized and then are used as one path of input;
(2) pre-training model selection: pre-training models obtained by training on VGGFACE2, ImageNet and plates 365 data sets are respectively used as two faces of a person pair, a person pair combined area and a feature extraction network of a whole picture, and the final full-connection layer of each network is deleted;
(3) constructing a model: the model consists of three parts, wherein the first part consists of 4 pre-trained RESNET models and 1 full connection layer, and two human face features, character pair combined region features, scene features and human face spatial position features are respectively extracted; the second part is composed of two first-order graph neural networks and two second-order or third-order graph neural network cascades; the third part is 8 classifiers composed of two full-connection layers;
(4) model training and saving: respectively taking the 5 paths of data processed in the step (1) as the input of a pre-training model and a full connection layer, constructing an undirected graph through the characteristics extracted from the first part of the model, and using the undirected graph as the input of a graph neural network for reasoning, then respectively sending the reasoning results into 8 secondary classifiers for social relationship classification, finally calculating model loss according to the corresponding labels and the classification results in a data set and reversely propagating and updating model parameters, and repeating the step until a model with the highest accuracy is trained and stored;
(5) and (3) identifying social relations: and (4) carrying out social relationship recognition on the picture with the face boundary box information by using the model obtained by training in the step (4).
2. The method for identifying social relationships based on a high-order graph neural network as claimed in claim 1, wherein in step (2), corresponding task networks are selected for features of different levels, wherein features of human FACEs are extracted by two VGG-FACE (RESNET-50) models, features of human pairs in joint areas are extracted by a RESNET-101 model, and scene features are extracted by a RESNET-50 model suitable for a scene classification task.
3. The method for identifying social relationships based on a higher-order graph neural network as claimed in claim 1, wherein in step (3), 8 classification tasks are decomposed into 8 binary tasks, so that the model predicts the social relationships more accurately.
4. The method for identifying social relationships based on a high-order graph neural network as claimed in claim 1, wherein in step (4), the relationships of the features of each hierarchy are expressed in a graph mode, the features of each hierarchy are used as nodes, an undirected graph is constructed in a full-connected mode, and the graph is reasoned by simulating human thinking through the high-order graph neural network to obtain knowledge and structural information of node level.
CN202010845651.6A 2020-08-20 2020-08-20 Social relationship identification method based on high-order graph neural network Active CN114169515B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010845651.6A CN114169515B (en) 2020-08-20 2020-08-20 Social relationship identification method based on high-order graph neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010845651.6A CN114169515B (en) 2020-08-20 2020-08-20 Social relationship identification method based on high-order graph neural network

Publications (2)

Publication Number Publication Date
CN114169515A true CN114169515A (en) 2022-03-11
CN114169515B CN114169515B (en) 2023-04-28

Family

ID=80475345

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010845651.6A Active CN114169515B (en) 2020-08-20 2020-08-20 Social relationship identification method based on high-order graph neural network

Country Status (1)

Country Link
CN (1) CN114169515B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180144193A1 (en) * 2015-07-02 2018-05-24 Beijing Sensetime Technology Development Co., Ltd. Methods and systems for social relation identification
CN109471959A (en) * 2018-06-15 2019-03-15 中山大学 Personage's social relationships discrimination method and system in image based on figure inference pattern
CN109948447A (en) * 2019-02-21 2019-06-28 山东科技大学 The discovery of personage's cyberrelationship and evolution rendering method based on video image identification
WO2020114122A1 (en) * 2018-12-07 2020-06-11 阿里巴巴集团控股有限公司 Neural network system and method for analyzing relationship network graph

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180144193A1 (en) * 2015-07-02 2018-05-24 Beijing Sensetime Technology Development Co., Ltd. Methods and systems for social relation identification
CN109471959A (en) * 2018-06-15 2019-03-15 中山大学 Personage's social relationships discrimination method and system in image based on figure inference pattern
WO2020114122A1 (en) * 2018-12-07 2020-06-11 阿里巴巴集团控股有限公司 Neural network system and method for analyzing relationship network graph
CN109948447A (en) * 2019-02-21 2019-06-28 山东科技大学 The discovery of personage's cyberrelationship and evolution rendering method based on video image identification

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
何昊晨;张丹红;: "基于多维社交关系嵌入的深层图神经网络推荐方法" *

Also Published As

Publication number Publication date
CN114169515B (en) 2023-04-28

Similar Documents

Publication Publication Date Title
CN110399518B (en) Visual question-answer enhancement method based on graph convolution
CN112241481B (en) Cross-modal news event classification method and system based on graph neural network
CN110619081B (en) News pushing method based on interactive graph neural network
CN111753024B (en) Multi-source heterogeneous data entity alignment method oriented to public safety field
CN112784964A (en) Image classification method based on bridging knowledge distillation convolution neural network
Chen et al. Visual and textual sentiment analysis using deep fusion convolutional neural networks
CN113065974B (en) Link prediction method based on dynamic network representation learning
CN112016601B (en) Network model construction method based on knowledge graph enhanced small sample visual classification
CN114419642A (en) Method, device and system for extracting key value pair information in document image
CN113761250A (en) Model training method, merchant classification method and device
CN112685609A (en) Knowledge graph complementing method combining translation mechanism and convolutional neural network
CN111324773A (en) Background music construction method and device, electronic equipment and storage medium
CN115064020A (en) Intelligent teaching method, system and storage medium based on digital twin technology
CN114742071B (en) Cross-language ideas object recognition analysis method based on graph neural network
CN111353583B (en) Deep learning network based on group convolution characteristic topological space and training method thereof
CN113641797A (en) Data processing method, device, equipment, storage medium and computer program product
Shukla et al. Role of hybrid optimization in improving performance of sentiment classification system
CN116010813A (en) Community detection method based on influence degree of fusion label nodes of graph neural network
CN114417063A (en) Multi-view-based important node identification method for graph neural network
CN116955846A (en) Cascade information propagation prediction method integrating theme characteristics and cross attention
CN114169515B (en) Social relationship identification method based on high-order graph neural network
CN115858943A (en) Social relationship identification method based on Transformer
CN113744023B (en) Double-channel collaborative filtering recommendation method based on graph convolution network
CN113762372B (en) Method and device for identifying organization members in instant messaging information
CN113159007B (en) Gait emotion recognition method based on adaptive graph convolution

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant