CN117370578A - Method for supplementing food safety knowledge graph based on multi-mode information - Google Patents
Method for supplementing food safety knowledge graph based on multi-mode information Download PDFInfo
- Publication number
- CN117370578A CN117370578A CN202311392775.3A CN202311392775A CN117370578A CN 117370578 A CN117370578 A CN 117370578A CN 202311392775 A CN202311392775 A CN 202311392775A CN 117370578 A CN117370578 A CN 117370578A
- Authority
- CN
- China
- Prior art keywords
- knowledge graph
- information
- food safety
- entity
- modal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 230000001502 supplementing effect Effects 0.000 title claims abstract description 12
- 230000004927 fusion Effects 0.000 claims abstract description 12
- 238000013528 artificial neural network Methods 0.000 claims abstract description 5
- 238000013461 design Methods 0.000 claims abstract 3
- 230000000007 visual effect Effects 0.000 claims description 22
- 230000006870 function Effects 0.000 claims description 19
- 230000007246 mechanism Effects 0.000 claims description 17
- 238000010606 normalization Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 12
- 239000013598 vector Substances 0.000 claims description 12
- 238000012512 characterization method Methods 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 9
- 238000005516 engineering process Methods 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 230000011218 segmentation Effects 0.000 claims description 4
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 230000002776 aggregation Effects 0.000 claims description 2
- 238000004220 aggregation Methods 0.000 claims description 2
- 230000000295 complement effect Effects 0.000 claims description 2
- 238000013075 data extraction Methods 0.000 claims description 2
- 238000007689 inspection Methods 0.000 claims description 2
- 238000004458 analytical method Methods 0.000 claims 1
- 230000002457 bidirectional effect Effects 0.000 claims 1
- 230000007812 deficiency Effects 0.000 claims 1
- 238000002372 labelling Methods 0.000 claims 1
- 238000013507 mapping Methods 0.000 claims 1
- 238000007781 pre-processing Methods 0.000 claims 1
- 238000012163 sequencing technique Methods 0.000 claims 1
- 238000012549 training Methods 0.000 claims 1
- 230000009469 supplementation Effects 0.000 abstract description 2
- 239000011159 matrix material Substances 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 231100000279 safety data Toxicity 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 241000938605 Crocodylia Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
- G06F16/367—Ontology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Biomedical Technology (AREA)
- Databases & Information Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Animal Behavior & Ethology (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Machine Translation (AREA)
Abstract
The invention designs a method for supplementing a food safety knowledge graph based on multi-modal information, which aims at the problems that the food safety knowledge graph is small in scale and has multi-modal attribute information and structure information are missing, constructs the food safety multi-modal knowledge graph containing multi-modal information, expands the scale of the food safety knowledge graph, designs a multi-path multi-modal feature encoder based on attention so as to acquire multi-modal attribute embedded representation of an entity, and reserves global structure information of the knowledge graph by constructing an implicit relation in the framework supplementing knowledge graph based on a graph neural network so as to better capture relation structure information of the knowledge graph, realize fusion of semantic information and structure information, improve robustness of link prediction and realize high-quality food safety knowledge graph supplementation.
Description
Technical Field
The invention relates to the field of knowledge maps and graphic neural networks, in particular to a method for supplementing a food safety knowledge map based on multi-mode information.
Background
At present, food safety is one of public safety problems concerned by the public, and is not only closely related to the physical health and life safety of the national people, but also indispensible from the economic development and social safety of China. By means of rapid development of multimedia and network technologies, the food safety field gradually emerges massive multi-mode data including national food safety spot check monitoring data, food safety related social network data, food safety standard data sets and the like. These data cover different modalities of text, vision, etc., are semantically interrelated, covering the "safety problem" description that arises in each link of food production to consumption. How to fully mine the potential association relation between the multi-source heterogeneous food safety knowledge and effectively manage by utilizing the information technology has important significance for solving the food related problems in medicine, biology or agriculture.
In recent years, knowledge graphs are used as a knowledge base for storing graph structure data, so that knowledge in the real world can be stored in the form of triples of a head entity, a relation and a tail entity, and the efficiency of data relation is improved. In view of the wide application of the knowledge graph in other fields, more and more researchers in the food safety field are focusing on the application of the knowledge graph. However, most of research on food safety knowledge maps at home and abroad is focused on the construction process at present, but the knowledge map construction process can inevitably cause knowledge loss, omission and the like because the food safety data has the characteristics of multisource, fragmentation, invisible knowledge expression and the like. However, the accuracy and the knowledge coverage rate of the knowledge graph can restrict the upper layer application, and the manual complement and examination mode cannot adapt to the large-scale knowledge graph. Therefore, the implicit relation of food data needs to be mined according to the existing food knowledge in the knowledge graph and by combining the deep learning technology, and the coverage of the knowledge graph is automatically improved.
The graph neural network is a main paradigm for machine learning on graph data in recent years, has great advantages in processing data of graph structures, and has been widely applied to relation prediction and entity classification tasks in the field of knowledge graph completion. However, most of the current graph neural network models are focused on acquiring structural information of the knowledge graph, and ignoring directionality of the relationship in the knowledge graph; in addition, since the food safety data has the characteristics of multiple sources and isomerism, the information obtained based on the data of a single mode is very limited, and the traditional graph neural network model has insufficient extraction capability on the multi-mode food safety knowledge characteristics, so that the robustness of the relationship prediction is insufficient. Therefore, how to perform multidimensional knowledge representation based on a multi-mode food safety knowledge graph so as to improve the quality and coverage of the food safety knowledge graph is a problem to be solved urgently.
Disclosure of Invention
The invention mainly solves the technical problems that: aiming at the characteristics of multi-source isomerism of food safety data, a method for supplementing a food safety knowledge graph based on multi-mode information is provided, so that the problem that the traditional graph neural network model is insufficient in multi-mode semantic feature extraction capability in the food safety knowledge graph is solved, meanwhile, a mutual information mechanism is combined, processing of path structure information in the knowledge graph is realized, global structure features are better reserved, fusion of entity multi-mode semantic features and relationship structure features is realized, and therefore, a relationship prediction result with high robustness is obtained, and high-quality food safety knowledge graph supplementation is realized.
In order to solve the technical problems, the invention adopts a technical scheme that: constructing a food safety multi-modal knowledge graph, designing two bottom layer encoders to extract text and visual features respectively, and enabling the extracted text and visual features to enter a multi-modal encoding layer for multi-modal fusion so as to obtain multi-modal semantic features of an entity; meanwhile, the implicit reverse relation in the knowledge graph is complemented, a relation attention mechanism is introduced on the basis, an adjacency relation structure of the entity and a path structure of the knowledge graph are respectively captured, the multi-modal semantic features of the entity and the adjacency relation structure features are fused through convolution, normalization and other operations to obtain multi-modal representation of the entity, and the multi-modal representation is fused with the relation path representation to obtain a final candidate triplet representation, and the method specifically comprises the following steps:
(1) Acquiring food safety related information in food safety spot inspection result related notices and standard files by utilizing technologies such as crawlers and word segmentation, marking semi-structured data by manual assistance, constructing a body frame of a knowledge graph, formatting and cleaning the semi-structured data and unstructured data, acquiring knowledge graph triple data, and constructing a food safety knowledge graph based on Neo4 j;
(2) Designing two bottom layer encoders to extract text and visual characteristics, extracting text characteristics by using a RoBERTa encoder, extracting visual characteristics by using a Vision Transformer encoder, realizing normalization processing of different dimension characteristics by using dimension normalization representation, and constructing a multi-path multi-mode attention network for fusing information of different modes;
(3) The implicit reverse relation is complemented according to the existing explicit relation in the knowledge graph, the weight values of different relations to the entities are obtained by using a relation attention mechanism, the adjacent relation and path structure information of the entities are respectively aggregated, and meanwhile, the global structure information of the knowledge graph is reserved, so that the structure information contained in the knowledge graph is completely captured;
(4) Respectively carrying out convolution and normalization operations by utilizing the multi-modal entity semantic information obtained in the step (2) and the entity adjacency relation information obtained in the step (3) to obtain entity characteristics with consistent scales, and then fusing the two to obtain final multi-modal entity characteristics; and fusing the multi-modal entity characteristics and the path structure characteristics of the candidate triples to obtain the vector characterization of the triples, and carrying out link prediction on the candidate triples through a scoring function.
In the step (1), related information of food safety is firstly extracted by using technologies such as reptiles, word segmentation and the like, a body frame of a knowledge graph is constructed according to the extracted semi-structured data, a data extraction mode is perfected, formatting processing and data cleaning are carried out on the extracted data, and knowledge graph triple data are obtained, so that a food safety knowledge graph is constructed;
in the step (2), aiming at the text entity and relation embedding of the multi-mode knowledge graph, a RoBERTa encoder is adopted, the context information can be considered, the long-distance dependency relation in the text is captured, vision Transformer is adopted as a visual encoder, the visual information in the multi-mode knowledge graph is embedded, the extracted text and visual characteristics are subjected to dimension normalization to obtain normalized representation, and a multi-channel multi-mode attention network is constructed to fuse the multi-mode information;
in the step (3), the implicit reverse relation is complemented according to the directional characteristic of the relation in the knowledge graph, so that the adjacency relation (comprising the implicit reverse relation) taking the current entity as a starting point can be only captured, and the attention weights of all adjacency relations to the current entity are calculated by utilizing a relation attention mechanism so as to be convenient for aggregating the adjacency relations of the entity; negative sampling is carried out by randomly replacing a head entity or a tail entity of the triplet, path structure information from a current entity to a target entity is aggregated based on a relationship attention mechanism, global structure information of a knowledge graph is reserved, and learned structure characteristics are continuously optimized by utilizing a local-global mutual information maximization mechanism;
in the step (4), feature fusion is carried out according to knowledge characterization of different scales finally obtained in the previous two steps, so as to form a new feature vector which has multiple dimensions and contains different modal details of triples. The scoring function is designed to score candidate triples, the edge loss function is calculated by using the negative triples obtained by randomly replacing the head entity or the tail entity, and the edge loss function is combined with the loss function in the mutual information mechanism to optimize model parameters, so that a highly accurate relation prediction result is generated.
The beneficial effects of the invention are as follows:
aiming at the problems of insufficient multi-mode semantic feature extraction capability and relation structure information deletion of an entity in the process of completing the knowledge graph, the invention utilizes a multi-mode semantic information encoder and a relation attention mechanism to realize the fusion of text and visual features, better reserves the relation structure information in the knowledge graph, extracts global and local structure information and realizes the fusion of the multi-mode semantic features and the structure features, thereby obtaining a high-robustness knowledge graph completion result.
Drawings
FIG. 1 is a schematic diagram of a food safety knowledge graph completion process based on multi-modal information in the invention;
FIG. 2 is a schematic diagram of a multi-modal semantic and structural information encoding module implementation in the present invention.
Detailed Description
The invention is described below with reference to the drawings and the detailed description. Wherein, figure 1 depicts a schematic diagram of a food safety knowledge graph completion process based on multi-modal information; FIG. 2 depicts a schematic diagram of a multi-modal semantic and structural information encoder implementation.
As shown in fig. 1, the steps of food safety knowledge graph completion based on multi-mode information in the invention are as follows:
(1) According to the method for supplementing the food safety knowledge graph by utilizing the multi-mode information, firstly, the information of the food safety national standard is obtained from a network through technologies such as crawlers and word segmentation, the food safety national standard file is downloaded, the downloaded food safety national standard file is identified, the table information in the file is analyzed to obtain semi-structured data and unstructured data, the semi-structured data is manually marked in an auxiliary mode, a body framework of the knowledge graph is constructed, formatting processing and data cleaning are carried out on the data, and the knowledge graph triplet data is obtained and stored in a Neo4j database.
(2) For text features in the multimodal knowledge graph, the invention adopts RoBERTa for text entity and relationship embedding due to its strong ability to text embedded representation. By L T The BERT coding layers are used as text coding layers, which contain L T A plurality of multi-head attention modules and a feedforward neural network module. The calculation method of the text initial input is as follows:
wherein,representing the initial state of the text encoding layer, X wd Word embedding, X, representing text se For sentence segment embedding, T pos Representing the position embedding. The output result of the text after passing through the Roberta model is represented as X T ＝[x 1 ，x 2 ，...，x i ，...，x n ]Wherein X is T As a feature of the current text, x i Is a representation of the ith character of the current text.
For visual features in the multi-modal knowledge-graph, embedding is performed by using a Vision Transformer encoder, a given solid image is unified into 256-256 images for input, and then the images are divided into u=256-256/p 2 The patches are pooled and projected asWherein p represents the side length of each patch, +.>Representing the embedding of the ith patch, d v Is the dimension of the visual vector. Connecting the patch embeddings of the images to obtain a visual sequence patch embedmentWhere m=u×o, o is the number of solid images, and the image representation calculation is as follows:
wherein,representing the corresponding position code embedding +_>Is the initial input state of the visual coding layer. Visual characteristic representation X of knowledge graph obtained after Vision Transformer V 。
Food safetyThe knowledge graph mainly faces the problems of heterogeneity and weak correlation among different modes, modeling is carried out on multi-mode features of an entity, information is integrated, after text and visual features are extracted through a text encoder and a visual encoder, normalization processing on the different dimensional features is achieved through dimension normalization representation, and X is obtained respectively t And X v Where the subscript t represents the text modality, v represents the visual modality, and { X } is normalized for the obtained multimodal t ，X v First, a generating function based on tensor ring is used to search multi-path multi-mode inquiry tensorAnd key tensor Where Ti represents the length of the sequence i.e { t, v }. Based on the tensor loop format of Q and K, the multi-modal attention (MMT) process can be expressed as the following equation:
in this process, first the Hadamard product is calculatedAnd->Similarity between->Thereby obtaining a multi-modal attention tensor +.>Then by +.>Averaging along the time dimension to obtain the relevant attention pooling matrix +.>Is made up of time dependencies within the modality. After obtaining the relevant attention pooling matrix, the values of the multi-modal attention are calculated as shown in the following formula:
wherein, atten i For modal sensing of a multi-path and multi-mode attention matrix, linear is linear operation, d i For feature dimensions, i ε { t, v }, j m E { t, v }, m= {1,2} and j m Not equal to i, calculateRefers to the mode->Is a product of (tensor contraction operation). Based on the multi-path multi-mode attention matrix, multi-path multi-mode semantic interaction information can be obtained, as shown in the following formula:
wherein i is { t, v }, X i Representing the representation of the information corresponding to the mode i, and the parameter gamma represents the contribution value of the original mode information. Based on the MMT mechanism, the lower-level multi-modal associations are delivered in a recursive manner to the more comprehensive, expressive, higher-level correlations. Specifically, the first MMT accepts an input { X ] t ，X v Multi-modal representation output set { Y } and computing multi-modal awareness t ，Y v Subsequently, the next MMT takes as input the output of the previous MMT and recursively performs a similar N MMT process as follows:
wherein i is [1, N-1].
(3) Because the association degrees of the adjacency relations of different types on the entities are different, a relation attention mechanism is introduced to acquire the importance of each adjacent node-edge to the target node, and the weight calculation process is as follows:
α i，j ＝softmax i，j (W[x i ||r ij ||x j ])
wherein alpha is i，j Representing node x i Adjacent node x of (a) j Sum edge r i，j W is a weight matrix, and capturing an adjacency relationship (comprising an implicit inverse relationship) taking a current entity as a starting point on the basis of calculating the weight, so as to complete aggregation of adjacency information of a target node; then collecting path structure information p between entity pairs based on a relationship attention mechanism, and simultaneously keeping global structure information of a knowledge graphAnd adding path structure information among entity pairs in all triples to obtain global structure information.
To better preserve global structural information in the knowledge graph, a mutual information maximization operation is performed next, and the process loss function is set as follows:
wherein N is positive sample number, M is negative sample number, X is initial random representation of the knowledge graph, A is adjacent matrix,the result obtained by carrying out random negative sampling on (X, A), D is a discriminator formed by bilinear functions, and E is a function for obtaining expected values.
(4) In order to fully utilize the multi-modal features of the entity in the food safety knowledge graph G, the multi-modal semantic features obtained in the step (2) and the features of the adjacent information structure obtained in the step (3) are respectively subjected to convolution and normalization operation to obtain feature vectors with the same scale, and then the feature vectors and the feature vectors are added and fused to obtain the final entity representation X. Characterization of head entity of candidate triples X u Characterization of tail entity X v The relation representation R and the local relation representation P obtained by aggregating all relation paths between the relation representation R and the local relation representation P uv Splicing to obtain multi-mode characterization vector T of the candidate triplet urv And constructing an edge loss function of the relation prediction through a scoring function f:
wherein, (u, r, v) is a positive sample in the knowledge graph, (u ', r ', v ') is a corresponding negative sample, gamma is an edge hyper-parameter, f is a scoring function of the candidate triplet (u, r, v), and the ConvE model is used as the scoring function for calculation, namely:
f(u，r，v)＝ConvE(T urv )
finally constructing the joint loss function of the steps (3) and (4):
L＝L sup +αL MI
and alpha is used for controlling the contribution value of a mutual information mechanism, and the parameters of the whole model are optimized through a joint loss function so as to generate a relation prediction result with high accuracy and realize high-quality knowledge graph completion.
It will be readily appreciated by those skilled in the art that the foregoing description is merely a preferred embodiment of the invention and is not intended to limit the invention, but any modifications, equivalents, improvements or alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (5)
1. The method for supplementing the food safety knowledge graph based on the multi-mode information is characterized by comprising the following steps of:
step 1: aiming at the information deficiency in the food safety knowledge graph, extracting the food safety national standard information, the selective examination bulletin and the food safety related pictures disclosed in the network, automatically marking the extracted structured data, and constructing a body frame of the food safety knowledge graph; manually assisted formatting and data cleaning are carried out on the semi-structured data and the unstructured data to obtain knowledge-graph triple data, and a food safety knowledge graph containing multi-mode information is constructed based on Neo4 j;
step 2: for a multi-mode semantic coding module of an entity, designing two bottom layer feature encoders to extract text and visual features respectively, and for the text features, embedding the text entity and the relation by adopting a RoBERTa encoder; for visual features, embedding with a Vision Transformer encoder; for the obtained different modal characteristics, carrying out normalization processing on the different dimensional characteristics by utilizing dimension normalization representation so as to facilitate deep fusion operation on the different modal characteristics, constructing a multi-path multi-modal attention network, and fusing different modal information to obtain a text and visual characteristic fusion vector of a food safety knowledge graph;
step 3: for the structural information coding module of the knowledge graph, firstly, the information of the reverse relation implicit in the knowledge graph is utilized to complement the information of the bidirectional path in the food safety knowledge graph; on the basis, capturing an adjacency relation structure of the entity and a path structure between entity pairs, and acquiring the importance of each adjacent node-edge to a target node by using a relation attention mechanism to obtain an adjacent feature vector of the entity in the food safety knowledge graph; based on the negative sampling operation of the triples, a self-supervision global-local information maximization mechanism is realized to acquire multi-dimensional structural information, fusion of local structural features and global structural features is realized, and the relation and entity associated information is highlighted;
step 4: the text and visual feature fusion vector obtained in the step 2 and the adjacent feature vector of the entity obtained in the step 3 are used for respectively carrying out convolution and normalization operations, so that heterogeneity among different feature types is eliminated, and the scale of the features is unified; then fusing different features to form a new multi-mode and multi-dimensional food safety knowledge graph representation, simultaneously containing semantic information of fine granularity of food safety entities and multi-scale relation structure information, and finally sequencing the features of the candidate triples by using a constructed scoring function; on the basis, the relationship prediction is carried out on the design scoring function so as to screen out correct entity relationship triples, thereby completing the knowledge graph.
2. The method for supplementing food safety knowledge graph based on multi-modal information according to claim 1, wherein the method comprises the following steps: in the step 1, a food safety knowledge graph is constructed, and the specific construction method is as follows:
(1) Acquiring food safety related knowledge by utilizing a web crawler and word segmentation technology for related bulletins and standard files of food safety sampling inspection results disclosed in a network;
(2) Based on rule matching and webpage and form analysis, semi-structured data and unstructured data are obtained, manual auxiliary labeling is carried out on the semi-structured data, a body frame of a food safety knowledge graph is constructed, and then a data extraction mode is perfected;
(3) And carrying out formatting treatment and data cleaning on the semi-structured data and the unstructured data according to the constructed body frame and the storage requirement of the knowledge graph to obtain food safety knowledge graph triple data, and storing the knowledge graph triple data into a Neo4j graph database.
3. The method for supplementing food safety knowledge graph based on multi-modal information according to claim 1, wherein the method comprises the following steps: in the step 2, an entity multi-mode semantic coding module is constructed, and the specific construction method is as follows:
(1) In order to embed text entities and relations of the multimodal knowledge graph into a high-dimensional space for computation, a RoBERTa encoder is used to embed the text entities and relations, a dynamic mask is used in the training process, a new mask pattern is generated each time a sequence is input to the model, and character-level and word-level characterizations are mixed in the text encoding part;
(2) Embedding visual information in the multi-mode knowledge graph by using a Vision Transformer encoder, dividing an image into a series of small rectangular areas patch with fixed sizes, and then mapping the rectangular areas patch into the input of the Vision Transformer encoder to capture long-distance dependency relationship among the patches;
(3) And processing the extracted text and visual features by adopting dimension normalization to realize normalization processing of the features with different dimensions, and obtaining normalized characterization so as to facilitate deep fusion operation of the features with different modes, and constructing a multi-path multi-mode attention network to fuse multi-mode information.
4. The method for supplementing food safety knowledge graph based on multi-modal information according to claim 1, wherein the method comprises the following steps: in the step 3, a map structure information coding module is constructed, and the specific construction method is as follows:
(1) The implicit reverse relation is complemented according to the existing explicit relation information in the knowledge graph; and completing the negative sample sampling process by randomly replacing entities or relations in the existing triples; encoding different entities by using numbers to finish preprocessing of knowledge graph data;
(2) The method comprises the steps of parameterizing edges by utilizing the idea of spatial convolution in a graph convolution neural network GCN, updating the hidden state of nodes by utilizing the characterization of fused adjacent node-edges, and acquiring the importance of each adjacent node-edge to a target entity node by utilizing a relationship attention mechanism so as to realize aggregation of adjacent structure information of the target entity node;
(3) And meanwhile, extracting path structure information between target entity pairs, acquiring relationship weights by using a relationship attention mechanism, aggregating relationship paths as local structure information, fusing all the local structure information to obtain global structure information, and learning the structure information by using a mutual information maximization mechanism.
5. The method for supplementing food safety knowledge graph based on multi-modal information according to claim 1, wherein the method comprises the following steps: in the step 4, a scoring function aiming at the triplet multi-mode characterization is constructed, and the specific construction method is as follows:
(1) Aiming at multi-modal semantic features of an entity and adjacent structural features of the entity, utilizing a convolution layer and a normalization layer to eliminate heterogeneity of different modal features;
(2) And adding the normalized multi-modal semantic features and the adjacent structural feature vectors to obtain final multi-modal features of the entity, realizing the fusion of the semantic features and the structural features of the entity, splicing the multi-modal features of the head entity and the tail entity in the candidate triplet with the path structural features, and inputting the multi-modal features into a scoring function for prediction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311392775.3A CN117370578A (en) | 2023-10-25 | 2023-10-25 | Method for supplementing food safety knowledge graph based on multi-mode information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311392775.3A CN117370578A (en) | 2023-10-25 | 2023-10-25 | Method for supplementing food safety knowledge graph based on multi-mode information |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117370578A true CN117370578A (en) | 2024-01-09 |
Family
ID=89394258
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311392775.3A Pending CN117370578A (en) | 2023-10-25 | 2023-10-25 | Method for supplementing food safety knowledge graph based on multi-mode information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117370578A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118171726A (en) * | 2024-05-15 | 2024-06-11 | 江西博微新技术有限公司 | Method, system, storage medium and computer for constructing project whole process knowledge graph |
| CN118212398A (en) * | 2024-03-19 | 2024-06-18 | 中国矿业大学 | Multi-mode target detection method based on knowledge complement balance network |
-
2023
- 2023-10-25 CN CN202311392775.3A patent/CN117370578A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118212398A (en) * | 2024-03-19 | 2024-06-18 | 中国矿业大学 | Multi-mode target detection method based on knowledge complement balance network |
| CN118171726A (en) * | 2024-05-15 | 2024-06-11 | 江西博微新技术有限公司 | Method, system, storage medium and computer for constructing project whole process knowledge graph |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107766555B (en) | Image retrieval method based on soft-constraint unsupervised cross-modal hashing | |
| CN111488734A (en) | Emotional feature representation learning system and method based on global interaction and syntactic dependency | |
| CN110990590A (en) | Dynamic financial knowledge map construction method based on reinforcement learning and transfer learning | |
| CN110598005A (en) | Public safety event-oriented multi-source heterogeneous data knowledge graph construction method | |
| CN117370578A (en) | Method for supplementing food safety knowledge graph based on multi-mode information | |
| CN112199532B (en) | Zero sample image retrieval method and device based on Hash coding and graph attention machine mechanism | |
| Lin et al. | Deep structured scene parsing by learning with image descriptions | |
| CN107346327A (en) | The zero sample Hash picture retrieval method based on supervision transfer | |
| CN114419304A (en) | Multi-modal document information extraction method based on graph neural network | |
| Zhang et al. | A multi-feature fusion model for Chinese relation extraction with entity sense | |
| CN111582506A (en) | Multi-label learning method based on global and local label relation | |
| CN116092577B (en) | Protein function prediction method based on multisource heterogeneous information aggregation | |
| CN114021584A (en) | Knowledge representation learning method based on graph convolution network and translation model | |
| CN113868448A (en) | Fine-grained scene level sketch-based image retrieval method and system | |
| Xiong et al. | Oracle bone inscriptions information processing based on multi-modal knowledge graph | |
| CN113988075A (en) | Network security field text data entity relation extraction method based on multi-task learning | |
| Wang et al. | KTN: Knowledge transfer network for learning multiperson 2D-3D correspondences | |
| CN116737979A (en) | Context-guided multi-modal-associated image text retrieval method and system | |
| Wang et al. | R2-trans: Fine-grained visual categorization with redundancy reduction | |
| Al-Tameemi et al. | Multi-model fusion framework using deep learning for visual-textual sentiment classification | |
| Zhu et al. | Dynamic Hierarchical Markov Random Fields for Integrated Web Data Extraction. | |
| Zhang et al. | End‐to‐end generation of structural topology for complex architectural layouts with graph neural networks | |
| CN111259176B (en) | Cross-modal Hash retrieval method based on matrix decomposition and integrated with supervision information | |
| Li et al. | [Retracted] Deep‐Learning‐Based 3D Reconstruction: A Review and Applications | |
| Sengar et al. | Generative Artificial Intelligence: A Systematic Review and Applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |