CN116383354A - Automatic visual question-answering method based on knowledge graph - Google Patents
Automatic visual question-answering method based on knowledge graph Download PDFInfo
- Publication number
- CN116383354A CN116383354A CN202310277208.7A CN202310277208A CN116383354A CN 116383354 A CN116383354 A CN 116383354A CN 202310277208 A CN202310277208 A CN 202310277208A CN 116383354 A CN116383354 A CN 116383354A
- Authority
- CN
- China
- Prior art keywords
- graph
- entity
- query
- knowledge
- entities
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 33
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000012800 visualization Methods 0.000 claims abstract description 41
- 230000002787 reinforcement Effects 0.000 claims abstract description 11
- 230000002776 aggregation Effects 0.000 claims description 15
- 238000004220 aggregation Methods 0.000 claims description 15
- 238000010586 diagram Methods 0.000 claims description 9
- 239000013598 vector Substances 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 5
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 claims description 4
- 230000006870 function Effects 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 2
- JXSJBGJIGXNWCI-UHFFFAOYSA-N diethyl 2-[(dimethoxyphosphorothioyl)thio]succinate Chemical compound CCOC(=O)CC(SP(=S)(OC)OC)C(=O)OCC JXSJBGJIGXNWCI-UHFFFAOYSA-N 0.000 claims description 2
- 230000007246 mechanism Effects 0.000 claims description 2
- 238000012549 training Methods 0.000 claims description 2
- 241000287196 Asthenes Species 0.000 claims 1
- 238000004364 calculation method Methods 0.000 claims 1
- 238000011160 research Methods 0.000 abstract description 6
- 238000011161 development Methods 0.000 abstract description 2
- 241001235534 Graphis <ascomycete fungus> Species 0.000 description 2
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000006116 polymerization reaction Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G06F40/295—Named entity recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a graph visual automatic question-answering method based on a knowledge graph, which is characterized in that a knowledge base expansion module is adopted to expand semantic and graph attributes of the knowledge graph, shallow semantic analysis and named entity recognition are utilized to extract topic entities in questions, a knowledge path from topic entities to answer entities is output based on an automatic question-answering model of reinforcement learning and BERT, and text answers and visual answers are output according to different answer types. Compared with the prior art, the method solves the problem of automatic question and answer of a complex visual type of graph visualization, constructs a knowledge graph through the data characteristics of the graph visualization, improves the robustness and generalization capability of question and answer by using a method based on reinforcement learning and BERT, and automatically questions and answers a network graph visual graph in various practical application scenes such as literature character relation research, intelligent education, commodity cargo analysis, scientific research hot spot analysis and the like, thereby having higher practical value and good development prospect.
Description
Technical Field
The invention relates to the technical field of graph visualization knowledge graph in natural language processing, in particular to a graph visualization automatic question-answering method based on knowledge graph.
Background
When a user is faced with decision tasks, graph visualization is a common type of visualization that analyzes topology data and answers questions. However, this is not an easy task when the user is faced with many complex analysis problems with respect to the visualization of the graph. Firstly, a user needs to understand the problem before observing the chart, and when the user observes the chart for visualization, the user needs to understand the steps of legend and text information, perceiving graphic data, analyzing communities, calculating attributes of various chart data and the like according to different problems.
Currently, some natural language systems have been used for automatic chart questions and answers to help users analyze charts and answer questions faster. However, most existing work has focused on simple visual charts, such as scatter, line and bar charts. These charts are easily converted into form data and are suitable for use in a form question-answering system. Graph visualization is typically used to represent topological relationships between nodes, such as character relationships in novels. People typically analyze data and answer questions through charts. In practical applications, the problem posed by the user to the graph visualization involves not only the topological relation of nodes, but also the properties specific to the graph visualization, such as node degree, graph community, etc. Because the graphic visualization contains more attributes than the simple visualization and the graphic data is not a two-dimensional structure, it is difficult to convert the graphic visualization into the form data to apply the existing automatic question-answering method of the graph. In addition, most existing automated question-answering systems answer using text information, which is tedious and departs from the original purpose of visualizing the data. The visual answer is matched with the text answer, so that the answer is more useful and has more convincing and interpretability.
Disclosure of Invention
The invention aims to provide a graph visualization automatic question-answering method based on a knowledge graph aiming at the defects of the prior art, which adopts the knowledge graph to store the visualized information data of the graph, expands the semantic and graph attribute of the knowledge graph through a knowledge base expansion module, extracts topic entities in questions through shallow semantic analysis and named entity recognition, outputs a knowledge path from the topic entities to answer entities through an automatic question-answering model based on reinforcement learning and BERT, finally outputs text answers and visual answers according to different answer types, effectively solves the problem of automatic question-answering of the complex visualization type of the graph visualization, constructs the knowledge graph through the data characteristics of the graph visualization, improves the robustness and generalization capability of the question-answering through a method based on reinforcement learning and BERT, and can automatically question-answer a network graph in various practical application scenes such as literature relationship research, intelligent education, commodity cargo analysis, scientific research hot spot analysis and the like.
The specific technical scheme for realizing the aim of the invention is as follows: a visual automatic question-answering method based on knowledge graph is characterized by adopting a framework GVGA (graphic visual question-answering) to answer questions related to visual graph and automatically generate visual answers, firstly converting visual graph into standard GML format, then extracting graphic data and visual attributes from GML, designing the data structure of graph according to the characteristics of graph and converting the graph data structure into knowledge graph, designing an expansion module for the knowledge base for expanding semantic information and graph analysis data, adopting shallow semantic analysis and named entity recognition to extract topic entities in questions, using an automatic question-answering model based on reinforcement learning and BERT to output knowledge paths from topic entities to answer entities, and finally outputting text answers and visual answers according to different answer types, and the specific process of the invention comprises the following steps:
a step: a graph is input to visualize G and problem Q, and G is converted into a GML standard format.
b, step b: converting native data in a graph visualization into entities and native triples for storage in oneKnowledge graph
Wherein h represents a head entity, R represents a relation, t represents a tail entity, E represents an entity set, R represents a relation set, and the original data comprises node information, side information, weight and the like originally contained in the visualization of the graph.
c, step c: will be
Carrying out knowledge graph expansion, and storing the derived triples obtained after expansion into +.>
The knowledge graph extension comprises semantic information extension and graph attribute extension, the derived triples are obtained by calculating data of the original triples or other derived triples, and the extended knowledge graph is constructed and comprises graph topology information, graph attribute information and graph semantic information of graph visualization.
The graph topology information is that each node and each edge in a network graph are taken as entities, the connection relation of the edges and the neighbor relation of the nodes are taken as relation predicates, a plurality of triples are constructed, meanwhile, the names of the entities are taken as metadata triples to be added into an annotation graph, one graph entity is introduced to represent the whole graph, and the edges and the nodes are added from the triples of the graph; the graph attribute information is an analysis index common in network graph analysis and comprises the degree (in degree and out degree) of a node, the degree center degree of the node, the aggregation degree of the node, the weight of edges, community information and the like; after detecting communities of the graph by using a Louvain algorithm, building an entity for each community, adding the average degree, the average side weight, the average center aggregation degree and other attributes of each community, and adding the information of the community belonging to the graph, the information of nodes and sides belonging to the community into a knowledge graph; the graph semantic information is obtained by using predicate aliasing mechanism to add information in triplesr substitution with r with semantic information s The method is added into the knowledge graph again, the single relation can be repeatedly added for a plurality of times, and the relation capable of having predicate aliases comprises the following steps: the node degree, the node degree center degree, the node aggregation degree, the edge weight, the community attribution, the community average weight, the community average degree, the community average aggregation degree, the edge connection relationship, the node neighbor relationship and the like.
d, step d: extracting topic entities in the input problem Q in the step a through a topic entity extraction module, and outputting candidate sets of topic entities
Includes a set {<t 1 ，c 1 >，<t 2 ，c 2 >,..}, wherein t i Is the entity name, c i Is a credibility score, and the credibility value interval is [0,1 ]]The higher the score, the more reliable the entity's predicted outcome.
The topic entity extraction module comprises named entity identification, multi-word entity identification, edge entity matching and graph entity addition, and is specifically as follows:
1) Named entity identification: tokenizing the input problem, sequentially carrying out named entity identification on each token by using a NER model in Flair, obtaining POS information of the sentence by utilizing a sequencer Tagger, and obtaining the POS information of the sentence in the following way
Word elements which are matched with POS labels as nouns in all entity names; if in the knowledge graph->
If there is a corresponding match, it is added to the knowledge-graph +.>
In (1) to obtain
Wherein->
The fraction output for the NER model.
2) Multi-lemma entity identification: sequentially matching entities with names containing spaces, namely entities corresponding to a plurality of word elements by using an N-Gram method, wherein the value of N is 1 to 4, sequentially matching, and if the matching is successful, then<t NGram ，c＝max(c n )>(c n E t) to candidate set
Is a kind of medium.
3) Edge entity identification: if a plurality of node entities are identified, detecting whether the word element between the two entities is a conjunctive word, and if so, detecting whether the two entities contain a connection relation in the diagram; if so, add the entity of the edge to the candidate set
In which, let the confidence scores c of two entities be c respectively a And c b Then add entity<t edge ，c＝max(c a ，c b )>。
4) Graph entity adds: considering that all the problems are related to the graph, if the above steps are carried out
Then the graph entity is to be rendered<t graph ，c＝1>Add to candidate set->
In (a) and (b); if at this time->
Will be<t graph ，c＝0.4>Add to candidate set->
Is a kind of medium.
And e, step e: as a candidate set
Constructing a candidate query graph, wherein the query graph comprises a core relation path from the topic entity to the answer entity, and the query graph is defined as QG= { N, E }, wherein N is a node set, and four types of nodes are all arranged: n is n i ∈{n g ，n ug ，n ag ，n an }，n g Is a grounding node and represents an entity existing in the knowledge graph; n is n ug Is a non-grounded node, can be used to represent multiple entities or an intermediate query result, n ag Is an aggregation node, which can be used to perform an aggregation operation, n an Answer nodes represent query answers; e represents the edge set { E } 1 ，e 2 ,..}, wherein e i Is the relation r in the triplet of the knowledge graph.
f, step f: iterative expansion inquiry diagram, order
For the set of query graphs at generation t, t=0,/for the selection of round t>
In each selection, all +.>
Attempting to attach a feasible relationship or a keyword-based aggregation operation to the end of the query graph; the possible relation is +.>
The relationship exists in the query graph, and the entity of the relationship is the tail node of the query graph; if only one topic entity in the current G is provided, the tail is designated as n after the feasible relation is attached an The method comprises the steps of carrying out a first treatment on the surface of the If it already exists, the original n an Change to n ug Will be new n after the attachment an As a new tail point.
g, the step: and (3) calculating the feature vector of the query graph, adopting a candidate query graph sorting model based on reinforcement learning, repeating the step f for searching and reserving 3 query graphs with the highest score by using the alternate beams of each round until the query graph cannot be expanded, and finally outputting the query graph with the highest score as an optimal query graph.
The reinforcement learning-based candidate query graph ranking model generates a 6-dimensional feature vector for each candidate query graph
The method specifically comprises the following steps:
1) BERT-based semantic matching: using a standard BERT model to scale the tokenized problem sequence s q And query graph sequence s g Semantic similarity between the two. s is(s) g The sequence [ CLS ] is generated by sequentially concatenating the ground entity name and the relationship name along the core relationship path]s q [SEP]s g Providing the BERT models to calculate their semantic similarity;
2) Entity confidence: a cumulative confidence score c for all topic entities;
3) Number of entities: n in query graph g Number of pieces;
4) Number of entity types: number of entity types;
5) Answer entity number: n in query graph an Is the number of (3);
6) Polymerization amount: n in query graph ag Is a number of (3).
Feeding the feature vector v of each candidate query graph Q into the fully connected layer of the enhancement model to obtain p (q|q); the training target is to learn a strategy function p θ (q|q) where θ represents a parameter in the model, using the F1 score between the predicted answer and the correct answer label as a reward.
And h, step h: generating SPARQL statement S from head node to tail node according to the optimal query graph, and executing the query statement on the knowledge graph to obtain text answer A t ，A t The method can be divided into literal answers, entity answers and statistical answers; in the visualization of highlights according to different answer typesAnd drawing an auxiliary information card comprising a statistical bar chart, a related community, related nodes and related edges to generate a visual answer A v 。
i, step: outputting text answer A t With visual answer A v And completing the visual automatic question and answer of the graph.
Compared with the prior art, the method solves the problem of automatic question and answer of the complex visualization type of graph visualization, constructs a knowledge graph through the data characteristics of the graph visualization, and improves the robustness and generalization capability of question and answer by using a method based on reinforcement learning and BERT. The method can automatically question and answer the visual chart of the network chart in various practical application scenes such as literature person relation research, intelligent education, commodity cargo analysis, scientific research hot spot analysis and the like, and has higher practical value and good development prospect.
Drawings
FIG. 1 is a schematic flow chart of the present invention;
FIG. 2 is a schematic view of the diagram of example 1;
FIG. 3 is a schematic diagram of the graph visualization transition GML of embodiment 1;
FIG. 4 is a schematic diagram of the knowledge graph constructed in example 1;
fig. 5 is a topic entity candidate set extraction schematic diagram of embodiment 1.
Detailed Description
Referring to fig. 1, the data extraction and automatic question answering of the network visual map according to the present invention comprises the following steps:
step one: inputting a graph to visualize G and a problem Q;
step two: converting G into GML standard format, and constructing knowledge graph
Step three: generating derived triples and expanding knowledge graph
Step four: extracting topic entities from Q to obtain topic entity candidate sets
Step five: will be
And->
Inputting automatic question-answering module based on BERT and reinforcement learning to output text answer A t ；
Step six: generating visual answer A s And outputting the visual automatic question and answer with the text answers to complete the visual automatic question and answer of the graph.
The invention will be described in further detail below with the example of questions and answers visualized on a novel character relationship diagram.
Example 1
Referring to fig. 2, step 1: a graphic visualization G of a novel character is input with a set of questions Q.
Referring to fig. 3, step 2: converting the visual G of the graph into a GML standard format, and constructing a knowledge graph
Referring to FIG. 4, a native triplet including graph topology data is added to a knowledge graph
I.e., b in fig. 4), and adds the entity and subordinate triples of the overall graph (i.e., e in fig. 4).
Step 3: adding the derived triples after semantic expansion (i.e., a and d in FIG. 4) and graph attribute analysis expansion (i.e., e in FIG. 4) to the knowledge graph
Is a kind of medium.
Referring to fig. 5, step 4: extracting topic entity candidate sets for each question in input Q
Like question "What is the weight of the edge between Myriel and Napoleon? The topic entity candidate set of' is {<"m.node@myriel":1>,<"m.node@napoleo":1>,<"m.edge@myranap":1>,<"m.grap@2y96ai":0.4>}。
Step 5: inputting the constructed knowledge graph and topic entity candidate set into an automatic question-answering module, and outputting an optimal query graph and a corresponding SPARQL statement SELECT? e1 WHERE { m.edge@myranap edge.property.weight? e1, obtaining the text answer '1' according to the query statement.
Step 6: the literal answer "1" is a literal answer and the associated graph visualization element is the side of the junction Myriel that is connected to Napolean, so the visual answer generation module will highlight the side of the graph visualization to which the two junctions are connected and output the information card of their associated attributes as auxiliary information.
Step 7: and outputting the literal answer and the visual answer together to complete the visual automatic question and answer of the graph.
The invention is further described above without limiting the scope of the patent, which is defined by the appended claims and equivalents thereof without departing from the spirit and scope of the inventive concept.
Claims (5)
1. The automatic graph visualization question-answering method based on the knowledge graph is characterized by converting graph visualization into a standard GML format, extracting graph data and visual attributes from the GML, designing a data structure of the graph according to the characteristics of the graph, converting the graph data into the knowledge graph to perform automatic question-answering of the graph visualization, and specifically comprises the following steps of:
a step: inputting a graph to visualize G and a problem Q, and converting G into a GML standard format;
b, step b: converting the original data in the visualization of the graph into an entity and an original triplet to store in a knowledge graph
Wherein h represents a head entity, R represents a relation, t represents a tail entity, E represents an entity set, R represents a relation set, and the original data comprises node information, side information and weight originally contained in the visualization of the graph;
c, step c: for knowledge graph
Expanding, and storing the derived triples into a knowledge graph ++>
The knowledge graph extension comprises semantic information extension and graph attribute extension, and the derived triples are obtained by data calculation of original triples or other derived triples;
d, step d: extracting topic entities in the input problem Q in the step a through a topic entity extraction module, and outputting candidate sets of topic entities
Includes a set {<t 1 ,c 1 >,<t 2 ,c 2 >,..}, wherein t i Is the entity name, c i Is a credibility score, and the value interval is 0,1]The higher the score, the more reliable the entity's predicted outcome;
and e, step e: as a candidate set
Constructing a candidate query graph, wherein the query graph comprises a core relation path from the topic entity to the answer entity, and the query graph is defined as QG= { N, E }, wherein N is a node set, and four types of nodes are all arranged: n is n i ∈{n g ，n ug ，n ag ，n an }，n g A grounding node for representing an entity existing in the knowledge graph; n is n ug A non-grounded node representing a plurality of entities or an intermediate query result; n is n ag Aggregation nodes for performing aggregation operations; n is n an Answer nodes representing answers to the query; for edge set { e, e 2 ,..}, wherein e i Is the relationship in the knowledge graph triplet;
f, step f: iteratively expanding a query graph to enable parameters
For the set of query graphs at the first generation, t=0,/for the first generation>
In each selection, all +.>
Attempting to attach a feasible relationship or a keyword-based aggregation operation to the end of the query graph; the feasible relationship is a knowledge graph +.>
The relationship exists in the query graph, and the entity of the relationship is the tail node of the query graph; if only one topic entity exists in the current visualization G, designating the tail as n after attaching the feasible relationship an The method comprises the steps of carrying out a first treatment on the surface of the If it already exists, the original n an Change to n ug Will be new n after the attachment an As a new tail point；
g, the step: calculating feature vectors of the query graphs, sorting the query graphs by adopting a candidate query graph sorting model based on reinforcement learning, and repeating the step f for searching and reserving 3 query graphs with highest scores by using the alternate beams of each round until the query graphs cannot be expanded, and finally outputting the query graph with the highest scores as an optimal query graph;
and h, step h: generating SPARQL statement S from head node to tail node according to the optimal query graph, and executing the query statement on the knowledge graph to obtain text answer A t The text answer A t For literal answers, entity answers and statistical answers, according to different answer types, relevant elements in the visualization of the graph are highlighted, and auxiliary information cards comprising statistical bar graphs, relevant communities, relevant nodes and relevant edges are drawn to generate visual answer A v ；
i, step: outputting text answer A t With visual answer A v And completing the visual automatic question and answer of the graph.
2. The knowledge-graph-based automatic question-answering method according to claim 1, wherein the graph visualization includes: d3, ECharts, matplotlib, scipy open source visualization framework drawn vector diagrams or diversified network diagrams of bitmaps.
3. The knowledge-based graph visualization automatic question-answering method according to claim 1, wherein the knowledge-graph extension includes graph topology information, graph attribute information and graph semantic information of the graph visualization; the graph topology information is that each node and each edge in a network graph are taken as entities, the connection relation of the edges and the neighbor relation of the nodes are taken as relation predicates, a plurality of triples are constructed, meanwhile, the names of the entities are taken as metadata triples to be added into an annotation graph, one graph entity is introduced to represent the whole graph, and the edges and the nodes are added from the triples of the graph; the map attribute information includes: the degree of the nodes, the degree center of the nodes, the aggregation degree of the nodes, the weight of the edges and community information; after the community information uses the Louvain algorithm to detect communities of the graph, the community information is used for each communityThe group builds an entity, and adds average degree, average side weight and average center aggregation degree attribute of each community; the graph semantic information uses predicate aliasing mechanism to replace r in the triplet of added information with r with semantic information s Adding the knowledge graph again, and repeatedly adding a single relation for a plurality of times; the relationships of predicate aliases include: the node degree, the node degree center degree, the node aggregation degree, the edge weight, the community attribution, the community average weight, the community average degree, the community average aggregation degree, the edge connection relationship and the node neighbor relationship.
4. The knowledge-graph-based visual automatic question-answering method according to claim 1, wherein the topic entity extraction module comprises named entity recognition, multi-word entity recognition, edge entity matching and graph entity addition, the named entity recognition tokenizes an input question, uses an NER model in Flair to sequentially recognize a named entity for each word element, and uses a sequencer tagger to obtain POS information of sentences in the knowledge graph
Word elements which are matched with POS labels as nouns in all entity names; if in the knowledge graph->
If there is a corresponding match, it is added to the candidate set +.>
In (1) get->
Wherein t is NER Entity output for NER model +.>
The score output for the NER model;
the multi-epoch entity recognition usesThe N-Gram method sequentially matches the entities with names containing spaces, namely corresponding to a plurality of word elements, takes n=1-4 to sequentially match, and if the matching is successful, the method comprises the steps of<t NGram ,c＝max(c n )>(c n E t) to candidate set
Wherein t is NGram An entity identified for N-Gram; c n Is a confidence score;
the edge entity identification is to identify a plurality of node entities, detect whether the word element between the two entities is a conjunctive word, if so, detect whether the two entities contain a connection relationship in the graph; if so, add the entity of the edge to the candidate set
In (a) and (b); let the confidence scores c of two entities be c respectively a And c b Then add graph entity<t edge ,c＝max(c a ,c b )>；
The addition of the graph entities is that the graph entities are identified by named entities and edge entities, such as
Then the graph entity is to be rendered<t graph ,c＝1>Add to candidate set->
In (a) and (b); for example->
Will be<t graph ,c＝0.4>Add to candidate set->
In (1)/(2)>
Is represented as emptyParameters of the set.
5. The knowledge-based graph visualization automatic question-answering method according to claim 1, wherein the reinforcement learning-based candidate query graph ranking model generates a 6-dimensional feature vector for each candidate query graph q
The method specifically comprises the following steps: based on the semantic matching, entity confidence, entity type number, answer entity number and aggregation number of BERT, the feature vector v of each candidate query graph Q is fed into the full-connection layer of the enhancement model to obtain p (q|Q), and the training target is a learning strategy function p q (q|q), where θ represents a parameter in the model; p is a policy function; using F1 points between predicted answers and correct answer labels as rewards, the BERT-based semantic matching uses a standard BERT model to scale a sequence of tagged questions s q And query graph sequence s g Semantic similarity between, where s g The sequence [ CLS ] is generated by sequentially concatenating the ground entity name and the relationship name along the core relationship path]s q [SEP]s g Providing the BERT models to calculate their semantic similarity; the number of the entities is n in the query graph g Number, where CLS is a sentence start tag; SEP is a flag separating two sentences; the number of entity types is the number of entity types; the number of answer entities is n in the query graph an Is the number of (3); the aggregate number is n in the query graph ag Is a number of (3).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310277208.7A CN116383354A (en) | 2023-03-21 | 2023-03-21 | Automatic visual question-answering method based on knowledge graph |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310277208.7A CN116383354A (en) | 2023-03-21 | 2023-03-21 | Automatic visual question-answering method based on knowledge graph |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116383354A true CN116383354A (en) | 2023-07-04 |
Family
ID=86968629
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310277208.7A Pending CN116383354A (en) | 2023-03-21 | 2023-03-21 | Automatic visual question-answering method based on knowledge graph |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116383354A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116821712A (en) * | 2023-08-25 | 2023-09-29 | 中电科大数据研究院有限公司 | Semantic matching method and device for unstructured text and knowledge graph |
-
2023
- 2023-03-21 CN CN202310277208.7A patent/CN116383354A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116821712A (en) * | 2023-08-25 | 2023-09-29 | 中电科大数据研究院有限公司 | Semantic matching method and device for unstructured text and knowledge graph |
| CN116821712B (en) * | 2023-08-25 | 2023-12-19 | 中电科大数据研究院有限公司 | Semantic matching method and device for unstructured text and knowledge graph |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | The gap of semantic parsing: A survey on automatic math word problem solvers | |
| CN111259653B (en) | Knowledge graph question-answering method, system and terminal based on entity relationship disambiguation | |
| CN111950285B (en) | Medical knowledge graph intelligent automatic construction system and method with multi-mode data fusion | |
| CN110990590A (en) | Dynamic financial knowledge map construction method based on reinforcement learning and transfer learning | |
| CN110188147B (en) | Knowledge graph-based document entity relationship discovery method and system | |
| Wilkinson et al. | Neural Ctrl-F: segmentation-free query-by-string word spotting in handwritten manuscript collections | |
| CN111858896B (en) | Knowledge base question-answering method based on deep learning | |
| CN112597316B (en) | Method and device for interpretive reasoning question-answering | |
| CN112989208B (en) | Information recommendation method and device, electronic equipment and storage medium | |
| CN112115252B (en) | Intelligent auxiliary writing processing method and device, electronic equipment and storage medium | |
| CN112328800A (en) | System and method for automatically generating programming specification question answers | |
| CN110968708A (en) | Method and system for labeling education information resource attributes | |
| CN117033571A (en) | Knowledge question-answering system construction method and system | |
| CN116383354A (en) | Automatic visual question-answering method based on knowledge graph | |
| CN114357022B (en) | Media content association mining method based on event relation discovery | |
| CN114492443A (en) | Method and system for training entity recognition model and entity recognition method and system | |
| CN113868406A (en) | Search method, search system, and computer-readable storage medium | |
| CN114117000A (en) | Response method, device, equipment and storage medium | |
| CN114239828A (en) | Supply chain affair map construction method based on causal relationship | |
| CN113505195A (en) | Knowledge base, construction method and retrieval method thereof, and question setting method and system based on knowledge base | |
| Zschech et al. | Towards a text-based recommender system for data mining method selection | |
| Tilak et al. | Visual entity linking | |
| CN115982322A (en) | Water conservancy industry design field knowledge graph retrieval method and retrieval system | |
| CN112989811B (en) | History book reading auxiliary system based on BiLSTM-CRF and control method thereof | |
| CN114579705A (en) | Learning auxiliary method and system for education of sustainable development |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |