CN118093782A

CN118093782A - Processing method and device for retrieving multi-mode information based on knowledge graph

Info

Publication number: CN118093782A
Application number: CN202410366906.9A
Authority: CN
Inventors: 刘明
Original assignee: Beijing Borui Tongyun Technology Co ltd
Current assignee: Beijing Borui Tongyun Technology Co ltd
Priority date: 2024-03-28
Filing date: 2024-03-28
Publication date: 2024-05-28
Anticipated expiration: 2044-03-28
Also published as: CN118093782B

Abstract

The embodiment of the invention relates to a processing method and a device for retrieving multi-mode information based on a knowledge graph, wherein the method comprises the following steps: constructing a domain ontology graph based on the knowledge graph structure; constructing a multi-mode document map based on the knowledge graph structure; receiving a first query text; performing domain expansion on the first query text based on the domain ontology graph to obtain a corresponding first expansion query text; and carrying out multi-mode information retrieval based on the first expansion query text and the multi-mode document map to obtain a corresponding first retrieval report. The invention can improve the retrieval matching degree and the retrieval efficiency.

Description

Processing method and device for retrieving multi-mode information based on knowledge graph

Technical Field

The invention relates to the technical field of data processing, in particular to a processing method and a device for retrieving multi-mode information based on a knowledge graph.

Background

One implementation scheme of the multi-modal information retrieval is as follows: collecting information sources of different types (or different modes) as retrieval resources, such as texts, images, audio, video and the like; labeling text labels in the professional field for each information source; and filtering the information sources based on the matching degree of the search text input by the user and the text labels of the information sources when the information search is processed, and outputting the information sources with higher matching degree to the user. This conventional implementation exposes some problems in specific applications: 1) The user's professional degree is not enough, the professional characteristics of the input search text are not obvious, and the matching degree between the search text and the text label with stronger professional degree is low, so that enough information sources can not be searched for the user; 2) The text labels are too short to fully describe the content of the search resource, which may also result in a low matching degree of the text labels and the search text; 3) And the retrieval is carried out in a resource-by-resource matching mode, so that the retrieval efficiency is low.

Disclosure of Invention

The invention aims at overcoming the defects of the prior art and provides a processing method, a device, electronic equipment and a computer readable storage medium for retrieving multi-mode information based on a knowledge graph. The invention constructs two knowledge maps in advance, namely a domain ontology map and a multi-mode document map respectively; wherein the domain ontology graph stores professional and non-professional vocabulary (concepts, attributes, instances) of each designated domain based on a directed graph of a tree structure; the multi-mode document graph is an undirected graph, nodes in the multi-mode document graph are in one-to-one correspondence with search resources, each node is used for configuring a title abstract text, namely a node text for the corresponding resource to more fully describe the resource, and the similarity of the resources of every two nodes can be set by connecting edge weights among nodes of the same type (web page, text, image, audio, video and the like); when a query text input by a user is received, the method and the device firstly expand the field of the query text based on the field ontology graph to supplement more auxiliary query information for the query text, then query the multi-mode document graph based on the expanded query text, and provide associated retrieval assistance by utilizing the node connection edge weight of the multi-mode document graph in the query process. The invention supplements the auxiliary query information, expands the description of the resource and assists the search of the association of the resource to the search text with insufficient professional degree, thereby improving the search matching degree and the search efficiency.

To achieve the above object, a first aspect of the present invention provides a processing method for retrieving multi-modal information based on a knowledge graph, where the method includes:

Constructing a domain ontology graph based on the knowledge graph structure;

Constructing a multi-mode document map based on the knowledge graph structure;

Receiving a first query text; performing domain expansion on the first query text based on the domain ontology graph to obtain a corresponding first expansion query text; and carrying out multi-mode information retrieval based on the first expansion query text and the multi-mode document map to obtain a corresponding first retrieval report.

Preferably, the domain ontology graph comprises a plurality of first nodes and a plurality of first connection edges;

Each first node corresponds to a concept vocabulary text, an attribute vocabulary text or an instance vocabulary text; the first node comprises a first node identifier, a first node type, a first node text, a first text type and a first text vector; the first node identifier is a unique identifier of the first node; the first node type comprises root nodes and non-root nodes, and the first node type is unique in number of the first nodes of the root nodes; the first node text is a concept vocabulary text, an attribute vocabulary text or an instance vocabulary text corresponding to the first node; the first text type includes concepts, attributes, and instances; the first text vector is a coded vector obtained by coding the first node text according to a preset text coding rule;

Each first connection edge is a directed edge and is used for connecting two first nodes, the two connected first nodes are corresponding first source nodes and first target nodes respectively, and the direction of each first connection edge is the connection direction from the first source node to the first target node; the first connection edge comprises a first connection edge identifier, a first source node identifier, a first target node identifier and a first connection edge type; the first connection edge identifier is a unique identifier of the first connection edge; the first node identifiers of the first source node and the first target node respectively correspond to the first source node identifier and the first target node identifier; the first connection edge type comprises a lower-level concept connection edge, a lower-level instance connection edge and a lower-level attribute connection edge;

each first node is connected with a plurality of other first nodes through one or more first connecting edges; the first node with the first node type being a root node is connected with a plurality of other first nodes only through one or more first connecting edges with the connecting edge type being lower-level concept connecting edges.

Preferably, the multi-mode document map includes a plurality of second nodes and a plurality of second connection edges;

Each second node corresponds to a document information source, and the information source type of the document information source comprises a webpage type, an electronic document type, an image file type, an audio file type and a video file type; the second node comprises a second node identifier, a second node type, a second node time, a second node download address, a second node text and a second text vector; the second node identifier is a unique identifier of the second node; the second node type comprises a webpage type, an electronic document type, an image file type, an audio file type and a video file type; the second node time is the release time of the corresponding document information source; the second node storage address is a download address of the document information source corresponding to the second node; the second node text consists of a title text and a abstract text of the document information source corresponding to the second node; the second text vector is a coding vector obtained by coding the second node text according to the text coding rule;

Each second connecting edge is an undirected edge and is used for connecting two second nodes, namely a corresponding first connecting node and a corresponding second connecting node, and the types of the second nodes of the two second nodes connected by each second connecting edge are the same; the second connecting edge comprises a second connecting edge identifier, a first connecting point identifier, a second connecting point identifier and a second connecting edge weight; the second connection edge identifier is a unique identifier of the first connection edge; the first node identifiers of the first and second connection nodes respectively corresponding to the first and second connection point identifiers; the weight of the second connecting edge is the similarity of the two document information sources corresponding to the second connecting edge;

Each second node is connected with a plurality of other second nodes through one or more second connecting edges.

Preferably, the building of the domain ontology graph based on the knowledge graph structure specifically includes:

Setting root concept vocabulary text of a designated field; acquiring a plurality of concept vocabulary texts except the root concept vocabulary text in the appointed field through big data acquisition to form a sub-concept vocabulary set; acquiring a plurality of attribute vocabulary texts and/or a plurality of instance vocabulary texts of each concept vocabulary text in the appointed field through big data acquisition to form a corresponding sub concept attribute instance set; the sub-concept vocabulary set includes a plurality of the concept vocabulary texts; the sub-concept attribute instance sets are in one-to-one correspondence with the concept vocabulary text; the sub-concept attribute instance set comprises a plurality of attribute vocabulary texts and/or a plurality of instance vocabulary texts;

Selecting a plurality of next-level concept vocabulary texts corresponding to the root concept vocabulary text from the sub-concept vocabulary set to form a corresponding first-level concept vocabulary set, and deleting the first-level concept vocabulary set from the sub-concept vocabulary set; selecting a secondary concept vocabulary set corresponding to a plurality of next-level concept vocabulary texts corresponding to the concept vocabulary texts of the primary concept vocabulary set from the sub concept vocabulary set, and deleting all the secondary concept vocabulary sets from the sub concept vocabulary set; and so on until the sub-concept vocabulary is emptied;

Constructing a first node with the first node type as a root node based on the root concept vocabulary text, and recording the first node as a first root node; constructing a first node with the first node type being a non-root node based on each conceptual vocabulary text, and recording the first node as a first non-root node; taking the first root node as a corresponding current father node, taking each first non-root node corresponding to the first-level concept word set corresponding to the current father node as a corresponding first child node, and constructing a corresponding first connection edge according to the connection direction from the current father node to each first child node; then, each first child node is used as a corresponding current father node, each first non-root node corresponding to the secondary concept word set corresponding to the current father node is recorded as a corresponding second child node, and a corresponding first connection edge is constructed according to the connection direction from the current father node to each second child node; and so on until all the first non-root nodes are connected with the corresponding father nodes and/or all the child nodes; and all the obtained first nodes form a corresponding current node set; and all the obtained first connecting edges form a corresponding current connecting edge set;

traversing all the first non-root nodes; traversing, wherein the first non-root node traversed currently is used as a corresponding current sub-concept node; the sub-concept attribute instance set corresponding to the current sub-concept node is used as a corresponding current attribute instance set; constructing a first node of which the first node type is a non-root node on the basis of each attribute vocabulary text of the current attribute instance set as a corresponding first sub-concept attribute node, constructing a first node of which the first node type is a non-root node on the basis of each instance vocabulary text of the current attribute instance set as a corresponding first sub-concept instance node, constructing a corresponding first connection edge according to the connection direction from the current sub-concept node to each first sub-concept attribute node, and constructing a corresponding first connection edge according to the connection direction from the current sub-concept node to each first sub-concept instance node; adding all the obtained first sub-concept attribute nodes and all the obtained first sub-concept instance nodes to the current node set; adding all the obtained first connection edges to the current connection edge set;

And in the current node set, allocating a unique identifier for each first node as the first node identifier of the current first node; the root concept vocabulary text, the attribute vocabulary text or the instance vocabulary text corresponding to each first node are used as the first node text of the current first node, the corresponding first text type is set as a concept when the current first node text is one of the root concept vocabulary text or the concept vocabulary text, the corresponding first text type is set as an attribute when the current first node text is one of the attribute vocabulary text, and the corresponding first text type is set as an instance when the current first node text is one of the instance vocabulary text; coding each first node text according to the text coding rule to obtain corresponding first coding vectors, and taking each first coding vector as the corresponding first text vector of the first node;

In the current connection edge set, a unique identifier is allocated to each first connection edge as the first connection edge identifier of the current first connection edge; setting the first source node identification and the first target node identification of the current first connection edge based on the first node identifications of the first source node and the first target node corresponding to the first connection edge; identifying the two first text types corresponding to the first source node and the first target node corresponding to the first connection edges; if the first text type corresponding to the first source node is a concept and the first text type corresponding to the first target node is a concept, setting the first connection edge type of the current first connection edge as a lower concept connection edge; if the first text type corresponding to the first source node is a concept and the first text type corresponding to the first target node is an instance, setting the first connection edge type of the current first connection edge as a lower instance connection edge; if the first text type corresponding to the first source node is a concept and the first text type corresponding to the first target node is an attribute, setting the first connection edge type of the current first connection edge as a lower attribute connection edge.

Preferably, the constructing a multi-mode document map based on the knowledge graph structure specifically includes:

periodically collecting big data of the multi-mode documents released in the latest first period in the appointed field to form a corresponding first collection document set and storing the first collection document set; the first collection document set comprises a plurality of first collection documents; each first collection document corresponds to a document release time, a document type and a document storage address; defaulting the ending time of the latest first time period to be the current time, wherein the time interval between the starting time and the ending time of the latest first time period is a preset first duration; the document types include a web page type, an electronic document type, an image file type, an audio file type, and a video file type;

Identifying whether the multi-mode document map has completed initial construction; initializing the multi-modal document map based on the first collection document set if the multi-modal document map is not initially constructed; and if the multi-mode document map is already built initially, updating the multi-mode document map based on the first collection document set.

Further, the initializing the multi-modal document map based on the first collection document set specifically includes:

Performing title and abstract text recognition on each first collection document in the first collection document set, forming corresponding first title abstract text by the recognized title and abstract text, and encoding the first title abstract text according to the text encoding rule to obtain corresponding second encoding vector;

Generating a corresponding second node for each first collection document in the first collection document set; and allocating a unique identifier to each second node as the second node identifier of the current second node; setting the second node type, the second node time, the second node download address, the second node text and the second text vector of each second node as the corresponding document type, the document release time, the document saving address, the first title abstract text and the second code vector corresponding to the corresponding first acquisition document;

And aggregating a plurality of second nodes with the same second node type into a first class node set corresponding to the first class node set;

Traversing all the second nodes of each first-class node set; the second node which is traversed at present is taken as a corresponding current node, and other second nodes which are not constructed by connecting edges with the current node in any one of the current first-class node sets are recorded as corresponding first other nodes; vector similarity calculation is carried out on the second text vector of the current node and the second text vectors of the first other nodes, and the calculated result is used as corresponding first similarity; deleting the first similarity which does not exceed a preset first similarity threshold; sequencing all the rest first similarity according to the sequence from high similarity to low similarity to obtain a corresponding first similarity sequence; counting the sequence length of the first similarity sequence to obtain a corresponding first sequence length; when the first sequence length is not 0, identifying whether the first sequence length is larger than a preset first number; if the length of the first sequence is greater than the first number, extracting the first number of first similarity in the first similarity sequence, which is ranked at the front, to form a corresponding second similarity sequence; if the length of the first sequence is smaller than or equal to the first number, the first similarity sequence is used as the corresponding second similarity sequence; constructing a corresponding second connecting edge for the current node and the first other nodes corresponding to the first similarity of the second similarity sequence, distributing a unique identifier for each second connecting edge as a corresponding second connecting edge identifier, setting the first connecting point identifier and the second connecting point identifier of the current second connecting edge based on the two second node identifiers of the current node and the first other nodes corresponding to each second connecting edge, and setting the second connecting edge weight of the current second connecting edge based on the first similarity corresponding to each second connecting edge; when the traversing is finished, forming a corresponding first type connecting edge set by all the second connecting edges corresponding to the current first type node set; the first number is an integer greater than 0;

And the initial multi-mode document graph is formed by all the obtained first-type node sets and all the first-type connecting edge sets.

Further, the updating the multi-modal document map based on the first collection document set specifically includes:

Performing title and abstract text recognition on each first collection document in the first collection document set, forming a corresponding second title abstract text by the recognized title and abstract text, and encoding the second title abstract text according to the text encoding rule to obtain a corresponding third encoding vector;

Generating a corresponding second node for each first collection document in the first collection document set and recording the second node as a corresponding increment node; distributing a unique identifier to each incremental node as the second node identifier of the current incremental node; setting the second node type, the second node time, the second node download address, the second node text and the second text vector of each incremental node as the corresponding document type, the document release time, the document saving address, the second title abstract text and the third code vector corresponding to the corresponding first acquisition document;

And recording all second nodes of which the second node time satisfies the latest second period in the current multi-mode document map as corresponding stock nodes; the plurality of increment nodes and/or the plurality of stock nodes with the same second node type are gathered into a corresponding second class node set; defaulting the ending time of the latest second time period to be the current time, wherein the time interval between the starting time and the ending time of the latest second time period is a preset second duration;

Traversing all the second nodes of each second class node set; the second node which is traversed at present is taken as a corresponding current node, and other second nodes which are constructed by connecting edges which are not completed with the current node in any one of the second class node sets at present are recorded as corresponding second other nodes; vector similarity calculation is carried out on the second text vector of the current node and the second text vectors of the second other nodes, and the calculated result is used as corresponding second similarity; deleting the second similarity which does not exceed a preset first similarity threshold; sequencing all the remaining second similarity according to the sequence from high similarity to low similarity to obtain a corresponding third similarity sequence; counting the sequence length of the third similarity sequence to obtain a corresponding second sequence length; when the second sequence length is not 0, identifying whether the second sequence length is larger than a preset first number; if the length of the second sequence is greater than the first number, extracting the first number of second similarity in the third similarity sequence, which is ranked at the front, to form a corresponding fourth similarity sequence; if the length of the second sequence is smaller than or equal to the first number, the third similarity sequence is used as the corresponding fourth similarity sequence; constructing a corresponding second connecting edge for the second other nodes corresponding to the second similarity of the current node and the fourth similarity sequence, distributing a unique identifier for each second connecting edge as a corresponding second connecting edge identifier, setting the first connecting point identifier and the second connecting point identifier of the current second connecting edge based on the two second node identifiers of the current node and the second other nodes corresponding to each second connecting edge, and setting the second connecting edge weight of the current second connecting edge based on the second similarity corresponding to each second connecting edge; when the traversing is finished, forming a corresponding second-type connecting edge set by all the second connecting edges corresponding to the current second-type node set;

And adding the updated multi-modal document map to the current multi-modal document map by using the obtained incremental nodes and the obtained second-type connection edge sets.

Preferably, the performing the domain expansion on the first query text based on the domain ontology graph to obtain a corresponding first expanded query text specifically includes:

encoding the first query text according to the text encoding rule to obtain a corresponding first query encoding vector;

vector similarity calculation is carried out on the first query coding vector and the first text vectors of the first nodes with the first text types as concepts in the domain ontology graph, and the calculation result is used as corresponding first concept similarity; and taking the first concept similarity with the highest similarity as the corresponding maximum concept similarity;

Identifying whether the maximum concept similarity exceeds a preset second similarity threshold;

If the maximum concept similarity exceeds the second similarity threshold, the first node corresponding to the maximum concept similarity is used as a corresponding current retrieval node;

If the maximum concept similarity does not exceed the second similarity threshold, vector similarity calculation is performed on the first query coding vector and the first text vector of the first node of which each first text type is an attribute or an instance in the domain ontology graph, and the calculated result is used as corresponding first attribute/instance similarity; and taking the first attribute/instance similarity with the highest similarity as the corresponding maximum attribute/instance similarity; and identifying whether the maximum attribute/instance similarity exceeds the second similarity threshold; if the maximum attribute/instance similarity exceeds the second similarity threshold, taking the first node corresponding to the maximum attribute/instance similarity as the corresponding current retrieval node; if the maximum attribute/instance similarity does not exceed the second similarity threshold, setting the current retrieval node to be empty;

identifying whether the current retrieval node is empty;

if the current retrieval node is empty, setting the corresponding first expansion query text as the first query text;

If the current retrieval node is not empty, identifying the first text type of the current retrieval node; if the first text type is a concept, marking all the first nodes connected with the current retrieval node as corresponding first lower retrieval nodes, and extracting the current retrieval node and the corresponding first node texts of all the first lower retrieval nodes to form a corresponding first node text sequence; if the first text type is an attribute or an instance, the first node, which is connected with the current retrieval node and has the first text type of a concept, is taken as a corresponding parent retrieval node, all the first nodes connected with the parent retrieval node are recorded as corresponding second subordinate retrieval nodes, and the first node texts of the parent retrieval node and all the corresponding second subordinate retrieval nodes are extracted to form a corresponding first node text sequence; and performing text splicing on the obtained first node text sequence and the first query text to obtain the corresponding first expansion query text.

Preferably, the performing multi-mode information retrieval based on the first extended query text and the multi-mode document map to obtain a corresponding first retrieval report specifically includes:

step 91, coding the first expansion query text according to the text coding rule to obtain a corresponding second query coding vector;

step 92, setting a current segmentation period, specifically: taking the current time as the ending time of the current segmentation period, and taking the time obtained by subtracting a preset first time interval from the current time as the starting time of the current segmentation period;

step 93, forming a corresponding first node subset by the second nodes in the multi-mode document map, wherein the second node time satisfies the current segmentation period;

Step 94, calculating the vector similarity between the second text vector of each second node in the first node subset and the second query coding vector, and taking the calculation result as the corresponding first query similarity; deleting the first query similarity which does not exceed a preset third similarity threshold; identifying whether the number of the remaining first query similarities is lower than a preset second number; if yes, taking the starting time of the current segmentation period as a new ending time, taking the time obtained by subtracting the first time interval from the new ending time as a new starting time, forming a new current segmentation period by the new starting time and the new ending time, and returning to the step 93; if not, go to step 95;

step 95, taking the second node corresponding to each first query similarity as a corresponding first key node; in the multi-mode document graph, taking each first key node as a current vertex, and extracting all second nodes and all second connecting edges which are directly or indirectly connected with the current vertex to form a corresponding first key node subgraph; the unique vertexes of the first key node subgraphs are marked as corresponding first vertexes, and the second nodes except the first vertexes in the first key node subgraphs are marked as corresponding first non-vertexes; the first vertexes are in one-to-one correspondence with the first key nodes;

Step 96, in each first key node subgraph, marking the shortest path from any one of the first non-vertex to the first vertex as a corresponding first non-vertex root path, performing continuous multiplication calculation on the second connection side weights of all the second connection sides passing through on each first non-vertex root path, taking the obtained calculation result as a corresponding first non-vertex weight, taking the product of multiplication of each first non-vertex weight and the first query similarity corresponding to the corresponding first vertex as the first query similarity of the first non-vertex corresponding to the current first non-vertex weight, and deleting the first non-vertex, of which the first query similarity does not exceed the third similarity threshold, from the current first key node subgraph;

Step 97, forming a corresponding query node set by all the second nodes of all the first key node subgraphs; performing node de-duplication processing on the query node set;

Step 98, aggregating a plurality of second nodes with the same second node type in the de-duplicated query node set into a first type query node set corresponding to the first type query node set; and ordering all the second nodes in each first-type query node set according to the sequence from high to low of the corresponding first query similarity to form a corresponding first-type query node sequence; traversing all the second nodes of each first-type query node sequence one by one; traversing, wherein the second node traversed currently is used as a corresponding current node; the first search record corresponding to the current node, the second node time, the second node type, the second node text and the second node download address form a corresponding first search record; when the traversal is finished, all the first search records corresponding to the current first type query node sequence are sequenced from high to low according to the corresponding first query similarity to form a corresponding first type search record sequence;

and step 99, forming the corresponding first search report by all the obtained first search record sequences.

A second aspect of the embodiment of the present invention provides an apparatus for implementing the processing method for retrieving multi-modal information based on a knowledge graph according to the first aspect, where the apparatus includes: the system comprises a domain ontology graph construction module, a multi-mode document graph construction module and a multi-mode query module;

the domain ontology graph construction module is used for constructing a domain ontology graph based on the knowledge graph structure;

The multi-modal document map construction module is used for constructing a multi-modal document map based on the knowledge graph structure;

the multi-mode query module is used for receiving a first query text; performing domain expansion on the first query text based on the domain ontology graph to obtain a corresponding first expansion query text; and carrying out multi-mode information retrieval based on the first expansion query text and the multi-mode document map to obtain a corresponding first retrieval report.

A third aspect of an embodiment of the present invention provides an electronic device, including: memory, processor, and transceiver;

the processor is configured to couple to the memory, and read and execute the instructions in the memory, so as to implement the method steps described in the first aspect;

The transceiver is coupled to the processor and is controlled by the processor to transmit and receive messages.

A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions of the method of the first aspect.

The embodiment of the invention provides a processing method, a processing device, electronic equipment and a computer readable storage medium for retrieving multi-mode information based on a knowledge graph. From the above, it can be seen that, in the embodiment of the present invention, two knowledge maps are pre-built, which are a domain ontology map and a multi-modal document map respectively; wherein the domain ontology graph stores professional and non-professional vocabulary (concepts, attributes, instances) of each designated domain based on a directed graph of a tree structure; the multi-mode document graph is an undirected graph, nodes in the multi-mode document graph are in one-to-one correspondence with search resources, each node is used for configuring a title abstract text, namely a node text for the corresponding resource to more fully describe the resource, and the similarity of the resources of every two nodes can be set by connecting edge weights among nodes of the same type (web page, text, image, audio, video and the like); when a query text input by a user is received, the embodiment of the invention firstly expands the field of the query text based on the field ontology graph to supplement more auxiliary query information for the query text, then queries the multi-mode document graph based on the expanded query text, and provides associated retrieval assistance by utilizing the node connection edge weight of the multi-mode document graph in the query process. The embodiment of the invention supplements the auxiliary query information of the search text with insufficient specialty, expands the description of the resource and assists the search of the relevance of the resource, thereby improving the search matching degree and the search efficiency.

Drawings

Fig. 1 is a schematic diagram of a processing method for retrieving multi-modal information based on a knowledge-graph according to a first embodiment of the present invention;

FIG. 2a is a schematic diagram of a domain ontology diagram according to a first embodiment of the present invention;

FIG. 2b is a diagram of a multi-modal document map according to one embodiment of the present invention;

FIG. 3 is a block diagram of a processing device for retrieving multi-modal information based on a knowledge-graph according to a second embodiment of the present invention;

fig. 4 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The first embodiment of the invention provides a processing method for retrieving multi-mode information based on a knowledge graph, and the first embodiment of the invention can improve the real-time performance and the processing efficiency of user images; fig. 1 is a schematic diagram of a processing method for retrieving multi-mode information based on a knowledge graph according to a first embodiment of the present invention, as shown in fig. 1, the method mainly includes the following steps:

Step 1, constructing a domain ontology graph based on a knowledge graph structure;

as shown in fig. 2a, which is a schematic diagram of a domain ontology graph provided by a first embodiment of the present invention, the domain ontology graph of the embodiment of the present invention includes a plurality of first nodes and a plurality of first connection edges;

Each first node corresponds to a conceptual vocabulary text, an attribute vocabulary text or an instance vocabulary text; the first node includes a first node identification, a first node type, a first node text, a first text type, and a first text vector; the first node identifier is a unique identifier of the first node; the first node type comprises root nodes and non-root nodes, and is unique in number; the first node text is a conceptual vocabulary text, an attribute vocabulary text or an instance vocabulary text corresponding to the first node; the first text type includes concepts, attributes, and instances; the first text vector is a coded vector obtained by coding the first node text according to a preset text coding rule;

Each first connecting edge is a directed edge and is used for connecting two first nodes, the two connected first nodes are corresponding first source nodes and first target nodes respectively, and the direction of the first connecting edge is the connection direction from the first source nodes to the first target nodes; the first connection edge comprises a first connection edge identifier, a first source node identifier, a first target node identifier and a first connection edge type; the first connection edge identifier is a unique identifier of the first connection edge; the first node identifiers of the first source node and the first target node respectively correspond to the first source node identifier and the first target node identifier; the first connection edge type comprises a lower-level concept connection edge, a lower-level instance connection edge and a lower-level attribute connection edge; each first node is connected with a plurality of other first nodes through one or a plurality of first connecting edges; the first node with the first node type being a root node is connected with a plurality of other first nodes only through one or more first connecting edges with the connecting edge type being the connecting edge of the lower-level concept;

The text coding rule is a preset coding rule, and a conventional Word bag model coding rule, a TF-IDF coding rule, a Word2Vec coding rule, a GloVe coding rule and the like; domain Ontology (Domain Ontology) is a description of specific Domain professional concepts, and the hierarchy of the specific Domain professional concepts and the attribute and instance information corresponding to each concept can be summarized through a Domain Ontology graph;

the specific implementation steps of the current step 1 comprise:

Step 11, setting root concept vocabulary text of the appointed field; acquiring a plurality of concept vocabulary texts except the root concept vocabulary text in the appointed field through big data acquisition to form a sub-concept vocabulary set; acquiring a plurality of attribute vocabulary texts and/or a plurality of instance vocabulary texts of each concept vocabulary text in the appointed field through big data acquisition to form a corresponding sub-concept attribute instance set;

Wherein the sub-concept vocabulary set includes a plurality of concept vocabulary texts; the sub-concept attribute instance sets are in one-to-one correspondence with concept vocabulary text; the sub-concept attribute instance set comprises a plurality of attribute vocabulary text and/or a plurality of instance vocabulary text;

Step 12, selecting a plurality of next-level concept vocabulary texts corresponding to the root concept vocabulary text from the sub-concept vocabulary set to form a corresponding first-level concept vocabulary set, and deleting the first-level concept vocabulary set from the sub-concept vocabulary set; selecting a plurality of next-level concept vocabulary texts corresponding to each concept vocabulary text of the first-level concept vocabulary set from the sub-concept vocabulary set to form a corresponding second-level concept vocabulary set, and deleting all the second-level concept vocabulary sets from the sub-concept vocabulary set; and so on until the sub-concept vocabulary is emptied;

Step 13, constructing a first node with the first node type as a root node based on the root concept vocabulary text, and recording the first node as the first root node; constructing a first node with a first node type being a non-root node based on each conceptual vocabulary text, and recording the first node as a first non-root node; then the first root node is used as a corresponding current father node, each first non-root node corresponding to the first-level concept word set corresponding to the current father node is used as a corresponding first child node, and a corresponding first connection edge is constructed according to the connection direction from the current father node to each first child node; then, each first child node is used as a corresponding current father node, each first non-root node corresponding to a secondary concept word set corresponding to the current father node is recorded as a corresponding second child node, and a corresponding first connection edge is constructed according to the connection direction from the current father node to each second child node; and so on until all the first non-root nodes are connected with the corresponding parent nodes and/or all the child nodes; all the obtained first nodes form a corresponding current node set; all the obtained first connecting edges form a corresponding current connecting edge set;

Step 14, traversing all the first non-root nodes; traversing, wherein a first non-root node traversed currently is used as a corresponding current sub-concept node; and taking the sub-concept attribute instance set corresponding to the current sub-concept node as a corresponding current attribute instance set; constructing a first node with a first node type being a non-root node based on each attribute vocabulary text of the current attribute instance set as a corresponding first sub-concept attribute node, constructing a first node with a first node type being a non-root node based on each instance vocabulary text of the current attribute instance set as a corresponding first sub-concept instance node, constructing a corresponding first connection edge according to the connection direction from the current sub-concept node to each first sub-concept attribute node, and constructing a corresponding first connection edge according to the connection direction from the current sub-concept node to each first sub-concept instance node; adding all the obtained first sub-concept attribute nodes and all the first sub-concept instance nodes to the current node set; adding all the obtained first connection edges to the current connection edge set;

Step 15, in the current node set, a unique identifier is allocated to each first node as a first node identifier of the current first node; the root concept vocabulary text, the attribute vocabulary text or the instance vocabulary text corresponding to each first node are used as the first node text of the current first node, the corresponding first text type is set as a concept when the current first node text is a root concept vocabulary text or a concept vocabulary text, the corresponding first text type is set as an attribute when the current first node text is an attribute vocabulary text, and the corresponding first text type is set as an instance when the current first node text is an instance vocabulary text; coding each first node text according to a text coding rule to obtain corresponding first coding vectors, and taking each first coding vector as a first text vector of a corresponding first node;

Step 16, in the current connection edge set, a unique identifier is allocated to each first connection edge as a first connection edge identifier of the current first connection edge; setting a first source node identifier and a first target node identifier of a current first connection edge based on first node identifiers of a first source node and a first target node corresponding to each first connection edge; identifying two first text types corresponding to a first source node and a first target node corresponding to each first connection edge; if the first text type corresponding to the first source node is a concept and the first text type corresponding to the first target node is a concept, setting a first connection edge type of a current first connection edge as a lower concept connection edge; if the first text type corresponding to the first source node is a concept and the first text type corresponding to the first target node is an example, setting a first connection edge type of a current first connection edge as a lower-level example connection edge; if the first text type corresponding to the first source node is a concept and the first text type corresponding to the first target node is an attribute, setting the first connection edge type of the current first connection edge as a lower attribute connection edge.

Step2, constructing a multi-mode document map based on the knowledge graph structure;

as shown in fig. 2b, which is a schematic diagram of a multi-mode document map provided by the first embodiment of the present invention, the multi-mode document map of the embodiment of the present invention includes a plurality of second nodes and a plurality of second connection edges;

Each second node corresponds to a document information source, and the information source type of the document information source comprises a webpage type, an electronic document type, an image file type, an audio file type and a video file type; the second node comprises a second node identifier, a second node type, a second node time, a second node download address, a second node text and a second text vector; the second node identifier is a unique identifier of the second node; the second node type comprises a webpage type, an electronic document type, an image file type, an audio file type and a video file type; the second node time is the release time of the corresponding document information source; the second node storage address is the download address of the document information source corresponding to the second node; the second node text consists of a title text and a abstract text of a document information source corresponding to the second node; the second text vector is a coding vector obtained by coding the second node text according to a text coding rule;

Each second connecting edge is an undirected edge and is used for connecting two second nodes which are respectively corresponding first connecting nodes and second connecting nodes, and the second nodes of the two second nodes connected by each second connecting edge are the same in type; the second connecting edge comprises a second connecting edge identifier, a first connecting point identifier, a second connecting point identifier and a second connecting edge weight; the second connecting edge mark is a unique mark of the first connecting edge; first node identifiers of the first and second connection nodes respectively corresponding to the first and second connection point identifiers; the weight of the second connecting edge is the similarity of two document information sources corresponding to the second connecting edge; each second node is connected with a plurality of other second nodes through one or a plurality of second connecting edges;

the specific implementation steps of the current step 2 comprise:

Step 21, periodically collecting big data of the multi-mode documents released in the first period in the appointed field to form a corresponding first collection document set and storing the first collection document set;

Wherein the first collection document set includes a plurality of first collection documents; each first collection document corresponds to a document release time, a document type and a document storage address; defaulting the ending time of the latest first time period to be the current time, wherein the time interval between the starting time and the ending time of the latest first time period is a preset first time length, and the first time length is a preset time length parameter; the document types include a web page type, an electronic document type, an image file type, an audio file type, and a video file type;

step 22, identifying whether the initial construction of the multi-mode document map is completed;

step 23, initializing the multi-modal document map based on the first collection document set if the multi-modal document map is not initially constructed;

the method specifically comprises the following steps: step 231, performing title and abstract text recognition on each first collection document in the first collection document set, forming a corresponding first title abstract text by the recognized title and abstract text, and encoding the first title abstract text according to a text encoding rule to obtain a corresponding second encoding vector;

Step 232, generating a corresponding second node for each first collection document in the first collection document set; and a unique identifier is allocated to each second node as a second node identifier of the current second node; setting a second node type, a second node time, a second node downloading address, a second node text and a second text vector of each second node as a corresponding document type, a document release time, a document saving address, a first headline abstract text and a second coding vector corresponding to the corresponding first acquisition document;

Step 233, aggregating a plurality of second nodes with the same second node type into a first class node set corresponding to the first class node set;

Step 234, traversing all the second nodes of each first class node set; the second node which is traversed at present is taken as a corresponding current node, and other second nodes which are not constructed by connecting edges with the current node in any one of the current first type node set are recorded as corresponding first other nodes; vector similarity calculation is carried out on the second text vector of the current node and the second text vectors of all the first other nodes, and the calculated result is used as corresponding first similarity; deleting the first similarity which does not exceed a preset first similarity threshold; sequencing all the remaining first similarity according to the sequence from high to low of the similarity to obtain a corresponding first similarity sequence; counting the sequence length of the first similarity sequence to obtain a corresponding first sequence length; when the first sequence length is not 0, identifying whether the first sequence length is larger than a preset first number; if the length of the first sequence is greater than the first number, extracting the first similarity of the first number, which is arranged at the front, in the first similarity sequence to form a corresponding second similarity sequence; if the length of the first sequence is smaller than or equal to the first number, the first similarity sequence is used as a corresponding second similarity sequence; constructing a corresponding second connecting edge for the current node and the first other nodes corresponding to the first similarity of the second similarity sequence, distributing a unique identifier for each second connecting edge as a corresponding second connecting edge identifier, setting the first connecting edge identifier and the second connecting edge identifier of the current second connecting edge based on the current node corresponding to each second connecting edge and the two second node identifiers of the first other nodes, and setting the second connecting edge weight of the current second connecting edge based on the first similarity corresponding to each second connecting edge; when the traversing is finished, forming a corresponding first type connecting edge set by all second connecting edges corresponding to the current first type node set;

The first similarity threshold is a preset similarity parameter, and the first number is an integer greater than 0;

step 235, forming an initial multi-mode document graph by all the obtained first-type node sets and all the first-type connecting edge sets;

step 24, if the multi-modal document map has completed the initial construction, updating the multi-modal document map based on the first collection document set;

the method specifically comprises the following steps: step 241, performing title and abstract text recognition on each first collection document in the first collection document set, forming a corresponding second title abstract text by the recognized title and abstract text, and encoding the second title abstract text according to a text encoding rule to obtain a corresponding third encoding vector;

step 242, generating a corresponding second node for each first collection document in the first collection document set and recording the second node as a corresponding incremental node; distributing a unique identifier for each incremental node as a second node identifier of the current incremental node; setting a second node type, a second node time, a second node downloading address, a second node text and a second text vector of each increment node as a corresponding document type, a document release time, a document saving address, a second title abstract text and a third coding vector corresponding to the corresponding first acquisition document;

Step 243, and all second nodes in the current multi-mode document map, the second node time of which meets the latest second period, are recorded as corresponding stock nodes; a plurality of increment nodes and/or a plurality of stock nodes with the same second node type are gathered into a corresponding second class node set;

The ending time of the latest second time period defaults to the current time, and the time interval between the starting time and the ending time of the latest second time period is a preset second duration; the second time length is a preset time length parameter;

Step 244, traversing all second nodes of each second class node set; the second node which is traversed at present is taken as a corresponding current node, and other second nodes which are not constructed by connecting edges with the current node in any one of the second class node sets at present are recorded as corresponding second other nodes; vector similarity calculation is carried out on the second text vector of the current node and the second text vectors of all the second other nodes, and the calculated result is used as corresponding second similarity; and deleting the second similarity which does not exceed the first similarity threshold; sequencing all the remaining second similarities according to the sequence from high to low of the similarities to obtain a corresponding third similarity sequence; counting the sequence length of the third similarity sequence to obtain a corresponding second sequence length; and when the second sequence length is not 0, identifying whether the second sequence length is greater than the first number; if the length of the second sequence is greater than the first number, extracting the first number of second similarity in the first sequence of the third similarity to form a corresponding fourth similarity sequence; if the length of the second sequence is smaller than or equal to the first number, the third similarity sequence is used as a corresponding fourth similarity sequence; constructing a corresponding second connecting edge for the second other nodes corresponding to the second similarity of the current node and the fourth similarity sequence, distributing a unique identifier for each second connecting edge as a corresponding second connecting edge identifier, setting the first and second connecting point identifiers of the current second connecting edge based on the current node corresponding to each second connecting edge and the two second node identifiers of the second other nodes, and setting the second connecting edge weight of the current second connecting edge based on the second similarity corresponding to each second connecting edge; when the traversing is finished, forming a corresponding second-type connecting edge set by all second connecting edges corresponding to the current second-type node set;

And 245, adding the obtained incremental nodes and the obtained second-type connection edge sets to the current multi-mode document map to obtain an updated multi-mode document map.

Step 3, receiving a first query text; performing domain expansion on the first query text based on the domain ontology graph to obtain a corresponding first expansion query text; carrying out multi-mode information retrieval based on the first expansion query text and the multi-mode document map to obtain a corresponding first retrieval report;

The method specifically comprises the following steps: step 31, receiving a first query text;

step 32, performing domain expansion on the first query text based on the domain ontology graph to obtain a corresponding first expanded query text;

The method specifically comprises the following steps: step 321, coding the first query text according to the text coding rule to obtain a corresponding first query coding vector;

Step 322, performing vector similarity calculation on the first query coding vector and the first text vector of the first node with each first text type as a concept in the domain ontology graph, and taking the calculation result as a corresponding first concept similarity; and taking the first concept similarity with the highest similarity as the corresponding maximum concept similarity;

Step 323, identifying whether the maximum concept similarity exceeds a preset second similarity threshold;

Here, the second similarity threshold is a preset similarity parameter,

Step 324, if the maximum concept similarity exceeds the second similarity threshold, using the first node corresponding to the maximum concept similarity as the corresponding current search node;

Step 325, if the maximum concept similarity does not exceed the second similarity threshold, performing vector similarity calculation on the first query encoding vector and the first text vector of the first node of each first text type in the domain ontology graph, where the first text type is an attribute or an instance, and taking the calculation result as a corresponding first attribute/instance similarity; and taking the first attribute/instance similarity with the highest similarity as the corresponding maximum attribute/instance similarity; and identifying whether the maximum attribute/instance similarity exceeds a second similarity threshold; if the maximum attribute/instance similarity exceeds a second similarity threshold, taking the first node corresponding to the maximum attribute/instance similarity as a corresponding current retrieval node; if the maximum attribute/instance similarity does not exceed the second similarity threshold, setting the current retrieval node to be empty;

step 326, identifying whether the current retrieval node is empty;

step 327, if the current search node is empty, setting the corresponding first expansion query text as the first query text;

Step 328, if the current search node is not empty, identifying the first text type of the current search node; if the first text type is a concept, marking all first nodes connected with the current retrieval node as corresponding first lower retrieval nodes, and extracting first node texts of the current retrieval node and all corresponding first lower retrieval nodes to form a corresponding first node text sequence; if the first text type is an attribute or an instance, the first node of which the upper first text type is a concept and connected with the current retrieval node is used as a corresponding parent retrieval node, all the first nodes connected with the parent retrieval node are marked as corresponding second lower retrieval nodes, and the first node texts of the parent retrieval node and all the corresponding second lower retrieval nodes are extracted to form a corresponding first node text sequence; performing text splicing on the obtained first node text sequence and the first query text to obtain a corresponding first expansion query text;

Step 33, performing multi-mode information retrieval based on the first expansion query text and the multi-mode document map to obtain a corresponding first retrieval report;

The method specifically comprises the following steps: step 331, coding the first expansion query text according to a text coding rule to obtain a corresponding second query coding vector;

Step 332, setting a current segmentation period, specifically: taking the current time as the ending time of the current segmentation period, and taking the time obtained by subtracting a preset first time interval from the current time as the starting time of the current segmentation period; here, the first time interval is a preset duration parameter;

step 333, forming a corresponding first node subset by the second nodes in the multi-mode document map, wherein the second node time satisfies the current segmentation period;

Step 334, calculating the vector similarity between the second text vector of each second node in the first node subset and the second query coding vector, and taking the calculation result as the corresponding first query similarity; deleting the first query similarity which does not exceed a preset third similarity threshold; identifying whether the number of the remaining first query similarities is lower than a preset second number; if yes, taking the starting time of the current segmentation period as a new ending time, taking the time obtained by subtracting the first time interval from the new ending time as a new starting time, forming a new current segmentation period by the new starting time and the new ending time, and returning to the step 333; if not, go to step 335;

here, the third similarity threshold is a preset similarity parameter, and the second number is a preset integer greater than or equal to 0;

For example, when the first set time period is from 2023, 12, 1, zero to 2023, 12, 31, 24, the number of the first query similarities exceeding the third similarity threshold obtained through steps 333-334 is 2, which is recorded as the first query similarity A, B; knowing the second number=3, the second setting needs to be reset, and if the first time interval is 1 month, the time period after reset is 2023, 11, 1, zero to 2023, 11, 3, 24; then return to step 332; setting the number of the new batches of the first query similarity exceeding the third similarity threshold value obtained in the steps 333-334 to be 2, and recording the number as the first query similarity C, D; at this time, the total number of the accumulated first query similarities (a-D) is 4, and if the number exceeds the second number by 3, the process goes to step 335 to perform relevance search;

Step 335, using the second node corresponding to each first query similarity as the corresponding first key node; in the multi-mode document graph, taking each first key node as a current vertex, and extracting all second nodes and all second connecting edges which are directly or indirectly connected with the current vertex to form a corresponding first key node subgraph; marking the unique vertex of each first key node sub-graph as a corresponding first vertex, and marking each second node except the first vertex in each first key node sub-graph as a corresponding first non-vertex; wherein the first vertexes are in one-to-one correspondence with the first key nodes;

Step 336, in each first key node subgraph, marking the shortest path from any first non-vertex to the first vertex as a corresponding first non-vertex root path, performing continuous multiplication calculation on the second connection edge weights of all the second connection edges passing through each first non-vertex root path, taking the obtained calculation result as a corresponding first non-vertex weight, taking the product of multiplication of each first non-vertex weight and the first query similarity corresponding to the corresponding first vertex as the first query similarity of the first non-vertex corresponding to the current first non-vertex weight, and deleting the first non-vertex with the first query similarity not exceeding a third similarity threshold from the current first key node subgraph;

Step 337, forming a corresponding query node set by all second nodes of all first key node subgraphs; performing node de-duplication processing on the query node set;

Step 338, aggregating a plurality of second nodes with the same second node type in the de-duplicated query node set into a class to form a corresponding first class query node set; ordering all second nodes in each first-type query node set according to the sequence from high to low of the corresponding first query similarity to form a corresponding first-type query node sequence; traversing all second nodes of each first-class query node sequence one by one; traversing, wherein the second node traversed at present is used as the corresponding current node; the corresponding first search record is formed by the first query similarity, the second node time, the second node type, the second node text and the second node download address corresponding to the current node; when the traversal is finished, all the first search records corresponding to the current first-class query node sequence are sequenced from high to low according to the corresponding first query similarity to form a corresponding first-class search record sequence;

Step 339, composing the corresponding first search report by all the obtained first search record sequences.

Fig. 3 is a block diagram of a processing apparatus for retrieving multi-mode information based on a knowledge graph according to a second embodiment of the present invention, where the apparatus is a terminal device or a server for implementing the foregoing method embodiment, or may be an apparatus capable of enabling the foregoing terminal device or the server to implement the foregoing method embodiment, and for example, the apparatus may be an apparatus or a chip system of the foregoing terminal device or the server. As shown in fig. 3, the processing device for retrieving multi-modal information based on a knowledge graph includes: the domain ontology graph construction module 201, the multi-modal document graph construction module 202 and the multi-modal query module 203.

The domain ontology graph construction module 201 is configured to construct a domain ontology graph based on the knowledge graph structure.

The multimodal document map construction module 202 is configured to construct a multimodal document map based on the knowledge graph structure.

The multi-modal query module 203 is configured to receive a first query text; performing domain expansion on the first query text based on the domain ontology graph to obtain a corresponding first expansion query text; and carrying out multi-mode information retrieval based on the first expansion query text and the multi-mode document map to obtain a corresponding first retrieval report.

The processing device for retrieving multi-mode information based on the knowledge graph provided by the embodiment of the invention can execute the method steps in the method embodiment, and the implementation principle and the technical effect are similar and are not repeated here.

It should be noted that, it should be understood that the division of the modules of the above apparatus is merely a division of a logic function, and may be fully or partially integrated into a physical entity or may be physically separated. And these modules may all be implemented in software in the form of calls by the processing element; or can be realized in hardware; the method can also be realized in a form of calling software by a processing element, and the method can be realized in a form of hardware by a part of modules. For example, the domain ontology graph construction module may be a processing element which is set up separately, may be implemented in a chip of the above-mentioned apparatus, or may be stored in a memory of the above-mentioned apparatus in the form of program codes, and the functions of the above-mentioned determination module may be called and executed by a processing element of the above-mentioned apparatus. The implementation of the other modules is similar. In addition, all or part of the modules can be integrated together or can be independently implemented. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in a software form.

For example, the modules above may be one or more integrated circuits configured to implement the methods above, such as: one or more Application SPECIFIC INTEGRATED Circuits (ASIC), or one or more digital signal processors (DIGITAL SIGNAL Processor, DSP), or one or more field programmable gate arrays (Field Programmable GATE ARRAY, FPGA), etc. For another example, when a module above is implemented in the form of processing element scheduler code, the processing element may be a general purpose processor, such as a central processing unit (Central Processing Unit, CPU) or other processor that may invoke the program code. For another example, the modules may be integrated together and implemented in the form of a System-on-a-chip (SOC).

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces, in whole or in part, the processes or functions described in connection with the foregoing method embodiments. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, fiber optic, digital subscriber line ((Digital Subscriber Line, DSL)), or wireless (e.g., infrared, wireless, bluetooth, microwave, etc.), or a wireless (e.g., infrared, wireless, bluetooth, microwave, etc.), the computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server, data center, etc., that contains an integration of one or more available media.

Fig. 4 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention. The electronic device may be a terminal device or a server implementing the method of the foregoing embodiment, or may be a terminal device or a server implementing the method of the foregoing embodiment, which is connected to the foregoing terminal device or server. As shown in fig. 4, the electronic device may include: a processor 301 (e.g., a CPU), a memory 302, a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiving actions of the transceiver 303. The memory 302 may store various instructions for performing the various processing functions and implementing the processing steps described in the methods of the previous embodiments. Preferably, the electronic device according to the embodiment of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to implement communication connections between the elements. The communication port 306 is used for connection communication between the electronic device and other peripheral devices.

The system bus 305 referred to in fig. 4 may be a peripheral component interconnect standard (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, or the like. The system bus may be classified into an address bus, a data bus, a control bus, and the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus. The communication interface is used to enable communication between the database access apparatus and other devices (e.g., clients, read-write libraries, and read-only libraries). The Memory may include random access Memory (Random Access Memory, RAM) and may also include Non-Volatile Memory (Non-Volatile Memory), such as at least one disk Memory.

The processor may be a general-purpose processor, including a Central Processing Unit (CPU), a network processor (Network Processor, NP), a graphics processor (Graphics Processing Unit, GPU), etc.; but may also be a digital signal processor DSP, an application specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

It should be noted that, the embodiments of the present invention also provide a computer readable storage medium, where instructions are stored, when the computer readable storage medium runs on a computer, to cause the computer to perform the method and the process provided in the above embodiments.

The embodiment of the invention also provides a chip for running the instructions, and the chip is used for executing the processing steps described in the embodiment of the method.

The embodiment of the invention provides a processing method, a processing device, electronic equipment and a computer readable storage medium for retrieving multi-mode information based on a knowledge graph; as can be seen from the above summary, in the embodiment of the present invention, two knowledge maps are pre-constructed, which are a domain ontology map and a multi-modal document map respectively; wherein the domain ontology graph stores professional and non-professional vocabulary (concepts, attributes, instances) of each designated domain based on a directed graph of a tree structure; the multi-mode document graph is an undirected graph, nodes in the multi-mode document graph are in one-to-one correspondence with search resources, each node is used for configuring a title abstract text, namely a node text for the corresponding resource to more fully describe the resource, and the similarity of the resources of every two nodes can be set by connecting edge weights among nodes of the same type (web page, text, image, audio, video and the like); when a query text input by a user is received, the embodiment of the invention firstly expands the field of the query text based on the field ontology graph to supplement more auxiliary query information for the query text, then queries the multi-mode document graph based on the expanded query text, and provides associated retrieval assistance by utilizing the node connection edge weight of the multi-mode document graph in the query process. The embodiment of the invention supplements the auxiliary query information of the search text with insufficient specialty, expands the description of the resource and assists the search of the relevance of the resource, thereby improving the search matching degree and the search efficiency.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of function in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the scope of the invention, but to limit the invention to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims

1. A processing method for retrieving multi-modal information based on a knowledge graph, the method comprising:

Constructing a domain ontology graph based on the knowledge graph structure;

Constructing a multi-mode document map based on the knowledge graph structure;

2. The processing method for retrieving multimodal information based on knowledge-graph according to claim 1, wherein,

The domain ontology graph comprises a plurality of first nodes and a plurality of first connection edges;

3. The processing method for retrieving multimodal information based on knowledge-graph according to claim 2, wherein,

The multi-mode document map comprises a plurality of second nodes and a plurality of second connecting edges;

4. The knowledge-graph-based multi-modal information retrieval processing method according to claim 2, wherein the knowledge-graph-based structure building domain ontology graph specifically comprises:

5. The knowledge-graph-based multi-modal information retrieval processing method according to claim 2, wherein the knowledge-graph-based structure constructs a multi-modal document graph, specifically comprising:

6. The knowledge-graph-based multi-modal information retrieval processing method according to claim 5, wherein the initializing the multi-modal document graph based on the first collection document set specifically includes:

7. The knowledge-graph-based multi-modal information retrieval processing method according to claim 5, wherein the updating the multi-modal document graph based on the first collection document set specifically includes:

8. The method for processing the multi-modal information based on the knowledge graph according to claim 2, wherein the performing the domain expansion on the first query text based on the domain ontology graph to obtain a corresponding first expanded query text specifically includes:

identifying whether the current retrieval node is empty;

9. The method for processing the multi-modal information retrieval based on the knowledge graph according to claim 2, wherein the multi-modal information retrieval based on the first expanded query text and the multi-modal document graph obtains a corresponding first retrieval report, specifically comprising:

10. An apparatus for performing the knowledge-graph-based multi-modal information retrieval processing method of any one of claims 1-9, the apparatus comprising: the system comprises a domain ontology graph construction module, a multi-mode document graph construction module and a multi-mode query module;

11. An electronic device, comprising: memory, processor, and transceiver;

The processor being adapted to couple with the memory, read and execute instructions in the memory to implement the method of any one of claims 1-9;

12. A computer readable storage medium storing computer instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-9.