CN105550190A - Knowledge graph-oriented cross-media retrieval system - Google Patents

Abstract

In order to meet the needs of cross-media semantic description and knowledge acquisition, and effectively utilize the cross-media attributes and various associations covered in the knowledge map, the present invention proposes to establish a cross-media attribute perception model, and to carry out natural and social attributes contained in cross-media data. Perception and association analysis, establish a unified cross-media data association description mechanism, and uniformly quantify and express different types of association relationships; propose a method of mapping different forms of data covered by knowledge graphs to the same semantic space to realize semantic knowledge Consistent expression; for the query requests expressed by users in natural language, multimedia samples, or combinations of different types of media data, it is proposed to use various associations covered in the knowledge map to perform semantic analysis on user queries to understand user retrieval intentions, so as to retrieve more A method of relevant results that meets user query requirements; a cross-media retrieval system architecture and implementation method that introduces knowledge graphs is proposed.

Description

Cross-media retrieval system for knowledge graph

technical field

The invention belongs to the category of information retrieval technology, in particular to a cross-media retrieval system oriented to knowledge graphs. The introduction of knowledge graphs in cross-media retrieval helps to obtain contextual data of various dimensions, and even further reasoning to discover features in different contexts, so as to better understand the semantics of user query content and return retrievals that better meet user needs. result.

Background technique

At present, the development and popularization of the global network has reached an unprecedented scale. People have become accustomed to searching for various information on the Internet, and search engines have become the center of the Internet. Various domestic Internet giants are sparing no effort to improve their search engines, and the national "nuclear high-tech" major science and technology project has also listed "new generation search engines and browsers" as an important development direction supported during the "Twelfth Five-Year Plan" period. However, the information on the Internet is growing exponentially, and there are various types of information. There are intricate correlations between information in various media forms. Analysis and retrieval put forward higher requirements. Since the knowledge map is introduced into the cross-media retrieval system, it helps to obtain contextual data of various dimensions, better supports users to express retrieval intentions in natural language, multimedia samples, or combinations of different types of media data, and can also be further reasoned To discover the characteristics of different situations, to achieve more accurate user query semantic analysis and retrieval. Therefore, the present invention provides an implementation scheme of a cross-media retrieval system from the perspective of a knowledge map.

The knowledge graph was developed after Google acquired the open database company Metaweb in 2010. Metaweb was mainly focused on connecting different textual expressions with the same entity, and exploring the attributes of these entities (such as the age of stars) and the relationship between each other, and finally provided a new form of search. Although it cannot completely replace keyword search, Metaweb's index and search methods are more efficient in processing natural language queries. Similarly, in cross-media retrieval, with the help of knowledge graphs, users' query requests can be better understood and content related to the semantics of query requirements can be summarized, so as to find out more accurate and in-depth relevant information for users. In addition, the knowledge graph will also help users understand the relationship between things. When a user expresses a query request in natural language, multimedia samples, or a combination of different types of media data, such a query request may represent multiple meanings, and the knowledge graph can understand the differences and narrow the search results to the user's most the desired meaning. Furthermore, because the knowledge map builds a complete knowledge system related to search results, integrates many disciplines, and systematically displays the knowledge system related to user query semantics to users, so users may learn about certain knowledge during retrieval. A new fact or a new connection prompts it to conduct a whole new set of search queries, giving it greater depth and breadth. Therefore, introducing knowledge graphs into cross-media retrieval plays an important role in improving retrieval performance.

Therefore, the present invention takes the key technology of knowledge graph-oriented cross-media retrieval as the research object, and proposes a perceptual model of cross-media attributes, a unified quantitative expression of multiple associations, a consistent expression of cross-media knowledge, and a semantic analysis of user queries based on knowledge graphs. The method and the implementation scheme of the cross-media retrieval system oriented to knowledge graph. In the field of information retrieval, judging from the current development situation at home and abroad, facing knowledge graphs and cross-media has become an inevitable trend, so the present invention has very great practical application value and broad application prospects.

Contents of the invention

The purpose of the present invention is to provide a cross-media information retrieval tool, which introduces a knowledge graph into cross-media retrieval, performs semantic analysis and reasoning based on cross-media semantic associations and knowledge covered in the knowledge graph, and realizes cross-media retrieval. Specifically, the content of the present invention includes the following points.

(1) Aiming at the intricate cross-media data on the Internet, establish a cross-media attribute perception model and analyze the association relationships covered in it, and propose a unified cross-media data association description mechanism. Obtain the natural and social attributes of cross-media data through text analysis, entity extraction, metadata analysis, semantic annotation and user behavior analysis, and then perform association modeling on the complex relationship between natural and social attributes in cross-media data In the modeling process, various associations such as content association (same modality), semantic association (different modality), timing association, and structural association among cross-media data are considered. According to the links between the web pages where multimedia objects are located, based on The probabilistic graphical model performs probabilistic modeling and analysis on cross-media content and links, so as to uniformly quantify and express different types of associations.

(2) In order to meet the needs of cross-media semantic description and knowledge acquisition, a method of mapping different forms of data to the same semantic label space is proposed to achieve semantic consistency expression. When heterogeneous and complementary media forms such as text and images jointly express a semantic, by learning a certain mapping relationship, these heterogeneous modal information can be mapped to a semantic label space, so that heterogeneous data can be directly analyzed under an expression framework. Carry out similarity measurement, and establish an evaluation function based on semantic similarity, semantic coverage and semantic differentiation, evaluate the selectivity of semantic labels, use semantic label information to train classifiers for each form, and classify the results As a shared feature, data of different forms can also be mapped to the same semantic label space, so as to achieve semantic consistency expression.

(3) Propose a method for semantic analysis and reasoning based on associations covered by knowledge graphs when users express query requests in natural language, multimedia samples, or combinations of different types of media data. For the query content entered by the user, the content of the text and multimedia query is separately and jointly analyzed, and the user query intention is analyzed from the semantic level. Therefore, firstly enough cross-media information is collected from the Internet and semantic models are established for data of different media types to realize the feature description of cross-media data in the same semantic space. Then, the semantic distribution analysis of image data and text data is integrated, and the semantics of user queries are identified, and further associated semantic mining is carried out in combination with knowledge graphs. Based on the data semantic association, temporal association, and structural association covered by the knowledge graph, contextual data in various dimensions related to user query content are obtained, and features in different contexts are discovered through reasoning, so as to obtain more complete query semantics.

(4) Propose the architecture and implementation method of cross-media retrieval system that introduces knowledge graph. In addition to the basic components such as user query analysis, indexing, retrieval and sorting, the system also needs to create a knowledge map knowledge base with a certain scale and integrate it into the system. In the user query analysis section, it supports user-input query content in the form of natural language, cross-media samples, and data of different media types. When performing query semantic analysis, in addition to semantic analysis of various media types data input by users, joint semantic analysis and further reasoning should be carried out in combination with the knowledge graph, so that the time, place, and entity on the knowledge graph Situational knowledge such as social relations and social relations can better understand user query intentions. In the part of cross-media hash indexing and sorting, some existing algorithms are mainly called.

Description of drawings

Figure 1 shows cross-media attribute perception and association analysis;

Figure 2 is a semantic analysis of user queries based on knowledge graphs;

Figure 3 shows the architecture of cross-media retrieval system for knowledge graph.

detailed description

In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings.

1. Cross-media attribute perception and association analysis

The current way of knowledge dissemination is increasingly characterized by cross-media. Relevant knowledge and information of the same entity often come from multiple channels and are expressed collaboratively in various media forms, and contain various natural and social attributes. In order to make use of cross-media The associated knowledge contained in the data is used in cross-media retrieval. In the process of building a knowledge map, in addition to considering the semantic relationship between entities, the perception of cross-media attributes of entities should also be considered, and the establishment of cross-media attribute perception model and perform correlation analysis on it. In order to uniformly quantify and express different types of associations, and effectively predict potential associations, so that different associations can use each other, a unified description mechanism for cross-media data associations is established.

From multiple channels (including Weibo, WeChat, forums, news websites, professional websites, etc.), it is expressed in a variety of media forms (text, sound, image, video), and contains a variety of natural attributes (time, place, etc.) , characters, appearance information, etc.) and social attributes (such as popularity, evaluation and preference, etc.), based on the complementary information with the text accompanying information to extract the high-level semantics of other media type data, and then through text analysis, Entity extraction, metadata analysis, semantic annotation, and user behavior analysis and other technologies obtain the natural and social attributes of cross-media data, and then classify the new data through a set of support vector machine classifiers, thereby concentrating on noisy network images. Automatically extract and identify targets of the same category; or model the attention of users in the real world by analyzing the forwarding behavior of network users to cross-media data, and analyze the data content and user forwarding behavior of Weibo, WeChat, social networks, etc. , build a forwarding tree model, and use frequent subtrees to discover the repetitive and tendency rules of user behavior, so as to track and predict the attention of groups more accurately. Next, carry out association modeling on the complex relationship between natural attributes and social attributes in cross-media data, and consider the content association (same modality), semantic association (different modality), and Various associations such as temporal association and structural association, according to the links between the web pages where the multimedia objects are located, carry out probabilistic modeling and analysis of cross-media content and links based on the probabilistic graph model, so as to uniformly quantify and express different types of associations. And further realize the association prediction of cross-media data, as shown in Figure 1.

2. Consistent expression of cross-media knowledge

Since the existing knowledge representation and knowledge base resources are basically limited to a single mode, they can no longer meet the needs of cross-media semantic description and knowledge acquisition. Therefore, after covering cross-media attribute knowledge in the constructed knowledge map, it is necessary It is used in cross-media retrieval. On the basis of analyzing the semantic knowledge expression rules of single-modal data, a method of mapping different forms of data to the same semantic label space is proposed, so as to achieve semantic consistency expression. In order to expand from a single modality to the level of cross-media knowledge representation, a method for calculating and measuring the content of various media types in the knowledge graph is proposed, and the structural information of various media data is mapped to a certain space theoretically. Perform structural analysis, fusion, and inference.

After obtaining enough cross-media attribute knowledge and association relations, in order to use it in cross-media retrieval, a consistent representation mechanism for cross-media knowledge is established on different data granularities and different knowledge levels. When heterogeneous and complementary media forms such as text and images jointly express a semantic, by learning a certain mapping relationship and mapping these heterogeneous modal information to a shared subspace, the heterogeneous data similarity measure. For cross-media data that is relevant in content and semantics, a probabilistic generation model is used to transform data of different media types into a unified latent variable space for description, and the distribution of cross-media data on each latent variable is used as its semantic label. And according to the semantic similarity, semantic coverage and semantic differentiation, an evaluation function is established to evaluate the selectivity of semantic tags and establish semantic groups. Using the semantic label information of the semantic group, the same-modal data in different multimedia documents are extracted separately, and the semantic label of the group is used to train a classifier for each form, and the classification result is used as a shared feature, so that the data of different forms It can also be mapped to the same semantic label space to achieve semantic consistency expression.

The key to semantic tag selection is to calculate its semantic correlation with cross-media content, that is, the matching between semantic tags and semantic models. In order to be able to directly compare semantic tags with semantic models, semantic tags are expressed in a semantic distribution, Semantic similarity between semantic labels and semantic models is computed using KL distance. To obtain the semantic distribution {p(w|l)} of the semantic label l, {p(w|l, D)} is approximated by cross-media dataset D. In this way, the semantic similarity between the semantic label {p(w|l)} and the semantic model {p(w|θ)} can be calculated using the KL distance:

$\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; w</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

In order to ensure that the semantic tags have a high coverage of the semantic content of the cross-media data, the selected new semantic words can cover other semantic parts, rather than the content already covered by the existing semantic words. Marginal relevance achieves maximum relevance and difference semantic words:

$\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; \&lambda;S</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; )</span>\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; \text{'}</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

$\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; \text{'}</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; \text{'}</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; w</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; \text{'}</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; \text{'}</span><span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

Among them, S is the selected semantic word.

In addition, when labeling multiple semantic contents, in order to ensure that a semantic word will not have a high degree of correlation with multiple semantic contents, the distinction between different semantic contents must also be considered, that is, the degree of semantic differentiation. In this case In this case, it is necessary to adopt a semantic similarity calculation method that considers the degree of discrimination:

S'(l, θ _i ) = S(l, θ _i )-αS(l, θ _-i ) (4)

S(l, θ _-i )=-d(θ _-i ∥ l) (5) where θ _-1 represents k-1 semantic features other than the semantic feature θ ₁ , namely θ _{1,... i-1i+1,...k} , where k is the number of semantic features. The semantic similarity of cross-semantic features is calculated and sorted by S'(l, θ _i ), so that semantic words with certain coverage and differentiation can be generated for multiple semantic contents.

3. Semantic analysis of user queries based on knowledge graph

For the query content entered by the user, it is necessary to separately and jointly analyze the text and multimedia query content, and analyze the user query intention from the semantic level. Therefore, first, collect enough cross-media information from the Internet and establish semantic models for different media types of data, as shown in Figure 2: the text semantic model described by text words and the visual semantic model described by visual words; then use These two models transform both text data and image data in the document to be analyzed into the same semantic space, and describe it in the form of semantic probability distribution. After that, the semantic mapping of different media types of data is realized through semantic learning. In order to establish associations between data of different media types, and to mine the shared subspaces existing among related heterogeneous media data, for cross-media data with semantic correlation, such as image, video and other visual data related to text semantics, adopt Text data is used to learn visual semantics, describe text semantics in the form of visual words, and establish a mapping relationship between text semantics and visual semantics, so as to realize the feature description of cross-media data in the same semantic space.

After the semantic feature description of cross-media data is obtained, the semantic distribution analysis of image data and text data is integrated to identify the semantics of user queries, and further associated semantic mining is carried out in combination with knowledge graphs. Based on the data semantic association, temporal association, and structural association covered by the knowledge map, obtain contextual data in various dimensions related to user query content, such as time, location, entity and its social relationship, etc., and use reasoning to discover different situations. features, so as to obtain a more complete query semantics. Since the reasoning involves cross-media data, before reasoning, the conversion from cross-media to text mode is realized based on technologies such as image annotation and moving object action recognition in video, and formalized representation is performed, and then reasoning is realized based on text reasoning technology. In the conversion process, it is necessary to process cross-media data at the semantic level, which can be realized based on the established cross-media semantic model.

4. Cross-media retrieval system for knowledge graph

In order to realize a knowledge graph-oriented cross-media retrieval system, a cross-media retrieval system architecture that introduces knowledge graphs is firstly proposed, as shown in Figure 3. In addition to basic components such as user query analysis, indexing, retrieval, and sorting, the system also includes several parts such as cross-media attribute perception, correlation analysis, and consistency expression. First, collect enough multimedia data from the Internet, obtain the natural attributes and social attributes of the cross-media data based on the cross-media attribute perception model, and then analyze the entity object associations, semantic associations, time series associations, and structure of various media types. Correlation analysis and description. Afterwards, a knowledge graph of a certain scale is built on this basis. In order to utilize the cross-media knowledge covered in the knowledge graph, it is represented based on the proposed consistent expression framework.

In the user query analysis section, it supports user-input query content in the form of natural language, cross-media samples, and data of different media types. When performing query semantic analysis, in addition to semantic analysis of various media types data input by users, joint semantic analysis and further reasoning should be carried out in combination with the knowledge graph, so that the time, place, and entity on the knowledge graph Situational knowledge such as social relations and social relations can better understand user query intentions. In the part of cross-media hash indexing and sorting, some existing algorithms are mainly called.

Claims (5)

1. towards the cross-media retrieval system of knowledge mapping, it is characterized in that, this system covers the content of following aspect:

Across medium property perception and association analysis;

Consistance across media knowledge is expressed;

User's query semantics of knowledge based collection of illustrative plates is analyzed;

Towards cross-media retrieval system architecture and the realization of knowledge mapping.

2. system according to claim 1, is characterized in that, sets up across medium property sensor model and analyzes the incidence relation wherein contained, and what propose a kind of unification describes mechanism across media data association.Pass through text resolution, entity extracts, metadata analysis, the technology such as semantic tagger and user behavior analysis obtains across the natural quality of media data and social property, then association modeling is carried out to across the complex relationship in media data between natural quality and social property, consider across the relevance existed between media data (same mode) in modeling process, semantic association (different modalities), sequential correlation, the multiple association such as structure connection, according to the link between the webpage of multimedia object place, based on probability graph model to the modeling analysis carrying out randomization across media content and link, thus unified quantization expression is carried out to dissimilar incidence relation.

3. system according to claim 1, is characterized in that, in order to meet the needs across media semantic description and knowledge acquisition, proposes the method for the data-mapping of different shape to same semantic label space, realizes semantic consistency and expresses.Work as text, when the media modalities co expression one of the isomery complementations such as image is semantic, by learning certain mapping relations, these isomery modal informations are mapped to a semantic label space, thus directly similarity measurement is carried out to isomeric data under expressing framework at one, and according to semantic similarity, semantic coverage and semantic space calibration set up evaluation function, the alternative of semantic label is evaluated, semantic label information is utilized to be respectively each shape up exercise sorter, and using the result of classification as sharing feature, make the data of different shape also can be mapped to same semantic label space, thus realize semantic consistency expression.

4. system according to claim 1, is characterized in that, proposes when user to carry out the method for semantic analysis and reasoning to it with the association contained in conjunction with knowledge mapping during natural language, multimedia sample or dissimilar media data combination expression inquiry request.For the query contents of user's input, respectively the content of text and multimedia inquiry is carried out separately and Conjoint Analysis, carry out analyzing user queries intention from semantic level.Therefore first gather from internet and enough set up semantic model respectively across media information and for the data of different media types, realize across the feature interpretation of media data on same semantic space.Then the semantic distributional analysis of composite image data and text data and the semanteme of identification user inquiry, and carry out the semantic excavation of further association in conjunction with knowledge mapping.Knowledge based collection of illustrative plates contain data semantic association, sequential correlation and structure connection etc., obtain the context data of the various dimensions relevant to user's query contents, and find the feature under different situation by reasoning, thus obtain more perfect query semantics.

5. system according to claim 1, is characterized in that, system, except possessing the elements such as user's query analysis, index, retrieval and sequence, also will create knowledge mapping knowledge base of certain scale and be integrated in system.In user's query analysis part, support that user is with natural language, query contents across the input of the form such as media sample, different media types data.When carrying out query semantics and analyzing, except semantic analysis will be carried out respectively to the various media type data of user's input, also to carry out combination semantic analysis and further reasoning, to understand user's query intention better according to context knowledge such as the time on knowledge mapping, place, entity and social relationships thereof in conjunction with knowledge mapping to it.