JP2011248596A - Searching system and searching method for picture-containing documents - Google Patents

Abstract

PROBLEM TO BE SOLVED: To search a group of documents containing pictures with high accuracy for a desired picture, a document containing the desired picture and a picture related to the desired document.SOLUTION: A first index in which each graphic item in a document is expressed in pseudo-words obtained by quantizing feature amounts in the pertinent graphic item and terms extracted from a part to be described in the graphic item is generated. On the basis of the co-occurrence frequency of a pseudo-word and/or a term, the intensity of relevance between the pseudo-word and/or the term and the level of importance of the pseudo-word and/or the term are calculated. After that, a second index is generated in which the pseudo-word and/or the term to which the level of importance is assigned is made a weighted node and an edge to which the intensity of relevance between the pseudo-word and/or the term is assigned is made a weighted edge. A score representing the similarity between the second index generated by such preprocessing as the object of search and a processed query expressed in the weighted node and the weighted edge is calculated.

Description

  The present invention relates to a technique for retrieving a target image or a similar image from a document (for example, paper, Web) including an image (for example, a figure) using text information in the document including the image and the image.

  In recent years, the publication of various documents on the Web (Web) is increasing under the trend of opening scientific and technical papers and public information. Along with this, the importance of searching for documents including images (figures) is increasing. Among them, in a field where information is aggregated in an image and a field where information not described in a text part is likely to be included in an image, a search in which an image and text are associated is important.

  Currently, digital images that use image analysis results and medical image similarity search technology are used for searching only for images. On the other hand, for a search only for text, a document search including a term included in a query, a similar document search for searching a document similar to a document given as a query, and the like are used. In these text searches, a method is used in which an index for each word or term for which a concept is recognized is prepared in advance for each document. In addition, in a search that fuses an image and text, a method of searching for an image using text existing around the image, a method of using a feature amount of the image and text information, and the like are used. In the search for fusing images and text, there is a method of executing an image search after the text search (Non-patent Documents 1 and 2).

Kherfi, M.L., Ziou, D. and Bernardi, Learning from Negative Example in Relevance Feedback for Content-Based Image Retrieval, 16th International Conference on Pattern Recognition (ICPR'02)-(2002). Athitsos, V. and Swain, MJ. Distinguishing photograph and graphics on the World Wide Web, In Proc. Of the IEEE Workshop on Content-Based access of Image and Video Libraries (San Juan, Puerto Rico), 10-17 (1997) . T. Hoffman, Learning and representing topic.A hierarchical mixture model for word occurrence in document databases.In Workshop on learning from text and the web, CMU., 1998. Tuytelaars, T. and Gool, LC.Wide Baseline Stereo Matching based on Local, Affinely Invariant Regions, BMVC pp.412-415, 2000. Matas, J. Chum, O. Urban, M. and Pajdla, T. Robust wide baseline stereo from maximally stable extremal regions. In BMVC pp.384-393, 2002. Lowe, D. Object Recognition from Local Scale-Invariant Features.Proc. Of IEEE International Conference on Computer Vision (ICCV) pp1150-1157, 1999. Kadir, T., Zisserman, A., and Brady, M. An affine invariant salient region detector.In Proc. Of the 8th Eur. Conf. On Comp.Vision, Prague, Czech Republic, pp.345-457, 2004. Belongie, S. Malik, and J. Shape, Matching and Object.Recognizing Using Shape Contexts.IEEE Transactions on Pattern Analysis and Machine Intelligence, 24 (24), pp509-522, 2002. Lowe, D.G.Object recognition from local scale-invariant features.In Int. Conf.on Comp.Vision, Corfu, Greece, pp. 1150-1157, 1999. Ke, Y .. PCA-SIFT: A More Distinctive Representation for Local Image Descriptors. IEEE Comp. Soc. Conf. On Comp. Vision and Pattern Recognition (CVPR'04) pp.511-517, 2004. Schaffalitzky, F. and Zisserman, A., “Multi-View Matching for Unordered Image Sets,” Proc. Seventh European Conf. Computer Vision, pp. 414-431, 2002. Van Gool, L., Moons, T. and Ungureanu, D., “Affine / Photometric Invariants for Planar Intensity Patterns,” Proc. Fourth European Conf. Computer Vision, pp. 642-651, 1996. Brin, S. and Page, L. The anatomy of a large-scale hypertextual Web search engine, Computer Networks and ISDN Systems, 30 (1-7) 1998. Stephen E. Robertson, Steve Walker, and Micheline Hancock-Beaulieu. Okapi at TREC-7. In Proceedings of the Seventh Text REtrieval Conference. Gaithersburg, USA, November 1998.

  There is a case where it is desired to search for an image close to an image included in a document as a target image. In this case, if only the image information included in the document is used, the context around the image is ignored. For this reason, even if the images themselves are similar, an image having a different meaning may be searched. Still, if the number of images to be searched is small, the searched images are less likely to be a problem. However, when the number of images to be searched increases, finding the target image itself becomes difficult. On the other hand, when only the text information around the image is used, it is difficult to obtain a target image even if the context around the image is close to the intended image.

  In addition, it is conceivable to apply a method using both image information and text information. In this case, a text search is executed on the explanatory text located around the figure, and an image search is executed using the search result. However, with this method, the text search may end inappropriately due to problems such as synonyms. Also, this method may hit terms with different meanings due to problems such as ambiguity of terms. In these cases, the result of text search may not include an image suitable for image search. In this case, even if both image information and text information are used, sufficient search accuracy cannot be ensured.

  As a method for simultaneously handling image information and text information, a method is considered in which a feature amount is extracted from an image, quantized, and indexed in the same way as terms constituting a text, and a target image is searched together with explanatory text around the image. It is done. However, with this method, the feature quantity of the quantized image is handled in the same row as the word in the case of text search, so the case where the feature of the image appears for the first time by a set of multiple quantized feature quantities is considered. Can not do it. That is, the combined effect of quantized feature values cannot be sufficiently considered. Therefore, in this case as well, the similarity of images may not be searched with sufficient accuracy.

  Further, as a method for simultaneously handling image information and text information, a method of calculating the text and the image by latent semantic indexing (LSI) and fusing the text information and the image information can be considered. This type of method computes consistency across documents. For this reason, it is difficult to represent the relevance of individual images at the similarity level with LSI.

  In addition, as a method of associating text with the term itself, there is also a method such as multi-modal Hierarchical Aspect Models in which the text term added to the figure itself becomes an uncommon vocabulary (Non-Patent Literature). 3). However, although this method can see the relationship between text and diagrams, it cannot reveal the similarity between multiple diagrams.

  As described above, when the feature quantity extracted from each image is quantized into a pseudo word and searched in the same manner as text information, it is sufficient to simply search for images and documents as in the case of text information only. Cannot search for similar images with high search accuracy.

  Therefore, the inventor generates a network (second index) composed of nodes weighted by the importance of the term / pseudoword and edges weighted by the relationship between the term / pseudoword, We propose a method for searching diagrams or documents using the weights of nodes and edges of the network.

  In addition, the present inventor, when given a query, expands the query using edge strength and subgraphs, and uses not only nodes but also edge weights in scoring the target image or document Propose. When this query is expanded, the weights of nodes and edges derived from the weights derived from the original query terms and pseudo words are calculated. That is, when a query is expanded, the weight of the node or edge given from the beginning is not used, but the product of the ratio propagated (transitioned) from the original query and the weight given from the beginning is used. Thereby, the effect of the several combination between a term and a pseudo word can be calculated appropriately.

  As described above, in the present invention, when expressing the similarity of an image, each pseudo word constituting the image and each constituent term of the image description are not handled independently to calculate the similarity, Evaluate the similarity considering the combination of terms. For this reason, a search with higher accuracy can be realized.

  In addition, when generating the network (second index) described above, by solving the eigenvalue problem of the transition probability matrix expressing the strength of the relationship between the nodes (for example, terms / pseudowords), the weight of the node and the edge It is desirable to determine the strength. Each transition probability can be calculated using, for example, a co-occurrence frequency in a document / image between nodes (for example, using tf (word appearance frequency) * idf (reverse appearance frequency)). Further, when calculating the strength of the relationship between the pseudo words, the accuracy of the similarity search can be further improved by considering the physical distance between the pseudo words in the image together with the appearance frequency.

  The present invention is based not only on the importance of the pseudo word itself calculated based on the terms and / or image features of the document but also on the basis of the strength of the relationship between these terms and / or pseudo words. Document search, similar image search, and image search with explanatory text are performed. In addition, the present invention expands a query before executing a search operation based on not only the importance of the node but also the strength of the relationship between the nodes as necessary.

  According to the present invention, it is possible to evaluate not only the importance of terms and pseudo-words but also the degree of similarity of images under a condition that combines them with edge strength. In particular, in the image search, if the node information is used alone, the features of the figure cannot be expressed sufficiently. For this reason, the search accuracy can be improved as compared with the prior art by executing the search process under a condition in which the edge weight is combined with the edge weight.

  Further, in the case of expanding a query in the present invention, the structure of the expanded query between nodes is changed by changing the weight of the extended node and edge in consideration of propagation from the original query. Can be reflected.

  In the present invention, when a transition probability matrix expressing the strength of the relationship between nodes is solved as an eigenvalue problem, the importance of terms and pseudo words can be calculated under common conditions. That is, the search accuracy can be improved as compared with the case where relative weights are used.

The figure which shows the conceptual structure of a search system. The figure explaining the outline | summary of the search operation | movement performed with a search system. (a)-(e) is a figure explaining the example which produces a pseudo word from an image. (a) is a diagram showing an original image. (b) is a figure which shows the feature point detected from the image. (c) is a figure which shows the result of the feature-value calculated from the feature point. (d) is a diagram showing the result of clustering and quantizing the feature values into pseudo words. (e) is a figure which shows the example of expression by the pseudo word of an image. (a) And (b) is a figure which shows an example of an index. FIG. 5A is a diagram illustrating an example of an index of an image and an example of an index for an explanatory text of the image. (b) is a diagram showing an example of a document index. The figure which shows the importance of each term and pseudoword, and the figure which shows the strength of the relationship between pseudowords, between terms, and a pseudoword-term. The figure of the interface of an image and document search system. (a) is a figure which shows the example of a query screen. (b) is a diagram showing an example of a search result screen. The figure of the interface of an image and document search system. (c) is a figure which shows the example of a display screen of a search result. FIG. 6D is a diagram showing an example of a screen in which feature amounts are displayed in a search result diagram. (e) is a diagram showing search results displayed on a network composed of weighted nodes and weighted edges. (f) is a figure which shows the example of a screen used when a user searches using a part of figure from a search result. The figure which shows the relationship between the pseudo word and term which comprise a certain document and image group. (a) is a figure which shows the example of the directed graph which has a transition probability in an edge. (b) is a figure which shows the result of having calculated each importance of the pseudo word and the term which comprise a node by solving a transition probability matrix. The figure which shows the example of a document index corresponding to the document and image group of FIG. The figure which shows the example of an image index corresponding to the document and image group of FIG. The figure which shows the index example of the weight of the node (a term and a pseudo word) on a network corresponding to the document and image group of FIG. 7, and the weight of an edge. The figure explaining the example of a search of a document and a pseudo document (a figure) when query expansion is not performed when a query is given by Term B, Term C, and Term D. (a) is a diagram showing a search result of a document / pseudo-document (figure) when a query is given in Term B and one-stage query expansion is performed. (b) is a diagram showing a search result of a document / pseudo-document (figure) when a query is given by term B and two-stage query expansion is performed. (a) is a figure explaining the example which performs one step of query expansion only regarding a figure, when a query is given by term B. FIG. FIG. 6B is a diagram for explaining an example in which one-stage query expansion is performed only for terms when a query is given by term B. (a) is a figure explaining the example which does not perform query expansion, when a query is F1, F4, and F8. FIG. 6B is a diagram for explaining an example in which one-stage query expansion is performed only with respect to the figure when the queries are F1, F4, and F8. (a) is a figure explaining the example which does not perform a query expansion, when a query is given with the combination of term A and F1, F3, F8. FIG. 6B is a diagram for explaining an example in which one-stage query expansion is performed with respect to the figure and terms when the query is given by a combination of term A and F1, F3, and F8. The figure explaining the similarity of a figure. (a) is the figure which highlighted and displayed the feature-value which quantized a certain figure A black. FIG. 6B is a diagram in which a feature amount obtained by quantizing FIG. B is highlighted in black. (a) is the figure which displayed the relationship between pseudo words by solving the eigenvalue problem of the transition probability matrix of a certain figure A. FIG. (b) is a diagram showing the relationship between pseudo-words by solving the eigenvalue problem of the transition probability matrix of FIG. The figure explaining the example which performs one step of query expansion regarding a figure and a term when a query is given by term B.

  Embodiments of the present invention will be described below with reference to the drawings. Note that the contents of the apparatus configuration and processing operation described later are examples for explaining the invention. In the present invention, an invention that combines a device configuration and processing operation described later, an invention that combines a device configuration and processing operation described later with a known technology, and a part of the device configuration and processing operation described later are replaced with a known technology. Inventions are also included.

(Configuration of search system)
FIG. 1 shows a configuration example of a search system 10 that searches images, terms, and documents. The search system 10 includes a storage 20 and a computer 30. The storage 20 stores a document including an image, a text document, and an image as digital data. The computer 30 searches the storage 20 based on a query given through an input device (not shown) and presents the search result to the user. The computer 30 has a so-called computer configuration. That is, the computer 30 includes an arithmetic device, a storage device, an input / output device, and a control device. Note that an input device (for example, a mouse, a keyboard, a pointing device) and an output device (for example, a display device) (not shown) are connected to the computer 30.

  Signal processing related to the search operation is realized through a processing function of a program executed on the computer. However, it is not necessary for all signal processing to be executed on one computer. That is, signal processing to be described later may be executed in cooperation with a plurality of computers. Further, a part of signal processing described later may be executed on hardware.

  Further, the storage 20 constituting the search system 10 is not necessarily limited to one, and may be composed of a plurality of storages 20. The connection between the storage 20 and the computer 30 is generally assumed to be a connection via a network. The network may be an intranet or the Internet.

(Basic operation of the search system)
FIG. 2 shows an outline of the search operation executed by the search system 10. The search operation uses preprocessing 100, query input processing 108, preprocessing 109 for extracting information suitable for search from the input query, and extracted information (called processed query) 110. A search process 107 and a process 111 for presenting a search result (solution) are configured. In the search process 107, the processed query may be expanded as necessary.

The detailed contents of each process realized through the search program will be described below.
(Preprocessing)
First, the contents of the preprocessing 100 will be described. In the preprocessing 100, processing for extracting a feature amount from each of an image and a document included in the image-containing document DB 101 and processing for creating a search index from the extracted feature amount are executed. The image-containing document DB 101 here is constructed on the storage 20.

  First, in the preprocessing 100, the program divides the document in the image-containing document DB 101 into an image and an explanatory text thereof. Next, the program extracts feature amounts from the image and quantizes the extracted feature amounts. By the quantization, the feature amount is converted into a pseudo word (103). Intensity based regions (Non-patent document 4), Maximum stable extremal Regions detector (MSER) (Non-patent document 5), Hessian-affine (Non-patent document 6), Entropy-based regions (salient regions) (Non-Patent Document 7) and other various methods can be used. For calculation of the feature amount of the detected feature point, shape context (Non-patent document 8), Gradient orientation histogram-SIFT (Lowe'99) (Non-patent document 9), PCA-SIFT (Non-patent document 10), Complex filters (Non-patent document 11), Moment invariants (non-patent document 12), and other various methods can be used. Which method is optimal depends on the quality of the image group. For this reason, it is necessary to select a method suitable for the image group. Methods for quantizing the feature values obtained by these methods include k-means clustering. It is necessary to test the number of clusters so that the quantization value is suitable for the similarity search.

  FIGS. 3A to 3E show processing images from extraction of feature amounts of an image (FIG.) To quantization of feature amounts. Here, (a) of FIG. 3 is the original image (figure) 201. FIG. 3B is a diagram illustrating a state 202 in which feature points (indicated by an ellipse) are detected from the image 201. (C) of FIG. 3 is a figure which shows an example of the result 203 which computed the feature-value from the feature point. FIG. 3D is a diagram illustrating a state 204 in which each feature amount is clustered and quantized and converted into a multidimensional feature amount (pseudoword). Here, CL * (* is 1, 2,... N) indicates a pseudo word associated with the feature point. FIG. 3E shows a state 205 in which an image is represented as a set of quantized feature amounts (pseudowords).

  On the other hand, the program recognizes the term in the sentence and extracts the term from the description of the image (102). In this embodiment, the description of the image is defined as a sentence laid out around the image. The surrounding range is appropriately set in consideration of the accuracy of the search result. Here, it is desirable to recognize synonyms as the same concept by using dictionary data or the like. It is also desirable to determine the context and resolve terminology ambiguity as necessary.

  Next, the program creates an index by combining the pseudo word and the term describing the figure for each image (104). That is, pseudo-documentation is created. Examples of index creation are shown in FIGS. 4A shows an example of an image index 300, and FIG. 4B shows an example of a document index 303. The image index 300 includes a frequency set 301 of terms (Term) related to each image and a frequency set 302 of feature points extracted from the image. On the other hand, the document index 303 includes a set 304 of terms (Term) related to each document and a set 305 of frequency of feature points extracted from images related to the document. The index here corresponds to the first index in the claims.

  Thereafter, the program calculates the importance of the nodes (terms and pseudo words) and also calculates the strength of the relationship between the edges (between terms, between pseudo words, between terms and pseudo words) (105). . That is, a network including weighted nodes and strength edges is generated. After this, the program creates a search index that combines the calculated weighted nodes (pseudo-word and term importance) with the strength edges (edges with information on the strength of the relationship between the nodes). (106). The index here corresponds to the second index in the claims.

  FIG. 5 shows an example of an index composed of combinations of node weights 401 and edge weights 402 constituting the network. The importance of giving the node weight 401 and the edge weight 402 is generally different for each field. In other words, the numerical value of the weight is different for each field. Therefore, it is desirable to calculate the node weight 401 and the edge weight 402 for each field. In addition, for each document, the terms constituting the document and the network of the figure (index) may be calculated, and the degree of similarity (score) between the network of the document and the network of the query may be calculated. That is, the similarity between the document and the query may be searched.

  There are several methods for generating the network described above. An example is shown below. For example, it can be generated by creating a transition probability matrix between terms and pseudo-words and solving the eigenvalue problem. In this case, pseudo words and terms are treated as network nodes, and node relationships are treated as network edges. In addition, the importance of a pseudo word or term is treated as a node weight, and the strength of the relationship between the pseudo word and the term is treated as an edge weight. Here, the edge weight is defined as the product of the edge and the transition probability.

For example, the transition probability matrix T between nodes is a document retrieval system score S _ij (where i and j are terms or pseudowords) such as tf * idf and co-occurrence frequency starting from a certain term or pseudoword. Is defined as a transposed matrix of a matrix having the following equation (Equation 1) as an element.

Here, P _ij (where i and j are terms or pseudowords) is a transition probability of the edge E _ij (the edge from the node i to the node j). P _ij satisfies the relationship of the following equation (Equation 2).

  The computer solves the eigenvalue problem (Equation 3) of the transition probability matrix T by a known calculation process.

Here, λ is an eigenvalue, and V is an eigenvector. The weight of the i-th node is the i-th element of the eigenvector V. When normalization is performed, the node weight is the sum of the i-th element * 1 / eigenvector element of the eigenvector V. The strength of the edge E _ij is defined as the product of the node weight and the transition probability P _ij .

FIG. 7A shows an example in which the relationship between the elements constituting the transition probability matrix T is expressed in a network format represented by directed edges. In the figure, 521 is a node and 522 is an edge with a transition probability. FIG. 7B shows an example in which the weight W _i of the node i and the weight T _ij of the edge E _ij are mapped on the network. The weight W _i is calculated by solving the eigenvalue problem of the transition probability matrix T. In the figure, 523 is a weighted node, and 524 is an edge with a strong relationship.

If multiple eigenvalues exist and the eigenvector cannot be determined uniquely, for example, as shown in the following equation (Equation 4), the graph is strongly connected by adding 1 / M as the default value of P _ij And the eigenvector is determined uniquely.

Here, as the elements of the transition probability matrix T, scores of S _ij (= tf * idf), Okapi BM25 (Non-patent Document 14), hypergeometric distribution, and the like can be used. Note that tf giving the score of S _ij is the appearance frequency of the j-th term or pseudo word in the partial document set when the partial document set is obtained using the i-th term or pseudo word as a query. On the other hand, idf is calculated from log ₁₀ (the number of documents or pseudo documents appearing in the database / the number of documents / pseudo documents in which the j-th term or pseudo word appears).

  As described above, instead of solving the transition probability matrix T for the eigenvalues and eigenvectors, the importance of the node and the strength of the edge may be calculated by solving the following recurrence formula (Formula 5 or Formula 6). (Non-patent document 13).

In the above equation, I (N _i ) is the importance of the i-th node. Out (N _j ) indicates all sets of the k-th node that transitions from the j-th node to the k-th node. In (N _i ) indicates all sets of j-th nodes that transition from the j-th node to the i-th node. Furthermore, d is a constant and takes a value of about 0.8 to 0.99. As described above, score values such as tf * idf and Okapi BM25 can be used for the score S _ij . Further, the edge weight T and the node weight W may be determined using other methods. It is desirable that the inflow amount and the outflow amount from the edge with respect to a certain node are the same. That is, it is desirable that the following equation (Equation 7) holds.

(Operation screen)
Next, processing of the program relating to query input and search result display will be described. The screen described later corresponds to the operation screen displayed in the query input process 108 and the process 111 for presenting the search result (solution).

  In the present embodiment, the program uses text, a local figure, or a document as a query. Further, the program searches for the accompanying text information and the compound document of the figure, only the document, or only the figure. FIG. 6A shows a display example of the screen 501 of the search system used when inputting a query. The screen 501 includes a text query input field 502, a diagram query input field 503, a document query input field 504, a search target selection field 505, a search start button 506, and an upload button 506A. Among these, the document query corresponding to the input field 504 is a query that uses the combined text information and the figure composite document as a query. A document query can be input by inputting a document ID or by uploading a corresponding document. A circle in the search target selection field 505 is a button indicating selection of a search target and a selection state. When the search target is selected, for example, the circle mark changes to the circle mark. The upload button 506A is a button for taking in a diagram or a document (figure and description) stored in the computer 30 as a query.

  Next, processing of the program relating to presentation of a solution (that is, display of search results for a query) will be described. If necessary, a figure or document displayed as a search result, which is close to the target figure or document, can be selected as a query, and the newly set query can be searched again. FIG. 6B shows a display example of the search result display screen 507. The search result display screen 507 inputs a figure display column 509 searched for high similarity, a text display column 510 related to the figure, and a figure and text displayed in each display column to the program as a new query. And a search start button 511 for instructing execution of a search (relevance feedback) using a query newly specified in the screen. In the search result display screen 507, a detailed display button 512 for displaying the network structure characterizing the text associated with the figure, and a part of each figure displayed in the display field 509 are designated as a new query. A partial image selection button 513 is also displayed.

  FIG. 6B shows the relationship between the image contents and buttons displayed on the search result display screen 507. FIG. 6B is an example of a search result when the search target is “FIG. + Description”. FIG. 6C is the same diagram as FIG. 6B. FIG. 6D is a display example of the feature amount additionally displayed in the searched diagram when the detail display button 512 is operated. FIG. 6E is an example of a network structure displayed when the detail display button 512 is operated. The network structure shows the relationship between the corresponding diagram, document or diagram and the weighted nodes and weighted edges of the document. In the case of FIG. 6B, the nodes and edges where the query is hit are indicated by a combination of coloring and bold lines. By this screen display, it is possible to confirm on the network the node used for the search and the relationship between the nodes. This display can also be used as a reference for selecting relevance feedback. It can also be used as a reference for what queries should be included first. FIG. 6B is an example of a screen displayed when the partial image selection button 513 is operated. In the figure, a portion surrounded by a thick line indicates an image region used as a new query. When the search start button 511 is operated in (f) of FIG. 6B, a search using the interactively selected partial image as a new query is executed.

  In FIG. 6A, the partial image selection button 513 is not shown, but the partial image selection button 513 may be displayed on the screen for inputting the first query. In addition, when inputting a query, (d) and (e) in FIG. 6-2 may be displayed as the intermediate screens in FIGS. 6-1 (a) and (b). .

(Pre-search process and search process)
Next, processing of the program after the query is given will be described. If there is no query expansion, the program searches for the target document / image using the weight of the query node in the document / image and the weight between the query nodes. On the other hand, if there is query expansion, the program expands the query along the direction of the graph in the case of a directed graph, and in the case of an undirected graph the query is propagated to adjacent edges regardless of the direction of the graph. Expand. These processes correspond to the processing contents from the preprocessing 109 to the processing 111 for the query.

  In the search, not only the weight of the node but also the weight of the edge is used, and the document / image to be searched is scored.

  Note that propagation (transition) from the weight of the original query is also used when expanding the query. For this reason, in the expanded query, not only the numerical values of individual nodes and edges themselves, but also the weights of nodes and edges obtained by multiplying these values by propagation (transition) probabilities are used.

  For example, when there is no query expansion, the score of each document and pseudo document (image) is given by the following equation (Equation 8).

  On the other hand, when there is one-stage query expansion, the score of each document and pseudo document (image) is given by the following equation (Equation 9). Note that the one-stage query expansion means that the constituent elements of the query are extended to edges and nodes having a connection relationship on the network with respect to each element constituting the original query.

In Expressions 8 and 9, I _j represents the weight of the node j, E _jk represents the weight of the edge between the nodes j and k, and freq (t _k ) represents the node k (k-th term or pseudo word). The frequency of appearance in the document and / or pseudo document is indicated, and P _ij indicates the transition probability between the node i and the node j. Similarly, when the query is expanded in two stages, the product of transition probabilities from the original query is calculated for each weight of the original edge and the node. For the extended edge and the extended node in the second stage, values obtained by multiplying the transition probabilities of the two edges extending from the original query are used. The scoring method is not limited to the content of Equation 9. Moreover, how to take Min and Max can be changed as appropriate.

  Note that, before the actual search process is executed, the query feature amount is extracted for search in the query preprocess 109. For example, when the query is an image (including the image portion of the document), the program executes an image analysis process, quantizes the feature amount of the image, and converts it into a pseudo word (103). On the other hand, if the query is text (including the text portion of the document), the program retrieves the terms that make up the text (102). In this way, the processed query 110 for search processing is generated based on the pseudo-worded or extracted terms. The program uses the processed query 110 or a query expanded as necessary to execute a search process 107 for searching for a document and / or a pseudo document (figure), and displays the search result (solution) on the screen. Present.

Example 1
Subsequently, an embodiment of an index used in the above-described search system will be described. FIG. 8A shows an example of an index of a document including an image. In FIG. 8A, F ₁ ,..., F _N indicate quantized features extracted by image analysis of images in a document. Terminology is indicated by TermA, TermB,. Each pseudo word or term is associated with an appearance frequency that appears in an image or drawing. It includes a plurality of images on the document 1, and, if the plurality of images includes a pseudo word F ₁ ... .F _N, the frequency of occurrence of F ₁ ... .F _N is the sum of the frequency of appearance of each image Given. FIG. 8B illustrates an example in which the appearance frequency of the result (pseudoword) obtained by quantizing the feature amount of each image is indexed. In the figure, reference numeral 601 denotes a table showing the relationship between the nodes of terms / pseudo words related to images and the appearance frequency. Reference numeral 602 denotes a table showing the relationship between the transition probability between terms / pseudowords related to an image and the appearance frequency. FIG. 8-3 shows the relationship between the weight of a term or pseudo word, the strength of the relationship between nodes, and the transition probability between nodes in a certain image-containing document group. In the figure, reference numeral 603 denotes the relationship between the term or pseudo word and the weight. Reference numeral 604 denotes a relationship between the strength of the relationship between the nodes and the transition probability. The values of 603 and 604 are calculated as a solution to the eigenvalue problem of the transition probability matrix T or by another method and accumulated on the storage.

(Example 2)
FIG. 9 shows a network structure of an image-containing document set having the weight structure shown in FIG. Here, a case is considered where the scores of the image having the index shown in FIG. 8-2 and the query composed of Term B, Term C, and Term D are calculated. When there is no query expansion, edges exist between Term B and Term C and between Term C and Term D, as shown in FIG. For this reason, the score of the document with an image can be calculated by, for example, the following formula (Formula 10).

(Example 3)
FIG. 10A is a diagram showing a state in which a query is expanded by one step for Term B under the same conditions as in the second embodiment. In this case, the score of the document with an image can be calculated by, for example, the following formula (Formula 11).

Example 4
FIG. 10B is a diagram illustrating a state in which a query is expanded by two stages for Term B under the same conditions as in the second embodiment. The edge of the extended portion in the second stage is indicated by a broken line. In this case, the score of the document with an image can be calculated by the following formula (Formula 12), for example.

(Example 5)
FIG. 11A is a diagram showing a state where the query (Term B) is expanded by one step only with respect to the diagram under the same conditions as in the second embodiment. In the figure, dark gray nodes and thick edges are subject to scoring calculation in the search.

(Example 6)
FIG. 11B is a diagram illustrating a state in which the query (Term B) is expanded by one step only for terms under the same conditions as in the second embodiment. In the figure, dark gray nodes and thick edges are subject to scoring calculation in the search.

(Example 7)
FIG. 12A shows an example in which the figure is a query under the same conditions as in the second embodiment. In the example shown in the figure, the pseudo-words F1, F4, and F8 shown in FIG. In the figure, dark gray nodes and thick edges are subject to scoring calculation in the search.

(Example 8)
FIG. 12B is an example in which the figure is a query under the same conditions as in the second embodiment. The configuration pseudo-words F1, F4, and F8 in the figure are the same as the query in the seventh embodiment. However, in this embodiment, an example is shown in which the query is expanded from each node by one step only with respect to the diagram. In the figure, dark gray nodes and thick edges are subject to scoring calculation in the search.

Example 9
FIG. 13A shows a case where the query is given as “figure + description” under the same conditions as in the second embodiment. Specifically, the query is given as a combination of term A and F1, F3, F8. In some cases, the query is not expanded. In the figure, dark gray nodes and thick edges are subject to scoring calculation in the search.

(Example 10)
FIG. 13B shows a case where the query is given as “figure + description” under the same conditions as in the second embodiment, specifically, when the query is given as term A and F1, F3, F8. This is an example when the query is expanded by one step. In the figure, dark gray nodes and thick edges are subject to scoring calculation in the search.

(Example 11)
(A) and (b) of FIG. 14 show an example in which the relationship between the feature quantity and the feature quantity included in a certain figure is highlighted in black. A portion having a thick edge is a portion common to (a) and (b) of FIG. In this way, a partial similarity search is possible by looking at the degree of matching of the subgraphs. Such subgraphs can also be used in the query when searching.

  In addition, in the case of calculation using only such a diagram, only the relationship between the pseudo-words ij existing within a certain distance is calculated in the calculation of the original score Sij when the transition probability matrix is calculated first. It is good also as a calculation object. That is, an edge may not be stretched between nodes (pseudo words) exceeding a certain distance.

(Example 12)
FIGS. 15A and 15B show examples in which the network is generated by solving the eigenvalue problem of Equation 3 for each image. 15A and 15B can be generated by analyzing all images and indexing the weights of nodes and edges as shown in FIG. For example, it is assumed that a query is given as a whole in FIG. In this case, the common part of (a) and (b) (the node of the dark gray node and the thick edge in the figure) is the scoring calculation target when (b) is the search target (processing target).

(Example 13)
FIG. 16 shows an example in which the query (term B) is expanded by one step under the same conditions as in the third embodiment. The network of FIG. 16 is an undirected graph. Therefore, the query is expanded by propagating to all edges adjacent to term B regardless of the direction of the edge. In the figure, dark gray nodes and thick edges are subject to scoring calculation in the search.

(Summary)
As described above, in the case of the search system and method according to the embodiment, the search operation is executed by combining node weight (importance) and edge strength (strength relationship between nodes). For this reason, compared with the case where an image is searched only by the independent importance of a term or a pseudo word, the similarity of searched images can be increased.

  Also, when expanding a query, the weights of the nodes and edges that make up the query are recalculated based on the derivation relationship with the terms and pseudo-words that make up the original query. A search operation that accurately reflects the network structure becomes possible.

  Also, by solving the transition probability matrix as an eigenvalue problem or solving the relationship between nodes and edges as a recurrence formula, the importance between terms and pseudo-words can be calculated under common conditions. For this reason, the accuracy of the weights of the nodes and edges can be increased as compared with the prior art as compared with the case where relative weights are used.

10 Search system
20 Storage
30 calculator
100 Pretreatment
101 Document DB with images
107 Search processing
108 Query input processing
109 Preprocessing for queries
110 Processed queries
201 Original image (image A)
202 Image showing the result of recognizing feature points in image A
203 Example of calculation result of feature points of image A
204 Example showing the result after quantizing the feature points of image A
205 Example of image A using quantized feature points
300 image index
301 Term frequency index
302 Index of appearance frequency of quantized feature of image
303 Document Index
304 Term frequency index
305 Index of appearance frequency of quantized feature of image
401 Term frequency index
402 Index of term relationship strength
501 Search system screen
502 Text query input field
503 Figure Query Input Field
504 Document query input field
505 Search target selection field
506 Search start button
506A Upload button
507 Search result display screen
508 Select button
509 Figure display field
510 Text display field
511 Search start button
512 Detail display button
513 Partial image selection button
521 nodes
522 Transition probability
523 Weighted node
524 Edge showing the strength of the relationship

Claims (16)

For each figure in the document, a functional unit that generates a first index expressed by a pseudo word obtained by quantizing the feature amount of the figure and a term extracted from the explanation part of the figure,
A functional unit that calculates the strength of the relationship between the pseudo word and / or the term and the importance of the pseudo word and / or the term based on the co-occurrence frequency of the pseudo word and / or the term;
A function unit that generates the second index using the pseudo-word and / or the term with the importance as a weighted node and the edge with the strength of the relationship between the pseudo-word and / or the term as the weighted edge; ,
A function unit that generates a processed query including weighted nodes and weighted edges from the search query;
A function unit for searching a figure and / or a document similar to a search query by calculating a score representing a similarity between the second index composed of a weighted node and a weighted edge and the processed query; A search system comprising: The search system according to claim 1,
By solving the eigenvector of the transition probability matrix calculated based on the co-occurrence frequency between the pseudo-words and / or terms, the strength of the relationship between the pseudo-words and / or terms and the importance of the terms and / or pseudo-words A search system characterized by calculating The search system according to claim 1,
A transition system is defined for the weighted edge, and a weight inflow amount and an outflow amount from the edge of each node are the same. The search system according to claim 1,
The search system further comprising a function unit for extending the search query to adjacent nodes and edges on the network corresponding to the second index. The search system according to claim 4, wherein
In the search query after extension, a value obtained by multiplying the weight before extension by the propagation probability from the search query of extension source is used as the weight of the extended node and edge. The search system according to claim 2,
The transition probability matrix is created according to the field of documents to be searched. The search system according to claim 1,
A search system further comprising: a function unit for displaying a network corresponding to the second index on a user interface screen and displaying a feature point corresponding to the second index in a corresponding figure. . The search system according to claim 1,
The network configuration of the second index corresponding to the searched figure is displayed on the user interface screen, and the function unit further highlights the node and the edge for which the score is calculated on the user interface screen. Feature search system. The search system according to claim 1,
A search system, further comprising: a function unit that accepts, as a new search query, an area portion selected by a user in a diagram displayed on a user interface screen as a search result. For each figure in the document, a process for generating a first index expressed by a pseudo word obtained by quantizing the feature quantity of the figure and a term extracted from the explanation part of the figure,
Processing to calculate the strength of the relationship between the pseudo word and / or term and the importance of the pseudo word and / or term based on the co-occurrence frequency of the pseudo word and / or term;
A process of generating a second index using the pseudo-word and / or the term with the importance as a weighted node and the edge with the strength of the relationship between the pseudo-word and / or the term as a weighted edge;
Generating a processed query composed of weighted nodes and weighted edges from the search query;
Processing for searching a figure and / or document similar to a search query by calculating a score representing the similarity between the second index composed of weighted nodes and weighted edges and the processed query. A search method characterized by comprising: The search method according to claim 10,
By solving the eigenvector of the transition probability matrix calculated based on the co-occurrence frequency between the pseudo-words and / or terms, the strength of the relationship between the pseudo-words and / or terms and the importance of the terms and / or pseudo-words A search method characterized by calculating The search method according to claim 10,
The search method further comprising a process of expanding the search query to adjacent nodes and edges on the network corresponding to the second index. The search method according to claim 12, wherein
In the search query after extension, a value obtained by multiplying the weight before extension by the propagation probability from the search query of extension source is used as the weight of the extended node and edge. The search method according to claim 10,
A search method, further comprising: displaying a network corresponding to the second index on a user interface screen, and displaying a feature point corresponding to the second index superimposed on a corresponding diagram. The search method according to claim 10,
The network configuration of the second index corresponding to the searched figure is displayed on the user interface screen, and the processing further includes highlighting the node and the edge for which the score is calculated on the user interface screen. Search method. The search method according to claim 10,
A search method characterized by further comprising a process of accepting, as a new search query, an area portion selected by a user in the figure displayed on the user interface screen as a search result.

Images