JP2000148793A - Method and device for similar retrieval of composite media document and storage medium stored with similar retrieval program for composite media document - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform retrieval by giving structure information to part of a retrieval key even if information regarding a document structure is not known by deciding the similarity of a composite media document according to feature information extracted from a constituent element of the composite media document and feature information extracted from a constituent element of a retrieval condition. SOLUTION: The syntax of the given composite multimedia document is analyzed (S1) and 1st feature information is extracted from a constituent element of the composite media document (S2). The document and extracted feature information are stored (S3) and the composite media document is inputted as a retrieval condition (S4). The syntax of the retrieval condition is analyzed (S5) and 2nd feature information is extracted from a constituent element of the retrieval condition obtained as a result of the analysis (S6). Then the similarity between two composite media documents is decided (S7) according to the 1st feature information and 2nd feature information which are stored and similar composite media documents are outputted (S8).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for similarity search of a composite media document and a storage medium storing a similarity search program for a composite media document. The present invention relates to a method and an apparatus for searching similarity of a composite media document for searching and a storage medium storing a similarity search program for a composite media document.

[0002]

2. Description of the Related Art Conventional techniques related to structured document retrieval methods include StructureIndex and Content I of Nara Institute of Technology.
There are ndex combinations (Kanemoto, Kato, Kinutani, Yoshikawa, "Structured document search method that can be updated efficiently") and so on. In these systems, an index such as an inverted file relating to an appearance position of text information stored in a structured document such as an SGML document or an XML document or information of a document structure (element name or the like) is prepared in advance, and the text information or the text is prepared. A combination of information and structural information as a search key,
Structured documents can be searched by performing a logical search that determines whether the given keyword is included in the document or a range search that determines whether the given keyword appears in the specified range. I have to.

In systems such as JustBase ConceptBase and Komatsu Software VextSearch, a concept included in a text entered as a search key in a natural sentence or the like is compared with a concept included in a text to be searched, and similar documents ( Text-only) search.

[0004]

However, when the above-described conventional method is applied to a similarity search for a composite media document, a similarity search method for a composite media document has not been established. The following problems occur. Unless the user knows in advance information about the document structure, such as the element name (Note: tag name representing the structure information) in the structured document, it is not possible to perform a search with the structure information given as part of the search key .

[0005] Similar search of a composite media document using information and structure information of media other than text such as images and sounds as search keys cannot be performed. The present invention has been made in view of the above points, and when a composite media document is exemplified as a search condition, feature information such as text information, image and audio information, and structure information included in the exemplified document is stored. Text information, image and audio information, structural information, and other feature information, respectively, and compare the similarity of text information, similarity of image information, similarity of audio information, and similarity of structural information. Was calculated individually, multiplied by the weight value, and the overall evaluation value was calculated as the similarity at the document level, and by adjusting the weight value, the similarity of the text information was emphasized. Similar search method and apparatus for a composite media document and similar search program for a composite media document capable of performing similar search, similar search emphasizing similarity of image information, similar search emphasizing similarity of structural information, and the like And to provide the stored storage medium.

[0006]

FIG. 1 is a diagram for explaining the principle of the present invention. The present invention (claim 1) is a structured document comprising text information, image information, audio data as human voice data, music information including CDs and records, and audio information including music data. In a similar search method for a composite media document, a given composite media document is parsed (step 1), and first characteristic information is extracted from components of the composite media document obtained as a result of the analysis (step 2). The document and the extracted feature information are stored (step 3), the composite media document is input as a search condition (step 4), the input search condition is parsed (step 5), and the search condition obtained as a result of the analysis is obtained. The second feature information is extracted from the constituent elements of (2) (step 6), and two types of composite media documents are classified based on the stored first feature information and second feature information. It determines degrees (step 7), and outputs the similar mixed media document (Step 8).

According to the present invention (claim 2), at the time of retrieval,
The user inputs the exemplified document as a search key, extracts second feature information from the exemplified document, and extracts the extracted second feature information.
The similarity between documents is calculated based on the feature information of the first and second feature information. According to the present invention (claim 3), when calculating the similarity of a composite media document, information and structure information of media including texts, images, and sounds constituting the document exemplified as a search key are stored. The evaluation value based on the similarity determination result for each component with the information on the media including text, images, and sound and the structure information constituting the document,
Set as the similarity of the entire composite media document.

According to the present invention (claim 4), as the similarity determination result for each component, text information of the exemplified document,
A similarity determination result of the text information of the stored document, image information of the exemplified document, a similarity determination result with the image information of the stored document, audio information of the exemplified document, The similarity determination result with the voice information of the stored document, the structure information of the exemplified document, and the similarity determination result with the structure information of the stored document are used.

According to the present invention (claim 5), when calculating the similarity of a composite media document, the similarity is calculated for each feature information of text information, image information, audio information, and structural information included in the document. Then, a value obtained by multiplying the similarity by a weight value and taking a linear sum is defined as the similarity of the entire composite media document.

According to the present invention (claim 6), when calculating the similarity of a composite media document, if there is a plurality of the same media in the same document, text information and image information included in the exemplified document For all search keys for each medium including voice information, calculate all similarities for the search key in the stored document in the medium, and determine the highest similarity with the representative similarity for the search key. I do.

According to the present invention (claim 7), when calculating the similarity of a composite media document, the structure information of the exemplified document and the structure information of the stored document are each used as a search key with an order label. The degree of similarity of the structural information of a document is expressed by comparing the shapes of an ordered labeled tree expressing an exemplified document and an ordered labeled tree expressing an accumulated document by expressing the document as an ordered labeled tree. Set.

According to the present invention (claim 8), when setting the similarity of the structural information, the document structure is regarded as a tree, and the tree representing the exemplified document is converted into a tree representing the stored document. For this purpose, the number of times that an editing operation including insertion of a node, deletion of a node, and change of a node name necessary for the editing operation is performed, and an editing distance calculated from the cost required to perform the editing operation are set as the similarity of the document.

According to the present invention (claim 9), when calculating the similarity of a composite media document, the similarity based on the characteristic information of text information, image information, audio information, and structural information included in the document is calculated, The first stage selection based on the similarity calculation result based on the feature information of the structural information is performed, and the similarity based on the feature information of the text information, image information, and audio information included in the document is determined based on the similarity of the entire composite media document. Degree.

According to a tenth aspect of the present invention, when calculating the similarity of a composite media document, the similarity based on feature information of text information, image information, audio information, and structure information included in the document is calculated. The first stage selection based on the similarity calculation result based on the feature information of the text information, image information and audio information is performed, and the similarity based on the feature information of the structural information included in the document is determined based on the similarity of the entire composite media document. Degree.

According to the present invention (claim 11), when calculating the similarity of a composite media document, if a plurality of identical media exist in a document exemplified as a search key, the similarity of the media at the document level is calculated. Set the degree. According to the present invention (claim 12), when setting the similarity at the document level of a medium, each search key of a plurality of media included in a document exemplified as a search key is included in the exemplified document. For all search keys for each medium including text information, image information, and audio information, calculate all similarities for the search key in the medium in the stored document, and determine the highest similarity for the search key. As the representative similarity, the average value of the representative similarities is calculated, and the similarity at the document level of the media is set.

According to the present invention (claim 13), when the similarity at the document level of a medium is set, each search key of a plurality of media in a document exemplified as a search key is set in the exemplified document. For all search keys for each medium including text information, image information, and audio information included in, calculate all the similarities with respect to the search key in the medium in the stored document, and determine the one having the highest similarity. The similarity of the representative to the search key is set, and the highest similarity among the similarities of the representatives is set to the similarity at the document level of the media.

According to the present invention (claim 14), when performing a similarity search of a composite media document, the structure information of the document exemplified as a search key and the structure information of the stored document are each ordered tree (ordered tree). labeled tree), and expresses the feature information of each media in each document as an attribute-labeled ordered tree, which is stored as an attribute of a node in the ordered-labeled tree. The similarity of the composite media document is set by comparing the attribute and the shape of the attached tree with the attribute-ordered labeled tree expressing the stored document.

According to the present invention (claim 15), when performing a similarity search of a composite media document, feature information similar to feature information which is an attribute of each node of an attribute-ordered labeled tree representing an exemplified document. The similarity of the composite media document is set from the difference in the structural positional relationship of the nodes for the ordered labeled tree expressing the stored document having the node having the attribute as the attribute.

According to the present invention (claim 16), when setting the similarity of the structural information, the document structure is regarded as an ordered labeled tree, and based on the characteristic information on the ordered labeled tree,
The feature information is digitized and mapped in a multidimensional vector space, and the distance in the vector space is set as the similarity of the document. The present invention (claim 17) calculates the similarity of a document by digitizing and using the name of each node, the number of nodes, and the position information of each node as feature information on the ordered label tree. I do.

According to the present invention (claim 18), the feature information is text information, the concept represented by the description content of the text, the appearance frequency of each word, and the image information is the hue, saturation, and luminance of the image. For color information, color information, sound information, sound intensity and melody, and for structural information, the tree structure and node label name when the document structure is represented by a tree with order labels,
It is assumed that link information and the like are feature information extracted from components of the composite media document.

According to the present invention (claim 19), when determining the similarity, the similarity of the stored document with respect to the exemplified document is used as a search key for the text, image, The evaluation value based on the similarity determination result for each component of the information and structure information of the media including audio and the text, images, and the information and structure information of the media including audio that constitute the stored document, The overall similarity of the composite media document is set, and the similarities of the stored documents are arranged in descending order to rank and determine the similarity.

According to the present invention (claim 20), when the similarity is set, the similarity is set for each component of the composite media document, and text information, image information, audio information, and text information included in the document are set. Calculate the similarity for each feature information of the structure information,
A value obtained by multiplying the similarity by a weight value and taking a linear sum is defined as the similarity of the entire composite media document.

According to the present invention (claim 21), when the similarity of the entire composite media document is set, the similarity itself of each component or the similarity of each component is used as an evaluation value based on the similarity determination result for each component of the document. , A product obtained by multiplying the similarity of each component by a weight given by the user. FIG.
FIG. 1 is a diagram illustrating the principle of the present invention.

According to the present invention (claim 22), text information,
In addition to image information, voice data that is human voice data,
A composite media document similarity search device which is a structured document composed of music data including a CD or a record and audio information including music data, comprising: a composite media document input unit 10 for inputting a composite media document; A composite media document provided by the document input means 10;
Document analysis means 40 for parsing input search conditions
A feature information extracting unit 50 for extracting feature information from a component of a document obtained as a result of analysis by the document analyzing unit 40
Storage means 60 for storing the characteristic information extracted by the composite media document and characteristic information extraction means 50; and search condition input means 30 for inputting the composite media document as a search condition
Based on the characteristic information of the composite media document stored in the storage unit 60 and the characteristic information extracted by the characteristic information extraction unit 50 based on the result of analyzing the input search condition by the document analysis unit 40. Document comparing means 80 for determining the degree of similarity of two composite media documents, and output means 90 for outputting similar composite media documents based on the degree of similarity determined by the document comparing means 80.

According to the present invention (claim 23), the search condition input means 30 includes a means for inputting a document exemplified by a user as a search key, and the feature information extracting means 50 performs a search from a given composite media document. An input document feature information extracting unit for extracting a composite media document exemplified as a key, and a search feature information extracting unit for extracting feature information of search information from a document exemplified by a user. Based on the input document feature information extracted by the document feature information extraction unit and the search feature information extracted by the search feature information extraction unit, a similarity calculating the similarity between the composite media document and the composite media document exemplified as the search key Including a degree calculation means.

According to the present invention (claim 24), in the similarity calculating means, text information, image information, voice information and structure information constituting a document exemplified as a search key by the search condition input means 30 are stored. Text information, image information, and the like constituting the composite media document stored in the means;
A similarity setting unit that sets an evaluation value based on a similarity determination result for each component with the audio information and the structure information as a similarity of the entire composite media document.

According to the present invention (claim 25), in the similarity setting means, the similarity between the text information of the exemplified document and the text information of the document stored in the storage means is determined as the similarity determination result for each component. The determination result, the image information of the exemplified document, the similarity determination result of the image information of the document stored in the storage unit, the audio information of the exemplified document, and the similarity stored in the storage unit. The similarity determination result with the voice information of the document in question and the similarity determination result between the structure information of the example document and the structure information of the document stored in the storage unit are used.

According to the present invention (claim 26), the similarity setting means calculates the similarity for each of the text information, image information, audio information, and feature information of the structure information included in the document, and calculates the similarity. It includes a linear sum calculating means for multiplying the weight value and calculating the linear sum to obtain the similarity as the entire composite media document. According to the present invention (claim 27), in the document comparing means, when a plurality of the same media exist in the same document, texts, images,
For all search keys for each medium including voice, all similarities for the search key in the media in the stored document are calculated, and the one having the highest similarity is set as the representative similarity for the search key. A similarity determination unit is included.

According to the present invention (claim 28), in the similarity calculating means, each of the structure information of the exemplified document and the structure information of the stored document is used as a search key in an ordered labeled tree. By comparing the shapes of the ordered labeled tree representing the exemplified document and the ordered labeled tree representing the stored document, the ordered labeling which sets the similarity of the structural information of the document is performed. A tree shape comparing means is included.

According to the present invention (claim 29), in determining the similarity of the structural information, the tree structure comparing means considers the document structure as a tree and accumulates from the tree representing the exemplified document. Number of editing operations including insertion, deletion, and renaming of nodes required to convert the document into a tree representing the edited document, and an editing distance calculated from the cost required to perform the editing operation Is set as the similarity of the document.

The present invention (claim 30) provides a similarity calculating means for calculating similarity based on characteristic information of text information, image information, audio information, and structure information included in a document, and a feature of the structure information. Means for selecting the first step based on the result of the similarity calculation based on the information, and the similarity based on the feature information of the text information, image information and audio information included in the document, and the similarity as the entire composite media document. Means.

According to a thirty-first aspect of the present invention, there is provided a similarity calculating means for calculating a similarity based on characteristic information of text information, image information, voice information, and structural information included in a document; Means for selecting the first step based on the similarity calculation result based on the feature information of the information and the voice information, and the similarity based on the feature information of the structural information included in the document, Means.

According to the present invention (claim 32), in the similarity calculating means, when a plurality of identical media exist in a document exemplified as a search key, a document level for setting the similarity at the document level of the media is set. A similarity calculation means is included.
According to the present invention (claim 33), in the document level similarity calculating means, for each search key of a plurality of media in the document exemplified as a search key, text information, image information, Means for calculating all the similarities for the search key in the stored document for all the search keys for each medium including audio information, and determining the highest similarity for the representative similarity to the search key Means, a means for calculating an average value of representative similarities, and a document level similarity setting means for setting the similarity at the document level of the media.

According to the present invention (claim 34), in the document level similarity setting means, for each search key of a plurality of media in a document exemplified as a search key, text information included in the exemplified document is provided. Means for calculating all similarities to the search key in the stored document for all the search keys for each medium including image information and audio information; Means for determining the similarity of the representative and means for determining the highest similarity among the similarities of the representative at the document level of the media are included.

According to the present invention (claim 35), in the similarity calculating means, each of the structural information of the document exemplified as the retrieval key and the structural information of the stored document is represented as an ordered labeled tree. Then, the feature information of each media in each document is represented as an attribute-ordered tree stored as an attribute of a node in the order-labeled tree, and stored as an attribute-ordered tree representing the exemplified document. And a similarity search unit that sets the similarity of the composite media document by comparing the attribute and the shape of the tree with the attributed ordered label expressing the document.

According to the present invention (claim 36), in the similarity search means, a node having, as an attribute, feature information similar to feature information which is an attribute of each node of an attribute-ordered labeled tree expressing an exemplified document. Means for setting the similarity of the composite media document from the difference in the structural positional relationship of the node with respect to the attributed ordered labeled tree representing the stored document.

According to a thirty-seventh aspect of the present invention, the order-labeled tree shape comparing means regards the document structure as an order-labeled tree and, based on the characteristic information on the order-labeled tree, determines the document structure in a multidimensional vector space. Means for numerically converting the feature information and mapping, and means for setting a distance in a vector space as a document similarity.

According to the present invention (claim 38), similarity of a document can be obtained by digitizing and using the name of each node, the number of nodes, and the position information of each node as feature information on the ordered label tree. Calculate the degree. According to the present invention (claim 39), if the feature information is text information,
For the concept represented by the description content of the text, the appearance frequency of each word, for image information, the hue, saturation, luminance, color arrangement, and color arrangement of the image, for the sound information, the strength and melody of the sound, and for the structural information, In the case where the document structure is represented by a tree with an order label, the tree shape, the label name of the node, the link information, and the like are the feature information extracted from the components of the composite media document.

According to the present invention (claim 40), the document comparing means 8
0, the similarity of the stored document with respect to the exemplified document is determined by using the information of the media including text, image, and sound constituting the exemplified document as a search key and the structural information, and the accumulated document. Constituent text,
Means for setting an evaluation value based on a similarity determination result for each component with media information and structure information including images and sounds as the overall similarity of the composite media document, and arranging the similarities of the stored documents in descending order Means for determining the similarity.

According to the present invention (claim 41), in the similarity setting means, means for setting the similarity for each component of the composite media document, text information, image information, voice information, and the like included in the document are provided. It includes means for calculating the similarity for each feature information of the structure information, and means for multiplying the similarity by a weight value and taking a linear sum to obtain the similarity of the entire composite media document.

According to the present invention (claim 42), in the similarity setting means, the similarity of each component or the similarity of each component is used as an evaluation value based on the similarity determination result for each component of the document.
Means for using a product obtained by multiplying the similarity of each component by a weight given by a user is included. The present invention (Claim 4
3) Similarity of a composite media document which is a structured document composed of text information, image information, audio data which is human voice data, and music information including CDs and records and audio information including music data. A storage medium storing a search program, characterized by a document analysis process for parsing a given composite media document and an input search condition, and a component obtained from a document obtained as a result of analysis by the document analysis process. A feature information extraction process for extracting information, a storage process for storing the composite media document and the feature information extracted by the feature information extraction process in a storage unit, a search condition input process for inputting the composite media document as a search condition, and storage. Analyze the feature information of the composite media document stored in the means and the input search conditions by the document analysis process A document comparison process for determining the similarity between two composite media documents from the feature information extracted in the feature information extraction process based on the result of the comparison, and a similar composite media document based on the similarity determined in the document comparison process. And an output process for outputting

According to the present invention (claim 44), the search condition input process includes a process of inputting a document exemplified by a user as a search key, and in the feature information extraction process, feature information is provided from a given composite media document. An input document feature information extraction process for extracting, and a search feature information extraction process for extracting feature information of a composite media document exemplified as a search key from a document exemplified by a user, wherein the input document feature information is included in a document comparison process. A similarity calculation process for calculating a similarity between a composite media document and a composite media document exemplified as a search key based on the input document feature information extracted in the extraction process and the search feature information extracted in the search feature information extraction process. including.

According to the present invention (claim 45), in the similarity calculation process, text information, image information, voice information and structure information constituting a document exemplified as a search key in the search condition input process, and storage means An evaluation value based on the similarity determination result for each component with the text information, image information, audio information, and structural information constituting the composite media document stored in the composite media document is set as the similarity of the entire composite media document Includes a similarity setting process.

According to the present invention (claim 46), in the similarity setting process, the similarity between the text information of the exemplified document and the text information of the document stored in the storage means is determined as the similarity determination result for each component. The determination result, the similarity determination result between the image information of the exemplified document, and the image information of the document stored in the storage unit, the voice information of the exemplified document, and the similarity determination result stored in the storage unit A similarity determination result with the audio information of the document and a similarity determination result between the structure information of the example document and the structure information of the document stored in the storage unit are used.

According to the present invention (claim 47), in the similarity setting process, a similarity is calculated for each of text information, image information, audio information, and structural information included in a document, and the similarity is calculated. Includes a linear sum calculation process in which the weighted value is multiplied and the linear sum is taken as the similarity of the entire composite media document. According to the present invention (claim 48), in the document comparison process, when a plurality of the same media exist in the same document, all search keys for each medium including text, image, and sound included in the exemplified document are provided. , Calculating all similarities for the search key in the media in the stored documents, and taking the one with the highest similarity as the representative similarity for the search key.

According to the present invention (claim 49), in the similarity calculation process, the structure information of the exemplified document and the structure information of the stored document are each used as a search key in an ordered labeled tree. By comparing the shapes of the ordered labeled tree representing the exemplified document and the ordered labeled tree representing the stored document, the ordered labeling which sets the similarity of the structural information of the document is performed. Includes tree shape comparison process.

According to the present invention (claim 50), in the order-labeled tree shape comparison process, when judging the similarity of the structural information, the document structure is regarded as a tree and stored from the tree representing the exemplified document. Number of editing operations including insertion, deletion, and renaming of nodes required to convert the document into a tree representing the edited document, and an editing distance calculated from the cost required to perform the editing operation Is set as the similarity of the document.

According to the present invention (claim 51), in the similarity calculation process, a process of calculating a similarity based on feature information of text information, image information, audio information, and structure information included in a document; The process of selecting the first stage based on the similarity calculation result based on the information, and the similarity based on the feature information of the text information, image information, and audio information included in the document are compared with the similarity of the entire composite media document. Process.

According to the present invention (claim 52), in the similarity calculating process, a similarity calculating process based on characteristic information of text information, image information, audio information, and structural information included in a document, A process of selecting the first step based on the similarity calculation result based on the feature information of the information and the voice information, and the similarity based on the feature information of the structural information included in the document, Process.

According to the present invention (claim 53), in the similarity calculation process, when a plurality of same media exist in a document exemplified as a search key, a document level for setting the similarity of the media at the document level is set. Includes a similarity calculation process. According to the present invention (claim 54), in a document level similarity calculation process, for each search key of a plurality of media in a document exemplified as a search key, text information, image information, A process of calculating all similarities for the search key in the stored document for the media for all search keys for each medium including audio information, and determining the highest similarity for the representative similarity to the search key. And the process
The method includes a process of calculating the average value of the similarities of the representatives, and a document level similarity setting process of setting the similarities of the media at the document level.

According to a fifty-fifth aspect of the present invention, in the document level similarity setting process, for each search key of a plurality of media in a document exemplified as a search key, text information included in the exemplified document is provided. , Image information,
For all search keys for each media including audio information,
A process of calculating all similarities for the search key in the medium in the stored documents, a process of setting the highest similarity to the representative similarity to the search key, Making the media with a high similarity a similarity at the media document level.

According to the present invention (claim 56), in the similarity calculation process, each of the structure information of the document exemplified as the retrieval key and the structure information of the stored document is represented as an ordered labeled tree. Then, the feature information of each media in each document is represented as an attribute-ordered tree stored as an attribute of a node in the order-labeled tree, and stored as an attribute-ordered tree representing the exemplified document. A similarity search process for setting the similarity of the composite media document by comparing the attribute and the shape with the attributed ordered labeled tree expressing the represented document.

According to the present invention (claim 57), in the similarity search process, nodes having characteristic information similar to the characteristic information, which is the attribute of each node of the attributed ordered labeled tree representing the exemplified document, as an attribute. The method includes a process of setting the similarity of the composite media document from the difference in the structural positional relationship of the node with respect to the ordered labeled tree expressing the stored document.

According to the present invention (claim 58), in the order-labeled tree shape comparison process, the document structure is regarded as an order-labeled tree, and based on feature information on the order-labeled tree, the document structure is expressed in a multidimensional vector space. And a process of setting the distance in the vector space as the similarity of the document.

According to the present invention (claim 59), similarity of a document is obtained by digitizing and using the name of each node, the number of nodes, and the position information of each node as feature information on the ordered label tree. Calculate the degree. According to the present invention (claim 60), if the feature information is text information,
For the concept represented by the description content of the text, the appearance frequency of each word, for image information, the hue, saturation, luminance, color arrangement, and color arrangement of the image, for the sound information, the strength and melody of the sound, and for the structural information, In the case where the document structure is represented by a tree with an order label, the tree shape, the label name of the node, the link information, and the like are the feature information extracted from the components of the composite media document.

According to the present invention (claim 61), in the document comparison process, the similarity of the stored document to the exemplified document includes text, image, and voice constituting the exemplified document as search keys. Media information and structure information, and the text that constitutes the stored document,
A process of setting an evaluation value based on a similarity determination result for each component with media information and structure information including images and sounds as the overall similarity of a composite media document, and arranging the similarities of stored documents in descending order. And determining the degree of similarity.

According to the present invention (claim 62), in the similarity setting process, the process of setting the similarity for each component of the composite media document, the text information, image information, audio information, It includes a process of calculating the similarity for each feature information of the structure information, and a process of multiplying the similarity by a weight value and taking a linear sum to obtain a similarity of the entire composite media document.

According to the present invention (claim 63), in the similarity setting process, the similarity of each component or the similarity of each component is used as an evaluation value based on the similarity determination result for each component of the document. Multiplied by the weight given by the user. As described above, in the present invention, by parsing a given document, a unit of a component such as media information and structure information to be compared between the exemplified document and the stored document is determined. It is possible to decide.

Further, by characterizing a document with the extracted characteristic information, it is possible to perform a search based on information on the contents and logical structure of the document. In addition to text information,
Image information, audio information, structure information, and the like can also be used as a part of the search key. Furthermore, it is possible to access a document at high speed by creating an index from a stored text information, image information, audio information, structure information, and the like to a document containing the information.

In addition, since the contents of characteristic information such as text information and image information of a document can be confirmed on the display,
It is easy for a user to input a composite media document including intended feature information as a search key. In addition, similarity is calculated for each feature information included in the composite media document input as a search key, and an evaluation value based on the similarity is calculated. For example, the similarity is calculated for each feature information of the document,
By multiplying them by the weight value and calculating the sum as the similarity at the document level, the similarity of the composite media document using the similarity of the image, audio information, and structural information in addition to the text information as a search condition Search becomes possible.

Further, since the similarity is calculated for each feature information in the document, for each individual similarity calculation method, for example,
It is possible to easily adopt a similarity calculation method that differs only in the similarity calculation method of image information and partially replace the image information.

[0062]

DESCRIPTION OF THE PREFERRED EMBODIMENTS As shown in FIG. 3, the components of a composite media document include text information, image information, audio information, and structure information. Hereinafter, the similarity search in the composite media document will be described. FIG. 4 shows the configuration of the composite media document similarity search apparatus of the present invention.

The composite media document similarity search apparatus shown in FIG. 3 includes a composite media document input device 10, a search condition input device 20, a search condition input unit 30, a composite media document analysis unit 40, a feature information extraction unit 50, and a storage unit. 60, memory 70,
The document comparison unit 80 includes a display device 90. The composite media document input device 10 includes text information, image information,
A document containing voice information and structure information is input.

The search condition input device 20 includes a pointing device such as a mouse used by the user for input,
It is a keyboard or the like. The search condition input unit 30 allows a user to input a file name of a document from a keyboard serving as the search condition input device 20, operate a mouse to click a document icon, or obtain a document obtained by a previous search result. A composite media document to be input as a search key is obtained by clicking on with a mouse. More specifically, a composite media document serving as a search key for searching for a composite media document is exemplified. The search key is illustrated by designating the file name of the exemplified document or clicking the icon of the exemplified document on the display with a pointing device or the like. In addition, when exemplifying a document, the user can specify a portion in which similarity is to be emphasized, and appropriately change the value of the weight indicating the degree of importance in the feature information of the portion in which similarity is to be emphasized. It is possible to input. At this time, a value of a weight indicating which part of the document should be prioritized for similarity and the number k of documents to be returned as a search result are obtained from the user. Alternatively, use system defaults.

The composite media document analysis unit 40 analyzes the syntax of the document provided from the composite media document input device 10 or the search condition input unit 30, and analyzes the document such as text information, image information, audio information, and structure information. Detect components. Here, the composite media document analysis unit 40
The input document is analyzed using an L or XML parser, and components of the document such as text information, image information, audio information, and structure information are detected from the document.

The feature information extracting section 50 extracts feature information expressing features of document components such as text information, image information, audio information, and structure information. For example, text information stores the concept represented by the description of the text, image information stores the hue, saturation, brightness, color arrangement, etc. of the image information, and voice information stores the characteristic information such as the strength and melody of the sound. It is extracted together with the information of the document ID, the element name and the appearance position of the document. In the case of structural information, when the document structure is represented by a tree with an order label, the tree shape (hierarchical structure, etc.), the label names of the nodes, and the link information are extracted from the components of the composite media document. And

The storage unit 60 stores the given document in the memory 7
Store at 0. Also, an index is created from each piece of feature information to a document containing the feature information. The document comparison unit 80
By comparing characteristic information between the exemplified composite media document and the composite media document stored in the memory 70,
The similarity is obtained, and the one with the highest similarity is output. The similarity as a composite media document is based on text information, image information,
Assume that an evaluation value is calculated based on individual similarity calculation results such as voice information and structure information. For example, regarding similarities of text information, image information, audio information, and the like, based on a multidimensional vector space model, each feature information is mapped onto the multidimensional vector space, and the characteristics of the illustrated document on the multidimensional vector space are calculated. If the distance between the information and the feature information of the stored document is short, it is possible to adopt an approach of setting the similarity to be high. It is also possible to determine the similarity by arranging the similarities of the stored documents in descending order to rank the documents.

Hereinafter, the operation of the above configuration will be described by dividing it into a composite media document storage phase and a composite media document search phase. FIG. 5 is a flowchart of the composite media document storage phase of the present invention. Step 101) First, the composite media document input device 1
Input a composite media document from 0.

Step 102) The composite media document analysis unit 40 analyzes the syntax of the composite media document input from the composite media document input device 10 to obtain text information,
Detect document components such as image information, audio information, and structural information. Step 103) Next, the feature information extraction unit 50 determines the document components such as text information, image information, audio information, and structural information. For voice information, such as hue, saturation, luminance, and color arrangement, feature information such as the intensity of a sound and a melody is extracted together with the ID of the document in which the feature information is stored, the element name, and the appearance position. This process is repeated for all components.

Step 104) The storage unit 60 creates an index from the given document and each piece of characteristic information to a document containing the characteristic information, and stores the index in the memory 70. Next, the operation of the composite media document search phase will be described.
FIG. 6 is a flowchart of the composite media document search phase of the present invention. Step 201) The search condition input unit 30 allows the user to input a file name of a document from a keyboard serving as the search condition input device 20, operate a mouse to click a document icon, or obtain a document obtained by a previous search result. By clicking on with a mouse, a composite media document to be input as a search key is obtained. At this time, a value of a weight indicating which part of the document should be prioritized for similarity and the number k of documents to be returned as a search result are obtained from the user. Alternatively, use system defaults.

Step 202) Next, similarly to the processing in the composite media document storage phase, the composite media document analysis unit 40 analyzes the syntax of the composite media document input from the composite media search condition input unit 30, and performs text information , Document components such as image information, audio information, and structure information are detected. Step 203) The characteristic information extracting unit 50 converts the characteristic information of the document components such as text information, image information, audio information, and structural information into the ID of the document in which the characteristic information is stored, as in the composite media document storage phase. , Along with information on element names and appearance positions, and characteristic information on document components such as text information, image information, audio information, and structure information of the exemplified document. This process is repeated for all components.

Step 204) The document comparing section 80 compares the characteristic information of the exemplified document with the characteristic information of the document stored in the memory 70, calculates the similarity for each characteristic information, and calculates them. The evaluation value based on the result is calculated as the similarity as the composite media document. The method of calculating the similarity will be described later. Step 205) The document comparison unit 80 arranges the similarities in descending order, and selects the top k documents requested by the user from the index as documents having a high similarity.

Step 206) The selected document having a high similarity is displayed on the display device 90 as a search result. Next, a method of calculating the similarity in the above will be described. FIG. 7 is a flowchart (part 1) of performing document comparison for obtaining similarity according to the present invention.

Step 301) The document comparing section 80 has the feature information of the document (search condition) inputted from the feature information extracting section 50 from the search condition input section 30 and the feature information of the composite media document inputted from the storage section 60. To get. Step 302) If the feature information is structural information, the process proceeds to step 303; otherwise, the process proceeds to step 304.

Step 303) The document comparing section 80 regards the feature information of the search condition and the structure information of the feature information of the composite media document as a tree, calculates the edit distance between the two trees, and proceeds to step 306. I do. Further, a method of calculating a structural positional relationship between nodes, converting a tree into a multidimensional vector, and calculating a distance in a multidimensional vector space are also possible.

Step 304) The document comparing section 80 calculates a distance in the multidimensional vector space. Step 305) Among the same kind of feature information, the one with the smallest distance is selected as a representative. Step 306) Calculate the similarity at the document level.

Step 307) A document having a high similarity is selected from the index. Step 308) Output the selected document to the display device 90. As one method of calculating the similarity, for example, the similarity can be obtained as follows. (1) First similarity calculation method: information and structure information of media including texts, images, and sounds constituting a document exemplified as a search key, and a document stored in a memory 70 An evaluation value based on a similarity determination result for each component with information on media including text, images, and audio and structure information is set as the overall similarity of the composite media document. Here, as the evaluation value based on the similarity determination result, the similarity itself of each component or a value obtained by multiplying the similarity of each component by a weight given by a user is used.

(2) Second Similarity Calculation Method: The similarity of text information is calculated in a multidimensional vector space between the feature information of the input text information and the feature information of the text information stored in the memory 70. Calculate by calculating the distance.
The similarity of the image information is calculated by calculating the distance in the multidimensional vector space between the characteristic information of the input image information and the characteristic information of the image information stored in the memory 70.

The similarity of voice information is calculated by finding the distance in the multidimensional vector space between the feature information of the input voice information and the feature information of the voice information stored in the memory 70. In each of the above information, the information having a small distance in the multidimensional vector space is calculated as having a high similarity.

(3) Third similarity calculation method: In addition to the above method (2), when a plurality of the same media exist in the same document, the one having the highest similarity is represented by the representative similarity. Set the degree. For example, as shown in FIG. 8, in a document including a plurality of pieces of image information, the degree of similarity between the image information present in the document exemplified as the search key and the plurality of pieces of image information present in the stored document Is calculated, and the one with the highest similarity is set as the representative similarity. This is performed for all image information existing in the document exemplified as the search key.

In FIG. 8, the similarity between the image A in the example document and the images a, b, and c in the stored document is calculated.
The image having the highest similarity (for example, image a) is regarded as a similar image in the stored document with respect to image A in the example document. Further, image B in the example document and images a, b, c in the stored document
And the image with the highest similarity (for example, image c) is determined to be a similar image in the stored document with respect to image B in the example document.

(4) Fourth similarity calculation method: In addition, the structure information of the feature information is edited as necessary for converting the document structure from one tree to another tree by regarding the document structure as a tree. The distance is calculated and set so that the similarity increases as the editing distance decreases. The editing distance is calculated from the number of times of editing operations such as insertion of a node, deletion of a node, and change of a node name necessary for transforming a tree, and the cost required for performing the editing operation. Thereby, the similarity can be calculated, and the one with a small editing distance is calculated as the one with a high similarity.

(5) Fifth similarity calculation method: The similarity of text information, image information, audio information, structure information, etc. is calculated, and the similarity of any part in the document acquired by the search condition input unit 30 Weights, or based on system default values, multiply the individual weights given to the similarities of text information, image information, audio information, structural information, etc., and calculate the linear sum. Take.
The linear sum corresponds to the similarity as a composite media document.

(6) Sixth similarity calculation method: Next, when the document comparison unit 80 calculates the similarity of a composite media document, text information, image information, audio information, and structure information included in the document are included. After calculating the similarity based on the characteristic information of the document and performing the first stage selection based on the result of the similarity calculation based on the characteristic information of the structural information, text information, image information, and voice information included in the document are obtained. The similarity based on the characteristic information of the composite media document is set as the overall similarity.

Hereinafter, this method will be described in detail. FIG.
9 is a flowchart (part 2) of performing document comparison for obtaining similarity according to the present invention. Step 401) In the characteristic information information extraction unit 50,
The feature information for the input search condition is input.

Step 402) The similarity of the text information is obtained by calculating the distance in the multidimensional vector space between the characteristic information of the input text information and the characteristic information of the text information stored in the memory 70, or by inputting the distance. The calculation is performed by calculating a difference between values obtained from the appearance frequency of the feature information of the text information and the feature information of the stored text information.

The similarity of the image information is calculated by calculating the distance in the multidimensional vector space between the characteristic information of the input image information and the characteristic information of the image information stored in the memory 70. The similarity of the voice information is calculated by calculating the distance in the multidimensional vector space between the feature information of the input voice information and the feature information of the voice information stored in the memory 70. It should be noted that those having a small distance in the multidimensional vector space and those having a small absolute value of the difference between the values obtained from the appearance frequency and the like are calculated as having a high similarity.

In the feature information, the structural information is such that the document information is regarded as a tree, and an edit distance required to convert one tree to another tree is calculated. Is converted into a multidimensional vector to obtain a distance in a multidimensional vector space, and the similarity can be calculated. An object with a small editing distance or an object with a small distance in a multidimensional vector space is calculated as having a high similarity.

Step 403) Calculate similarities of text information, image information, audio information, structure information, and the like, and determine a weight value indicating which part of the document obtained by the search condition input unit 30 should be prioritized. Alternatively, the first selection based on the similarity of the structural information is performed based on the default value of the system. Step 404) The similarity of the text information, image information, audio information, and the like of the document remaining after the first stage selection corresponds to the similarity as the composite media document.

Step 405) A document having a high similarity is selected from the index. Step 406) Display the selected document on the display device 90. (7) Seventh similarity calculation method;
In calculating the similarity of composite media documents,
A similarity based on text information, image information, audio information, and structural information feature information included in the document is calculated, and a first stage based on the similarity calculation result based on the text information, image information, and audio information feature information is performed. After the selection, the similarity based on the feature information of the structural information included in the document is set as the similarity of the entire composite media document.

Hereinafter, this method will be described in detail. FIG.
9 is a flowchart (part 3) of performing document comparison for obtaining similarity according to the present invention. Step 501) The feature information of the input search condition is input. Step 502) The similarity of the text information is determined by calculating a distance in a multidimensional vector space between the feature information of the input text information and the feature information of the text information stored in the memory 70, or determining the similarity of the input text information. The calculation is performed by obtaining a difference between values obtained from the appearance frequency of the characteristic information and the characteristic information of the stored text information.

The similarity of the image information is calculated by finding the distance in the multidimensional vector space between the characteristic information of the input image information and the characteristic information of the stored image information. The similarity of audio information is calculated by calculating the distance in the multidimensional vector space between the characteristic information of the input audio information and the characteristic information of the stored audio information. Those having a small distance in the multidimensional vector space and those having a small absolute value of the difference between the values obtained from the appearance frequency and the like are calculated as having a high similarity.

The structure information of the feature information is such that the document structure is regarded as a tree, an edit distance required for converting one tree into another tree is calculated, and the feature information of the tree is calculated. It is possible to calculate the degree of similarity by numerically converting the data into a multidimensional vector to obtain a distance in a multidimensional vector space. Those having a small editing distance and those having a small distance in a multidimensional vector space are calculated as having a high similarity.

Step 503) Calculate the similarity of text information, image information, audio information, structure information, etc., and determine the weight of the part of the document acquired by the search condition input unit 30 that is to be prioritized. , Or based on system defaults, text information, image information,
The first stage of selection is performed based on the similarity of audio information and the like.

Step 504) After the selection at the first stage, the similarity of the structural information of the remaining document corresponds to the similarity as the composite media document. Step 505) Select a document having a high degree of similarity from the index. Step 506) Output the selected document to the display device 90. (8) Eighth similarity calculation method: When calculating the similarity of a composite media document, if there are a plurality of the same media in the document exemplified as the search key, the similarity of the media at the document level Set. For example, in the above (3), the case where a plurality of the same media exist in the stored documents to be searched has been described.
In the case where three different images exist in the document serving as the search key, how to set the similarity of the images in the document serving as the search key will be described.

When calculating the similarity of a composite media document, if there are a plurality of the same media in the document exemplified as a search key, the following two examples are used to set the similarity of the media at the document level. Based on this, a case where image information exists in a document will be described. When setting document-level similarity for media,
For each search key of the medium that exists in the document exemplified as the search key, calculate all the similarities to the search key in the medium in the stored document,
When the highest similarity is used as the representative similarity to the search key, and the average value of the representative similarities is calculated and used as the similarity at the document level of the media, the image in the example document shown in FIG. A as a similar image in the stored document for A
Get. Further, an image c is obtained as a similar image in the stored document with respect to the image B in the example document. The average value of the similarity between the image A and the image a and the similarity between the image B and the image c are calculated, and the calculated value is used as the value of the image information at the document level between the example document and the stored document. Set similarity.

When setting the similarity at the document level of the medium, for each of the plurality of search keys of the medium in the document exemplified as the search key, all the search keys corresponding to the search keys of the medium in the stored document are searched. When the similarity is calculated, the highest similarity is used as the representative similarity to the search key, and the highest similarity among the representative similarities is used as the media document level similarity. 8, an image a is obtained as a similar image in the stored document to the image A in the example document. Further, an image c is obtained as a similar image in the stored document with respect to the image B in the example document. The similarity between the image A and the image a and the similarity between the image B and the image c having the highest similarity (for example, the similarity between the image A and the image a) are exemplified. This is set as the similarity of the image information at the document level between the document and the stored document.

Next, when performing a similarity search for a composite media document, each of the structure information of the document exemplified as a search key and the structure information of the stored document is expressed as an ordered labeled tree. An example will be described. FIG. 11 is a diagram for describing expressing the composite media document of the present invention as a sequence-labeled tree with attributes.

When performing a similarity search for a composite media document,
An attribute-labeled ordered tree in which feature information of each media in each document is stored as an attribute of a node in the ordered labeled tree (a tree obtained by expanding the ordered-labeled tree)
Set the similarity of the composite media document by comparing the attribute and shape of the attributed ordered labeled tree representing the exemplified document and the attributed ordered labeled tree representing the stored document I do.

When setting the similarity of the composite media document, the feature information is quantified and mapped on a multidimensional vector space based on the feature information on the ordered labeled tree. Is set as the document similarity. Note that the digitization of the feature information is to digitize the name (label name) of each node, the number of nodes, and the position information of each node.

As described above, even if the user does not know the detailed document structure, it is possible to search for documents using structural information in addition to text information. Further, similarity search of a document can be performed by including image and sound information and structural information in addition to text information as a part of the search key. The search condition input unit 30 and the composite media document analysis unit 4 shown in FIG.
0, feature information extraction unit 50, storage unit 60, document comparison unit 80
Is constructed as a program, stored in a disk device connected to a computer used as a similar search device for a composite media document, or in a portable storage medium such as a floppy disk or a CD-ROM. The present invention can be easily realized by installing the present invention in a personal computer.

The present invention is not limited to the above-described embodiment, but can be variously modified and applied within the scope of the claims.

[0103]

As described above, according to the present invention, a document can be searched using structural information other than text information even if the user does not know the detailed document structure. Further, in addition to text information, image and audio information and structure information can be included in a part of the search key to search for similarity of documents.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining the principle of the present invention.

FIG. 2 is a principle configuration diagram of the present invention.

FIG. 3 is a diagram for explaining a composite media document of the present invention.

FIG. 4 is a block diagram of a similar media document similarity search apparatus of the present invention.

FIG. 5 is a flowchart of a composite media document storage phase of the present invention.

FIG. 6 is a flowchart of a composite media document search phase according to the present invention.

FIG. 7 is a flowchart (part 1) of performing document comparison for obtaining similarity according to the present invention.

FIG. 8 is a diagram for explaining a method of setting similarity when a plurality of the same media exist in the same document according to the present invention.

FIG. 9 is a flowchart (part 2) of performing document comparison for obtaining similarity according to the present invention.

FIG. 10 is a flowchart (part 3) of performing document comparison for obtaining similarity according to the present invention.

FIG. 11 is a diagram for describing expressing a composite media document as an attribute-labeled ordered labeled tree of the present invention.

[Explanation of symbols]

Reference Signs List 10 composite media document input unit, composite media document input device 20 search condition input device 30 search condition input unit, search condition input unit 40 document analysis unit, composite media document analysis unit 50 feature extraction unit, feature information extraction unit 60 storage unit Storage unit 70 Memory 80 Document comparison unit, Document comparison unit 90 Output unit, Display device

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Nobuo Taniguchi 3-19-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo Japan Telegraph and Telephone Corporation (72) Inventor Masashi Yamamuro 3--19, Nishishinjuku, Shinjuku-ku, Tokyo No. 2 F-term in Nippon Telegraph and Telephone Corporation (reference) 5B075 ND03 ND06 ND14 ND16 PP24 PR06 QM08

Claims (63)

[Claims] 1. A composite media document which is a structured document composed of text information, image information, voice data which is human voice data, music information including CDs and records, and voice information including music data. In the similarity search method, a given complex media document is parsed, and first feature information is extracted from components of the complex media document obtained as a result of the analysis, and the complex media document and the extracted feature information are extracted. And inputting the composite media document as a search condition, parsing the input search condition, extracting second characteristic information from the components of the search condition obtained as a result of the analysis, and storing the second feature information. Determining a similarity between two composite media documents based on the first characteristic information and the second characteristic information, and outputting a similar composite media document Similarity search method of a composite media document characterized by Rukoto. 2. At the time of search, a user inputs a document exemplified as a search key, extracts the second feature information from the exemplified document, and extracts the extracted second feature information and the first feature information. 2. The method according to claim 1, wherein the similarity between the documents is calculated based on the characteristic information. 3. When calculating the similarity of the composite media document, information and structure information of media including texts, images, and voices constituting the document exemplified as the search key are stored. An evaluation value based on a similarity determination result for each component with media information and structure information including text, images, and audio constituting said document, is set as the overall similarity of the composite media document. 2. A similarity search method of the composite media document described in 2. 4. The similarity determination result for each of the constituent elements includes: a text information of the exemplified document; a similarity determination result of accumulated text information of the document; and image information of the exemplified document. The similarity determination result with the image information of the stored document, the audio information of the exemplified document, the similarity determination result with the audio information of the accumulated document, and the structure of the exemplified document. 4. The method according to claim 3, wherein a similarity determination result between the information and the stored structure information of the document is used. 5. When calculating the similarity of the composite media document, a similarity is calculated for each feature information of text information, image information, audio information, and structure information included in the document, and the similarity is calculated. Is multiplied by the weight value, and the linear sum is calculated as
4. The similarity search method for a composite media document according to claim 3, wherein the similarity is set as the overall degree of the composite media document. 6. When calculating the similarity of the composite media document, when the same document includes a plurality of the same media, text information, image information, and audio information included in the exemplified document are included. For all search keys for each medium, all similarities for the search key in the medium in the stored document are calculated, and the highest similarity is used as a representative similarity for the search key. A method for similarity search of composite media documents according to 2 and 3. 7. When calculating the similarity of the composite media document, each of the structure information of the exemplified document and the structure information of the stored document is used as the search key in an ordered label tree (ordered tree). a labeled tree), said ordered labeled tree representing said exemplified document;
2. The similarity search method for a composite media document according to claim 1, wherein the similarity of the structure information of the document is set by comparing the shape of the stored document with an ordered label tree. 8. When setting the degree of similarity of the structure information, the document structure is regarded as a tree, and is necessary for converting a tree representing the exemplified document into a tree representing the stored document. 8. The number of times an editing operation including insertion of a new node, deletion of a node, and change of a node name is performed, and an editing distance calculated from a cost required for performing the editing operation is set as the similarity of the document. Similar search method for the described composite media document. 9. When calculating the similarity of a composite media document, the similarity based on feature information of text information, image information, audio information, and structure information included in the document is calculated, and the similarity is calculated based on the feature information of the structure information. 4. A first-stage selection based on a similarity calculation result, and a similarity based on feature information of text information, image information, and audio information included in the document is set as a similarity of the entire composite media document. Similar search method for the described composite media document. 10. When calculating a similarity of a composite media document, a similarity based on feature information of text information, image information, audio information, and structure information included in the document is calculated, and the text information and the image information are calculated. And a first stage selection based on the similarity calculation result based on the feature information of the audio information, and the similarity based on the feature information of the structural information included in the document is set as the similarity of the entire composite media document 4. The method according to claim 3, wherein the similarity search is performed. 11. When calculating the similarity of a composite media document, if there is a plurality of the same media in a document exemplified as a search key, the similarity at the document level of the media is set. And 3. A similarity search method for a composite media document according to 3. 12. When setting the similarity at the document level of the medium, for each search key of the medium that is present in a plurality of documents exemplified as search keys, a text included in the exemplified document For all search keys for each medium including information, image information, and audio information, calculate all the similarities to the search key in the stored document in the medium, and determine the search key having the highest similarity as the search key. The similarity search method for a composite media document according to claim 11, wherein the similarity of the representative is calculated, and an average value of the similarity of the representative is calculated, and the similarity at a document level of the media is set. 13. When setting the similarity at the document level of the medium, for each search key of the medium that is present in a plurality of documents exemplified as search keys, a text included in the exemplified document For all search keys for each medium including information, image information, and audio information, calculate all similarities for the search key in the stored document in the medium, and determine the search key having the highest similarity as the search key. 12. The similarity search method for a composite media document according to claim 11, wherein a similarity of the representative among the similarities of the representatives is defined as a similarity at a document level of the media. 14. When performing a similarity search of a composite media document, each of the structure information of the document exemplified as a search key and the structure information of the stored document is stored in an order-labeled tree (ordered tree).
d labeled tree), and expresses the attribute information of each media in each document as an attribute-ordered tree with attributes stored as attributes of nodes in the order-labeled tree, and expresses the exemplified document. 8. The similarity of a composite media document according to claim 7, wherein the similarity of the composite media document is set by comparing the attribute and the shape of the labeled ordered labeled tree and the attributed labeled tree representing the stored document. retrieval method. 15. When performing a similarity search of a composite media document, a node having feature information similar to feature information as an attribute of each node of an attribute-labeled ordered tree expressing the document as an attribute is obtained. 15. The similarity search method for a composite media document according to claim 14, wherein the similarity of the composite media document is set based on a difference in the structural positional relationship between the nodes for the tree with the attributed ordered label expressing the stored document. 16. When setting the similarity of the structure information, the document structure is regarded as an ordered label tree, and the feature information is stored in a multidimensional vector space based on the feature information on the ordered labeled tree. The method according to claim 7, wherein the distance in the vector space is set as the similarity of the document. 17. A method for calculating a document similarity by numerically using the name of each node, the number of nodes, and the position information of each node as feature information on the ordered label tree, as feature information on the ordered label tree. Item 16. A similarity search method for a composite media document according to Item 16. 18. The feature information includes, for text information, the concept represented by the description content of the text and the appearance frequency of each word, and for image information, the hue, saturation, luminance, color arrangement, and voice information of the image. If it is, the strength and melody of the sound, if it is structural information, the tree shape, the label name of the node, the link information, etc. in the case where the document structure is represented by a tree with order labels, from the components of the composite media document 2. The method according to claim 1, wherein the feature information is extracted. 19. When determining the similarity, the similarity of a stored document with respect to the exemplified document includes a text, an image, and a sound constituting the exemplified document as the search key. An evaluation value based on a similarity determination result for each component of the information and structure information of the medium and the information and structure information of the medium including text, images, and voices that constitute the stored document, 2. The similarity search method for a composite media document according to claim 1, wherein the similarity of the stored documents is set as the overall document similarity, and the similarities of the stored documents are ranked in descending order to determine the similarity. 20. When setting the similarity, a similarity is set for each component of the composite media document, and characteristics of text information, image information, audio information, and structure information included in the document are set. For each piece of information, calculate the similarity, multiply the similarity by the value of the weight, and take the linear sum,
4. The similarity search method for a composite media document according to claim 3, wherein the similarity is set as the overall degree of the composite media document. 21. When setting the similarity of the entire composite media document, the similarity itself of each component or the similarity of each component is used as an evaluation value based on the similarity determination result for each component of the document. 4. The method according to claim 3, wherein the similarity is multiplied by a weight given by a user. 22. A composite media document that is a structured document composed of text information, image information, audio data as human voice data, music information including CDs and records, and audio information including music data. A similar search device, a composite media document input unit for inputting a composite media document, a document analysis unit for parsing the composite media document provided by the composite media document input unit and the input search condition, A feature information extraction unit that extracts feature information from a component of a document obtained as a result of analysis by the document analysis unit; a storage unit that stores the composite media document and the feature information extracted by the feature information extraction unit; Search condition input means for inputting a composite media document as a search condition; and the composite media stored in the storage means. The similarity between two composite media documents is determined based on the characteristic information of the media document and the characteristic information extracted by the characteristic information extracting means based on the result of analyzing the input search condition by the document analyzing means. A composite media document similarity search device, comprising: a document comparison unit that performs similarity; and an output unit that outputs a similar composite media document based on the similarity determined by the document comparison unit. 23. The search condition inputting means includes means for inputting a document exemplified by a user as a search key, and the feature information extracting means includes a compound information exemplified as a search key from the given composite media document. Input document feature information extracting means for extracting a media document; and search feature information extracting means for extracting feature information of the search information from a document exemplified by the user, wherein the document comparing means includes the input document feature information. A similarity calculating a similarity between the composite media document and the composite media document exemplified as the search key based on the input document feature information extracted by the extraction unit and the search feature information extracted by the search feature information extraction unit. 23. The composite media document similarity search device according to claim 22, further comprising a degree calculation unit. 24. The similarity calculating means, text information, image information, audio information, and structure information constituting the document exemplified as the search key by the search condition input means, and stored in the storage means. A similarity that sets an evaluation value based on a similarity determination result for each component with the text information, image information, audio information, and structural information constituting the composite media document as the overall similarity of the composite media document 23. The apparatus according to claim 21, further comprising a degree setting unit. 25. The similarity determination unit, as the similarity determination result for each of the components, the similarity determination result between the text information of the exemplified document and the text information of the document stored in the storage unit. A similarity determination result between the image information of the exemplified document, the image information of the document stored in the storage unit, the voice information of the exemplified document, and the document stored in the storage unit. 25. The similarity of the composite media document according to claim 24, wherein the similarity determination result with the audio information of the document and the similarity determination result between the structure information of the exemplified document and the structure information of the document stored in the storage unit are used. Search device. 26. A pre-similarity setting means calculates a similarity for each of text information, image information, audio information, and structural information included in the document, and multiplies the similarity by a weight value. 25. The composite media document similarity search device according to claim 24, further comprising a linear sum calculation unit that calculates a linear sum as a similarity of the composite media document as a whole. 27. The document comparison means according to claim 27, wherein, when a plurality of the same media exist in the same document, all of the search keys for each medium including text, image, and voice included in the exemplified document are stored. 24. A representative similarity determining unit that calculates all similarities of the media in the generated document with respect to the search key, and sets the highest similarity as the representative similarity to the search key. Similarity search device for composite media documents. 28. The similarity calculating means, as the search key, expresses each of the exemplified document structure information and the stored document structure information as an ordered labeled tree, Said ordered labeled tree representing said exemplified document;
24. The similarity of the composite media document according to claim 23, further comprising: an ordered-labeled tree shape comparing unit configured to set the similarity of the structural information of the document by comparing the shape with the ordered-labeled tree representing the stored document. Search device. 29. The order-labeled tree shape comparing means, when judging the similarity of the structure information, regards the document structure as a tree, and stores the document structure as a tree. The number of edit operations including insertion of nodes, deletion of nodes, and change of node names necessary for conversion to a tree representing a document, and the edit distance calculated from the cost required for performing the edit operations are described above. 29. The composite media document similarity search device according to claim 28, further comprising an edit distance calculating unit that sets the document similarity. 30. A means for calculating similarity based on feature information of text information, image information, audio information, and structure information included in a document, and a similarity calculation based on feature information of structure information. Means for performing a first-stage selection based on the result; and means for setting a similarity based on feature information of text information, image information, and audio information included in the document as a similarity of the entire composite media document 25. The apparatus for retrieving similarity of a composite media document according to claim 24. 31. A means for calculating similarity based on feature information of text information, image information, audio information, and structural information included in a document, the text information, the image information, and the audio. Means for selecting a first stage based on a result of similarity calculation based on feature information of information; means for setting a similarity based on feature information of structural information included in the document as a similarity for the entire composite media document 25. The composite media document similarity search apparatus according to claim 24, comprising: 32. The similarity calculating means includes a document level similarity calculating means for setting a similarity at a document level of the medium when a plurality of same media exist in a document exemplified as a search key. 25. The composite media document similarity search device according to claim 23. 33. The document-level similarity calculating means, for each search key of the medium, which is present in a plurality of documents exemplified as search keys, for text information, image information, and voice included in the exemplified document. Means for calculating, for all search keys for each medium including information, all similarities for the search key in the stored document in the medium; and determining the highest similarity for the representative similarity to the search key. 33. The composite media document according to claim 32, comprising: means for calculating the degree of similarity; means for calculating an average value of the similarity of the representative; and document level similarity setting means for setting similarity at the document level of the medium. Similarity search device. 34. The document-level similarity setting means, for each search key of the medium, which is present in a plurality of documents exemplified as search keys, for text information, image information, and audio included in the exemplified documents. Means for calculating, for all search keys for each medium including information, all similarities for the search key in the stored document in the medium; and determining the highest similarity for the representative similarity to the search key. 33. The composite media document similarity search device according to claim 32, further comprising: means for setting the degree of similarity; and means for setting the highest similarity among the representative similarities to the similarity at the document level of the medium. 35. The similarity calculating means according to claim 16, wherein each of the structural information of the document exemplified as the search key and the structural information of the stored document is ordered tree (ordered tree).
d labeled tree), and expresses the feature information of each medium in each document as an attribute-ordered tree stored as an attribute of a node in the order-labeled tree, and expresses the exemplified document. By comparing the attribute and shape of the labeled ordered tree with the attributed ordered labeled tree and the attributed ordered labeled tree representing the stored document,
29. The similarity search apparatus for a composite media document according to claim 28, further comprising a similarity search means for setting a degree of similarity of the composite document. 36. The similarity retrieving means retrieves a stored document having a node having, as an attribute, feature information similar to feature information that is an attribute of each node of an attribute-labeled ordered tree representing an exemplified document. The similarity search apparatus for a composite media document according to claim 35, further comprising means for setting a similarity degree of the composite media document from the difference in the structural positional relationship of the node with respect to the expressed ordered label tree with attributes. 37. The order-labeled tree shape comparing means regards the document structure as an order-labeled tree, and converts the feature information into a numerical value in a multidimensional vector space based on the feature information on the order-labeled tree. 31. The similarity search apparatus for a composite media document according to claim 30, further comprising: means for mapping and mapping; and means for setting the distance in the vector space as the degree of similarity of the document. 38. A method for calculating the similarity of a document by numerically using the name of each node, the number of nodes, and the position information of each node as the characteristic information on the ordered label tree. Item 37. A similar media document retrieval apparatus according to Item 37. 39. The feature information includes text information, the concept represented by the description content of the text, the appearance frequency of each word, and image information, the image hue, saturation, luminance, color arrangement, and audio information. Then, if it is sound strength, melody, or structural information, the tree structure, node label names, link information, etc., when the document structure is represented by an ordered labeled tree, are derived from the components of the composite media document. 23. The similar media document search apparatus according to claim 22, wherein the extracted feature information is extracted feature information. 40. The document comparison means according to claim 26, wherein the similarity of the stored document to the exemplified document is determined based on information of media including texts, images, and sounds constituting the document exemplified as the search key. The evaluation value based on the similarity determination result for each component of the structural information and the stored information of the media including text, images, and sounds constituting the document and the structural information is compared with the entire composite media document. 23. The similarity search apparatus for a composite media document according to claim 22, comprising: means for setting the degree of similarity; and means for determining the degree of similarity by arranging the similarities of the stored documents in descending order and determining the degree of similarity. 41. The similarity setting means: means for setting a similarity for each component of the composite media document; and features of text information, image information, audio information, and structure information included in the document. Means for calculating a similarity for each piece of information; and a value obtained by multiplying the similarity by a weight value and taking a linear sum,
25. The apparatus for retrieving similarity of a composite media document according to claim 24, further comprising: means for setting the degree of similarity as the entire composite media document. 42. The similarity degree setting means, as an evaluation value based on a similarity determination result for each element of the document, gives a similarity degree of each element itself or a similarity degree of each element from a user. 25. The similar media document retrieval apparatus according to claim 24, further comprising means for using the product obtained by multiplying the weights. 43. A composite media document which is a structured document composed of text information, image information, voice data as human voice data, and music information including CDs and records and voice information including music data. A storage medium storing a similar search program, wherein the given composite media document and a document analysis process for parsing the input search condition; and a configuration of a document obtained as a result of the analysis by the document analysis process A feature information extraction process for extracting feature information from an element, a storage process for storing the composite media document and the feature information extracted in the feature information extraction process in a storage unit, and a search for inputting the composite media document as a search condition A condition input process; characteristic information of the composite media document stored in the storage means; A document comparison process of determining a similarity between two composite media documents from feature information extracted in the feature information extraction process based on a result of analyzing the retrieved search conditions in the document analysis process; Based on the determined similarity,
An output process for outputting a similar composite media document. A storage medium storing a similar search program for a composite media document. 44. The search condition input process includes a process of inputting a document exemplified by a user as a search key, and the feature information extraction process extracts feature information from a given composite media document. A feature information extraction process for extracting feature information of a composite media document exemplified as a search key from a document exemplified by the user; a document comparison process including the input document feature information extraction A similarity calculating a similarity between the composite media document and the composite media document exemplified as the search key based on the input document characteristic information extracted in the process and the search characteristic information extracted in the search characteristic information extraction process. The storage storing the composite media document similarity search program according to claim 43 including a calculation process. Medium. 45. The similarity calculation process includes: text information, image information, audio information, and structure information constituting a document exemplified as the search key in the search condition input process; Setting an evaluation value based on a similarity determination result for each component with the text information, image information, audio information, and structural information constituting the composite media document as the similarity of the entire composite media document 44. The method of claim 43, including a setting process.
45. A storage medium storing the similarity search program for composite media documents according to 44. 46. The similarity determination process, wherein the similarity determination result for each of the constituent elements includes: a similarity determination result between the text information of the exemplified document and the text information of the document stored in the storage unit. A similarity determination result between the image information of the exemplified document, the image information of the document stored in the storage unit, the voice information of the exemplified document, and the document stored in the storage unit. 47. The similarity of the composite media document according to claim 45, wherein the similarity determination result with the audio information of the above and the similarity determination result between the structure information of the exemplified document and the structure information of the document stored in the storage unit are used. A storage medium storing a search program. 47. A pre-similarity setting process calculates a similarity for each of text information, image information, audio information, and feature information of structural information included in the document, and multiplies the similarity by a weight value. 46. The storage medium storing the similarity search program for a composite media document according to claim 45, further comprising a linear sum calculation process of taking a linear sum as a similarity as the entire composite media document. 48. The document comparison process according to claim 28, wherein, when a plurality of the same media exist in the same document, all the search keys for each media including text, image, and sound included in the exemplified document are stored. 45. A representative similarity determination process comprising calculating all similarities for the search key in the media in the generated document and setting the highest similarity as the representative similarity to the search key. Storage medium storing a similarity search program for composite media documents. 49. The similarity calculation process, wherein, as the search key, each of the exemplified document structure information and the stored document structure information is represented as an ordered labeled tree, An ordered label tree that sets the similarity of document structure information by comparing the shapes of the ordered labeled tree representing the exemplified document and the ordered labeled tree representing the stored document. 46. A storage medium storing a composite media document similarity search program according to claim 44 including a shape comparison process. 50. The order-labeled tree shape comparison process, when determining the similarity of the structure information, considers the document structure as a tree and stores the document structure as a tree representing the exemplified document. The number of edit operations including insertion of nodes, deletion of nodes, and change of node names necessary for conversion to a tree representing a document, and the edit distance calculated from the cost required for performing the edit operations are described above. 50. A storage medium storing a similarity search program for a composite media document according to claim 49, comprising an edit distance calculation process for setting the similarity of the document. 51. The similarity calculation process includes a process of calculating a similarity based on text information, image information, audio information, and feature information of structure information included in a document; and a process of calculating a similarity based on feature information of structure information. A process of selecting a first stage based on the result, and a process of setting a similarity based on feature information of text information, image information, and audio information included in the document as a similarity of the entire composite media document. A storage medium storing the composite media document similarity search program according to claim 44. 52. The similarity calculation process includes: a process of calculating a similarity based on text information, image information, audio information, and feature information of structure information included in a document; and a process of calculating the similarity based on the text information, the image information, and the audio. A process of selecting a first stage based on a result of similarity calculation based on feature information of information, and a process of setting a similarity based on feature information of structural information included in the document as a similarity of the entire composite media document A storage medium storing the composite media document similarity search program according to claim 43 or claim 44. 53. The similarity calculation process includes a document level similarity calculation process for setting a similarity at a document level of a medium when a plurality of same media exist in a document exemplified as a search key. Claim 4
45. A storage medium storing the similar media program search program according to 3 or 44. 54. The document-level similarity calculation process includes, for each search key of the medium that is present in a plurality of documents exemplified as a search key, text information, image information, and sound included in the exemplified document. A process of calculating all similarities for the search key in the stored document for the media for all search keys for each medium including information; and determining the highest similarity for the representative similarity to the search key. 54. The composite media document of claim 53, comprising: a process of calculating the degree of similarity; a process of calculating an average value of the similarity of the representative; and a document level similarity setting process of setting a similarity of the medium at a document level. A storage medium storing a similar search program. 55. The document-level similarity setting process includes, for each search key of the medium, which is present in a plurality of documents exemplified as search keys, for text information, image information, and voice included in the exemplified documents. A process of calculating all similarities for the search key in the stored document for the media for all search keys for each medium including information; and determining the highest similarity for the representative similarity to the search key. 55. The similarity search program for a composite media document according to claim 54, further comprising: a process of setting the highest similarity among the representative similarities to a similarity at a document level of the media. Storage media. 56. The similarity calculation process includes the steps of: adding each of structural information of a document exemplified as a search key and structural information of a stored document to an order-labeled tree (ordered tree);
d labeled tree), and expresses the feature information of each medium in each document as an attribute-ordered tree stored as an attribute of a node in the order-labeled tree, and expresses the exemplified document. By comparing the attribute and shape of the labeled ordered tree with the attributed ordered labeled tree and the attributed ordered labeled tree representing the stored document,
50. The storage medium storing a similarity search program for a composite media document according to claim 49, comprising a similarity search process for setting a similarity of the composite document. 57. The similarity search process includes the steps of: executing a stored document having a node having attribute information similar to characteristic information which is an attribute of each node of an attribute-labeled ordered tree representing an exemplified document; 57. A storage medium storing a similarity search program for a composite media document according to claim 56, comprising a process of setting the similarity of the composite media document from the difference in the structural positional relationship between the nodes with respect to the represented ordered label tree with attributes. . 58. The order-labeled tree shape comparison process considers the document structure as an order-labeled tree, and converts the feature information into a numerical value in a multidimensional vector space based on the feature information on the order-labeled tree. 50. The storage medium according to claim 49, further comprising: a process of mapping and mapping; and a process of setting the distance in the vector space as a document similarity. 59. A method for calculating the similarity of a document by numerically using the name of each node, the number of nodes, and the position information of each node as the characteristic information on the ordered label tree. 59. A storage medium storing the composite media document similarity search program according to item 58. 60. The feature information, if it is text information, the concept represented by the description content of the text and the appearance frequency of each word, and if it is image information, the hue, saturation, luminance, color arrangement, and voice information of the image Then, if it is sound strength, melody, or structural information, the tree structure, node label names, link information, etc., when the document structure is represented by an ordered labeled tree, are derived from the components of the composite media document. 44. A storage medium storing a similar search program for a composite media document according to claim 43, wherein said program is a feature information to be extracted. 61. The document comparison process, wherein the similarity of a stored document to the exemplified document is determined by using information on media including text, images, and audio constituting the exemplified document as the search key. The evaluation value based on the similarity determination result for each component of the structural information and the stored information of the media including text, images, and sounds constituting the document and the structural information is compared with the entire composite media document. 44. A process for setting the degree of similarity as a degree, and a step of determining a degree of similarity by arranging similarities of the stored documents in descending order.
A storage medium storing a similar search program for the described composite media document. 62. The similarity setting process includes: a process of setting a similarity for each component of the composite media document; and features of text information, image information, audio information, and structure information included in the document. For each piece of information, a process of calculating the similarity, and multiplying the similarity by a weight value and taking a linear sum,
45. A storage medium storing a composite media document similarity search program according to claim 44, further comprising: a process of setting the similarity as the entire composite media document. 63. The similarity setting process, wherein a user gives the similarity of each component or the similarity of each component as an evaluation value based on the similarity determination result for each component of the document. 45. The storage medium storing the composite media document similarity search program according to claim 44, further comprising a process of utilizing the product obtained by multiplying the weights by the obtained weight.