JP2013239132A - Similarity computation device, similar program search device, and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a similarity computation device capable of properly evaluating similarity between texts even in the presence of inconsistent use of words therein, and a similar program search device.SOLUTION: A similarity computation device includes: an expression extraction unit which derives expression trend information, representing appearance trend of types of expressions, by extracting expressions from text data; an expression amplification unit which amplifies the expression trend information, extracted by the expression extraction unit, by extracting related expressions associated with the expressions extracted by the expression extraction unit by using a relationship between expressions and related expressions, and generates amplified expression trend information; and a similarity computation unit which computes similarity between first and second text data based on a relationship between first amplified expression trend information, output by the expression amplification unit based on the first text data entered to the expression extraction unit, and second amplified expression trend information output by the expression amplification unit based on the second text data entered to the expression extraction unit.

Description

  The present invention relates to natural language processing. In particular, the present invention relates to a similarity calculation device that calculates the similarity between texts.

  There is a technique for evaluating similarity using an outline of a program related to content when searching for other video content similar to a certain video content (such as a broadcast program). The basis of this technology is a method of evaluating the similarity between texts written in natural language with objective numerical values. For example, Patent Literature 1 discloses a technique for extracting a word used in a program summary sentence and evaluating similarity by using the degree of overlap between two extracted word sets as an index. In addition, the summary sentence regarding a broadcast program can be acquired from the data of EPG (electronic program guide) transmitted with a broadcast wave, for example.

JP 2011-043908 A

  However, in the prior art described in Patent Document 1 and the like, the similarity is evaluated only by the appearance state of the surface word. In other words, in the prior art, only words with the same notation on the surface are judged to match, and words that are similar but have different notation are judged to be different words. Sometimes it was not enough. Similarity may not be legitimately evaluated when the words used are also shaken. A typical example is the use of shaking in the words used. For example, “earthquake” and “earthquake” are similar words, but these two words were treated as different words on the surface. In addition, regarding the summary sentences of broadcast programs such as television and other general texts, if the total length of the text is short, the amount of words appearing on the surface layer is insufficient, and the text length is too short. Because of its short length, the tendency of appearance of words tends to occur statistically.

  For these reasons, there is a problem that the similarity between texts may not be properly evaluated due to fluctuations in notation.

  The present invention has been made in view of the circumstances as described above. When the text length is short or the notation included in the text is shaken (that is, biased to the tendency of the expression appearing on the surface layer). There is also provided a similarity calculation device that can legitimately evaluate the similarity between texts. Further, there is provided a similar program search device that searches a program similar to a specific program from a large number of programs using such a similarity calculation device.

  In order to solve the above problem, the present invention uses a relationship between language expressions acquired in advance (here, a relationship is, for example, a similarity relationship, a causal relationship, an upper-lower relationship, an attribute relationship, etc.) Use a means to amplify the expression tendency. Also, a means for calculating the similarity of the original text by comparing the expression tendencies after amplification is used. In addition, in order to improve the accuracy of the expression tendency after amplification, a filtering unit may be used to prevent an element having low similarity with the original text from affecting the expression tendency after amplification. . A plurality of modes using these means are as follows.

  [1] A similarity calculation apparatus according to an aspect of the present invention relates to an expression extraction unit that obtains expression tendency information representing an appearance tendency of the expression type by extracting an expression from text data, and the expression and the expression. Amplifying the expression tendency information extracted by the expression extraction unit by extracting the related expression related to the expression extracted by the expression extraction unit using the relationship with the related expression, and the amplified expression tendency information The expression amplifying unit for outputting, the first post-amplification expression tendency information output from the expression amplifying unit based on the first text data input to the expression extracting unit, and the input to the expression extracting unit The first text data and the second text data are based on the relationship between the second amplified expression tendency information output from the expression amplifying unit based on the second text data. Comprising a similarity calculating section for calculating a degree of similarity between the.

Here, expression is an element of expression in the original language. Expressions include, for example, characters, words, clauses, N-grams of characters (N chains), N-grams of words, and the like.
The expression tendency information is data representing what expression tends to appear in certain text data. For example, data that lists all the expressions that appear in certain text data is expression tendency information. Further, for example, data representing the appearance frequency (number of times) for each expression appearing in certain text data is expression tendency information. As a specific example, all the words appearing in a certain text data are extracted and listed by separating them with a comma “,” or the like is expression tendency information.
Amplifying the expression tendency information is a data operation as follows. That is, the expression included in the expression tendency information is maintained in the space of the information indicating the tendency of the expression to appear, while maintaining the original expression tendency, or keeping the blur from the original expression tendency information within the predetermined range. It is to increase the number of types or increase the value of the appearance frequency of the expression as a whole.
According to said structure, the similarity calculation part is based on the relationship between 1st and 2nd post-amplification expression tendency information, and the similarity between said 1st text data and said 2nd text data Is calculated. That is, as the similarity calculated by the similarity calculation unit, a value excluding the influence of the shortness of the original text data and the fluctuation of the notation appearing on the surface layer can be obtained.

  [2] Further, according to one aspect of the present invention, in the above similarity calculation apparatus, the expression amplifying unit amplifies the expression tendency information by extracting a related expression related to the extracted related expression. The expression amplifying unit includes an expression included in the first post-amplification expression tendency information amplified based on the first text data, and the second post-amplification amplified based on the second text data. Relevance representing whether or not there is a relation between nodes with respect to a node corresponding to the expression included in the expression tendency information, a node corresponding to the first text data, and a node corresponding to the second text data Generating similarity data, and the similarity calculation unit calculates the strength of the relationship between the first text data and the second text data calculated based on the relationship data. Calculated as acetonide, characterized in that.

  With this configuration, the expression amplifying unit extracts a related expression related to the expression extracted by the expression extracting unit and also extracts a related expression related to the related expression. In other words, the expression amplifying unit may recursively extract related information. The relevance data is typically expressed as data having a graph structure indicating the presence or absence of a relationship between nodes. In the relevance data, the presence or absence of a relationship between two specific nodes is represented by the presence or absence of an edge between the two nodes. Based on the above configuration, in general, there is a common node between a node obtained as a result of amplification based on the first text data and a node obtained as a result of amplification based on the second text data. Can exist. This includes related expressions related to expressions included in the first text data (including related expressions obtained as a result of multi-level relationships) and related expressions related to expressions included in the first text data (similarly). ) Is the same thing as the same expression. That is, when the relevance data is represented by a graph structure, a node corresponding to the first text data and a node corresponding to the second text data are connected via zero or more nodes. Further, since the first and second text data each include a plurality of expressions, in general, there are a plurality of nodes corresponding to the first text data and a plurality of nodes corresponding to the second text data. It can be in a state of being connected through the path. Then, the similarity calculation unit, based on such relevance data, in other words, the relevance between the first text data and the second text data according to the connection mode between the nodes in the graph structure. Is calculated as a similarity.

  [3] Further, according to one aspect of the present invention, in the similarity calculation apparatus according to [2], the similarity calculation unit is data representing ease of transition from node to node based on the association data. Is the relevance score between the nodes, and the relevance from the node corresponding to the first text data to all the nodes included in the path from the node corresponding to the second text data The sum total of the scores is calculated as the similarity.

As one aspect, using a transition matrix whose element is the relationship between nodes (transition probability value), random transitions between nodes based on the relationship (in other words, based on random variables) are repeated multiple times. The transition probability value from a certain node to another node at that time is taken as the strength of relevance between the nodes. Furthermore, as one aspect thereof, the convergence value of the transition probability from one node to another node when the transition between the nodes as described above is repeated infinitely is set as the strength of relevance between the nodes.
Also, the sum of the respective relevance scores from the node corresponding to the first text data to all the nodes included in the path corresponding to the second text data is the similarity between the two text data. By calculating as a degree, good results are obtained.

  [4] Further, according to one aspect of the present invention, in the similarity calculation apparatus according to [2], the similarity calculation unit is data representing ease of transition from node to node based on the association data. And calculating the relevance score from the node corresponding to the first text data to the node corresponding to the second text data as the similarity. Features.

  In this configuration, the relevance score from the node corresponding to the first text data to the node corresponding to the second text data is calculated as the similarity between the text data. In this case, the amount of calculation for calculating the similarity can be reduced.

  [5] Further, according to one aspect of the present invention, in the similarity calculation apparatus according to [1], the similarity calculation unit is configured to use the expression amplification unit based on first text data input to the expression extraction unit. Between the first post-amplification expression tendency information output from and the second post-amplification expression tendency information output from the expression amplification section based on the second text data input to the expression extraction section. The first degree of expression tendency information after amplification includes an expression included in the first text data and an associated expression directly related to the expression included in the first text data. Information representing a distribution, and the second post-amplification expression tendency information includes a distribution of an expression included in the second text data and a related expression directly related to the expression included in the second text data. It is information that represents To.

  The similarity calculation device with this configuration is based on the configuration of the similarity calculation device described in [1]. Therefore, the related expression extraction unit extracts the related expression “using the relationship between the expression and the related expression related to the expression”. The expression amplifying unit amplifies the expression tendency information by the extracted related expression. As an aspect, the expression tendency information after amplification is obtained by adding a list of related expressions to the expression tendency information extracted by the expression extraction unit. As another example, the obtained related expression list itself is used as the expression tendency information after amplification.

  [6] Further, according to one aspect of the present invention, in the similarity calculation apparatus according to [5], the expression amplifying unit has a predetermined similarity between the extracted related expressions and the text data. The apparatus further comprises a related expression filtering unit that excludes the related expressions that are less than a threshold value.

  With this configuration, the related expression filtering unit excludes the related expressions that have a weak relationship with the text data. As a result, the accuracy of the post-amplification expression tendency information amplified using the related expression is further increased. As one aspect, the related expression filtering unit is configured such that each of the related expressions extracted by the related expression extracting unit is an expression included in the original text data other than an expression having a direct relationship with the related expression. It is determined whether or not to exclude the related expression depending on whether or not the similarity between them is higher than a predetermined threshold.

  [7] Further, a similar program search device according to an aspect of the present invention includes the similarity calculation device described above, and the program summary sentence text data related to the designated program is used as the first text data in the similarity calculation device. The program summary sentence text data relating to another program is input to the similarity calculation device as the second text data, and the designated program and the other program calculated by the similarity calculation device are input. And the specified program is selected from among the plurality of other programs based on the similarity between the specified program and the specified program calculated for the plurality of the other programs. The similar other program is searched.

  [8] One aspect of the present invention relates to an expression extraction unit that obtains expression tendency information representing an appearance tendency of the expression type by extracting an expression from text data, and the expression and the expression. Amplifying the expression tendency information extracted by the expression extraction unit by extracting the related expression related to the expression extracted by the expression extraction unit using the relationship with the related expression, and the amplified expression tendency information The expression amplifying unit for outputting, the first post-amplification expression tendency information output from the expression amplifying unit based on the first text data input to the expression extracting unit, and the input to the expression extracting unit The first text data and the second text based on the relationship between the second amplified expression tendency information output from the expression amplifier based on the second text data Is a program which functions as a similarity calculation anda similarity calculating section for calculating a degree of similarity between the over data.

  According to the present invention, it is possible to legitimately evaluate the similarity between text data even when the text data is not sufficiently long or when the expression appearing on the surface layer of the text data is biased. Further, by applying such a legitimate evaluation to a summary sentence of a broadcast program or the like, it becomes possible to legitimately evaluate the similarity between programs. Then, it becomes possible to search for another program to be compared based on the legitimate similarity to the designated program.

It is a block diagram which shows the function structure of the similar program search apparatus by the 1st Embodiment of this invention. It is the schematic which shows the example of a data structure of the program information storage part by the embodiment. It is the flowchart which showed the procedure of the whole process of the similar program search apparatus by the embodiment. It is a flowchart which shows the procedure of the process which the related word filtering part by the embodiment filters a related word. It is a block diagram which shows the structure of the similar program search apparatus by the 2nd Embodiment of this invention. It is the schematic which shows the example of data of the graph which the related graph construction part by the same embodiment builds. It is the schematic which shows the range of the relevance score used when calculating the similarity between document nodes using the "method 1" in the graph data which the related graph construction part by the same embodiment constructs. It is the schematic which shows the range of the relevance score used when calculating the similarity between document nodes using the "method 2" in the graph data which the related graph construction part by the same embodiment builds.

[First Embodiment]
An embodiment of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing a functional configuration of a similar program search apparatus according to this embodiment. As shown in the figure, the similar program search device 2 includes a similarity calculation device 1, a program information storage unit 11, a ranking unit 17, and a search result output unit 18. Also, the similarity calculation device 1 includes a document reading unit 12, a word extraction unit 13 (expression extraction unit), a related word extraction unit 14 (related expression extraction unit), and a related word filtering unit 15 (related expression filtering unit). And a similarity calculation unit 16. A combination of the related word extraction unit 14 and the related word filtering unit 15 is an expression amplification unit 25.

The similarity calculation device 1 calculates the similarity between texts.
The program information storage unit 11 stores information related to programs such as broadcasting. The program information storage unit 11 stores data (video, audio, etc.) of program content itself and text data of a program summary sentence describing the program. As the program information storage unit 11 and other information storage means, a magnetic disk device, a semiconductor memory, or the like is used.
The document reading unit 12 sequentially reads the text of the program summary sentence stored in the program information storage unit 11.

  The word extraction unit 13 obtains expression tendency information representing the appearance tendency of the type of expression by extracting a word (expression) from the text data. Specifically, the word extraction unit 13 performs a morphological analysis process on the text of the program summary sentence read by the document reading unit 12 and extracts words (expressions such as nouns and verbs) included in the program summary sentence. The list of words extracted by the word extraction unit 13 (for example, data listing words separated by commas) is the above-described expression tendency information. The morphological analysis process can be executed by using existing technology. For example, a morphological analyzer MeCab (Yet Another Part-of-Speech and Morphological Analyzer) can be used (reference URL: http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html).

  The related word extracting unit 14 extracts a related word (related expression) for the nouns among the words extracted by the word extracting unit 13. For this purpose, the related word extracting unit 14 acquires a relationship between words in advance and stores it in an internal storage unit (not shown). Here, the relationship between words includes a similarity relationship, a causal relationship, an upper-lower relationship, an attribute relationship, a relationship between a certain thing and its material, a relationship between a certain place and a special product at that place, etc. . In order to acquire the relationship between words in advance, for example, an inter-word relationship acquisition tool is used based on text obtained from the web. The inter-word relationship acquisition tool itself can be realized using existing technology. As an example, the upper and lower relationship is a relationship that holds between X and Y when it can be said that “X is Y” or “X is a kind of Y” in Japanese. It is a subordinate concept, and Y is a superordinate concept. In order to acquire words of upper and lower relations, the technique described in the reference URL [http://alaginrc.nict.go.jp/hyponymy/] can be used. In this method, data in XML (Extensible Markup Language) format is acquired from the web encyclopedia, etc. via the Internet, and a pair of a high-order concept word and a low-order concept word is obtained using the result of machine learning in advance. To win. This approach can capture millions of relationships. In addition, the semantic relationship extraction tool is used to acquire various semantic relationships such as a causal relationship, a relationship between a place and a specialty, and a relationship between a musician and a song title. The semantic relation extraction tool itself uses existing technology as described in the reference URL [http://alaginrc.nict.go.jp/resources/nictmastar/resource-info/abstract.html]. Can be obtained. Instead of using these tools, the relationship between words may be acquired and accumulated manually and used in the related word extraction unit 14.

  The related word filtering unit 15 filters related words (nouns) acquired by the related word extracting unit 14. In other words, the related word filtering unit 15 has the related word extracted by the related word extracting unit 14 and the similarity to the program summary sentence is less than a predetermined threshold (that is, the similarity is low). Is excluded. That is, the related word filtering unit 15 deletes related words that are not related to the context of the original program summary sentence from the related words acquired by the related word extracting unit 14. Details of this filtering process will be described later.

  That is, the expression amplifying unit 25 including the related word extracting unit 14 and the related word filtering unit 15 has a predetermined relationship (similarity relationship) between the word (expression) and the word and the related word (related expression). Is used to amplify the expression tendency information extracted by the word extraction unit 13 and output the amplified expression tendency information. That is, based on the expression tendency information extracted by the word extraction unit 13, among the related words extracted by the related word extraction unit 14, a list of related words that are not excluded by the related word filtering unit 15 is added. It is expression tendency information after amplification.

Similarity calculating unit 16, the related word filtering obtained in relation to the program summary statement d ₀ result, the related word filtering obtained in relation to each of the other program outline sentence d ₁ to d _M Calculate the similarity between the results. This similarity calculated by the similarity calculation unit 16 represents the similarity between the program summary sentence d ₀ and each of the program summary sentences d ₁ to d _M.

Ranking unit 17, the similarity calculating unit 16 is calculated, the program summary statement d _0, based from program summary statement d ₁ on the similarity between the respective up d _M, the d _M from program summary statement d ₁ Ranking and output. This ranking, that is, a ranking of program content corresponding to d _M from each of the program outline statement d _1.

Search result output unit 18, ranking information ranking unit 17 outputs (ranked as the score of each program (similarity) is also included), and similar to the designated program (corresponding to the program summary sentence d ₀₎ search Output as a result. The ranking unit outputs the search result in a text data format or HTML (Hypertext Markup Language) data format.

FIG. 2 is a schematic diagram illustrating an example of a data configuration of the program information storage unit 11. As shown in the drawing, the program information storage unit 11 has a tabular data structure and includes items of a program summary sentence and program content. One row of this table corresponds to one program. The item of the program summary sentence stores text data representing the program contents and the like. The program content item stores video data for a television program, audio data for a radio program, and the like. In the illustrated example, the program information storage unit 11 stores (M + 1) program summary sentences d ₀ , d ₁ ,..., D _M.

Next, the operation of the similar program search device 2 will be described.
FIG. 3 is a flowchart showing a processing procedure of the similar program search device 2. Hereinafter, it demonstrates along this flowchart.

First, in step S01, the document reading unit 12 reads one program summary sentence (d ₀ ) designated by the user or the like from the program summary sentences stored in the program information storage unit 11. Specific examples of the program summary text read here are as shown in Table 1 below. As shown in Table 1, the program summary sentence is text data, and includes a program title and subtitle, and a summary sentence explaining the program contents.

  Next, in step S02, the word extraction part 13 extracts the word contained in said program summary sentence. At this time, the part of speech of the extraction target word may be a plurality of combinations such as (1) noun only, (2) noun and verb, and (3) noun, verb and adjective. Note that particles, auxiliary verbs, and the like do not often represent the characteristics of sentences and are therefore not included in the extraction target. Here, the word list when only nouns are extracted is shown in Table 2 below.

  Next, in step S03, the related word extraction unit 14 extracts words (related words) related to the word extracted by the word extraction unit 13. Here, as the related words, various relationships such as a high-order relationship, a causal relationship, a relationship between a place and a specialty, a relationship between an object and its material can be used. When the related word extraction unit 14 extracts related words, a relationship limited to only one type (for example, only the upper and lower relationship) may be used, or a combination of a plurality of types of relationships (for example, a higher level) Subordinate relations, causal relations, relations between places and specialties, relations between objects and their materials) may all be used. Tables 3 to 6 show examples of a part of related words when all the relationships are extracted from the word groups listed in Table 2.

  Table 3 is a list of related words extracted by upper and lower relations. In Table 3, only some of the related words related to the words “Kusaya fungus”, “secret”, “Niijima”, “dried fish”, and “studio park” are shown, and others are omitted.

  Table 4 is a list of related words extracted by the causal relationship. In Table 4, only some of the related words related to the words “salt”, “microorganism”, and “salt water” are shown, and the others are omitted.

  Table 5 is a list of related words extracted by the relationship between a certain thing and its material. In Table 5, only some of the related words related to the words “fish”, “dried fish”, and “salt” are shown, and the others are omitted.

  Table 6 is a list of related words extracted based on the relationship between a place and a specialty of the place. In Table 5, only some of the related words related to the word “Tokyo” are shown, and the others are omitted.

  Next, in step S04, the related word filtering unit 15 evaluates the relationship between the related word extracted by the related word extracting unit 14 and the original program summary sentence, so that the context in the program summary sentence is Exclude irrelevant things. The detailed procedure of this filtering process will be described later with reference to another drawing.

The processing from the next steps S05 to S08 is the same as the processing from steps S01 to S04 already described. However, while the subject of the processing from steps S01 to S04 was program summary statement _{d 0,} the target of processing from step S05 to S08, except _{d 0,} that is, either from _{d 1} to _{d M} It is.

In the next step S09, the similarity calculation unit 16 calculates the similarity between program summary sentences. Specifically, the similarity calculating unit 16, a list of words that have been extracted by the word extraction section 13 from the program summary statement d ₀ of the designated target program, the program summary statement (d ₁ of the program to be compared and enter a list of words that have been extracted by the word extraction section 13 from one) to d _M. In addition, the similarity calculation unit 16 extracts based on the related word filtering result extracted based on the program summary sentence d ₀ and the program summary sentence (any _one from d ₁ to d _M ) of the program to be compared. The related word filtering result is also input. And the similarity calculation part 16 calculates the similarity between both word sets using the method by an existing technique. Various methods can be used as a method of calculating the similarity, and as an example, a method of calculating the cosine similarity between word appearance vectors representing the appearance status of words can be used. The outline of this method is as follows. That is, when the number of types of related words extracted based on a document is N (N is, for example, a value of about tens of thousands to hundreds of thousands), the i-th (1 ≦ i ≦ N) of them An N-dimensional vector having the value of the appearance frequency of related words as an element is obtained. Here, the appearance frequency may be limited to any value of 0 or 1, or may be a value obtained by counting the number of appearances in the list of related words. The cosine similarity between the _two N-dimensional vectors v ₁ and v ₂ obtained in this way is calculated by the following equation. Note that the word extracted based on the program summary sentence d ₀ and its related word (first amplified expression tendency information) correspond to v ₁ and extracted based on any of the program summary sentences d _{1 to} d _M. by words and related words that (second post-amplification expressed tendency information) corresponding to v _2.

(Cosine similarity) = (v ₁ · v ₂ ) / (| v ₁ | · | v ₂ |)

However, in the above equation, (v ₁ · v ₂ ) in the numerator is the inner product of both vectors. Further, | v ₁ | and | v ₂ | in the denominator are norms (sizes) of the respective vectors.

  And the similarity calculation part 16 stores the similarity calculated regarding the present comparison object program in a database (not shown).

In step S10, the similar program search device 2 determines whether or not the processing from steps S05 to S09 has been completed for all of the program summary sentences d _{1 to} d _M. If it has been completed (step S10: YES), the process proceeds to the next step S11. If it has not been completed (step S10: NO), the process returns to step S05 to process an unprocessed program summary sentence.

When the processing proceeds to step S11, the ranking unit 17 ranks the program summary sentences d _{1 to} d _M in descending order of similarity based on the similarity calculated by the similarity calculation unit 16. This has the same meaning as ranking program contents respectively corresponding to the program summary sentences d _{1 to} d _M. Further, the search result output unit 18 outputs the program summary sentences in a ranking format based on the similarity information and ranking information for each program-related sentence output from the ranking unit 17. The search result output unit 18 outputs, for example, a ranking page in an HTML format so that the user can access the program content data by following the link from the page.

  FIG. 4 is a flowchart showing a detailed procedure of filtering processing by the related word filtering unit 15. Hereinafter, it demonstrates along this flowchart.

First, in step S <b> 21, the related word filtering unit 15 selects one of the related words extracted by the related word extracting unit 14.
Next, in step S22, the related word filtering unit 15 provisionally sets the related word selected in step S21 as an exclusion target.
Next, in step S23, the related word filtering unit 15 selects one unprocessed word among the words in the program summary sentence. However, at this time, words in the program summary sentence that are directly related to the related word selected most recently in step S21 are excluded. In other words, the related word filtering unit 15 excludes words directly related to the related word selected most recently in step S21 from the word list extracted from the program summary sentence by the word extracting unit 13, and adds the remaining words. Select one of them.

  Next, in step S24, the related word filtering unit 15 determines whether the similarity between the related word selected in step S21 and one word in the program summary sentence selected in step S23 is greater than or equal to a predetermined threshold. judge. When the similarity between the two words is equal to or greater than the threshold (that is, the degree of similarity is high) (step S24: YES), the process proceeds to the next step S25, where the similarity is less than the threshold (that is, similar). If it is low) (step S24: NO), the process jumps to step S26.

Here, the similarity evaluation between words can use an index such as an existing distribution similarity. As an example, a large number of types of words are clustered based on the appearance frequency in actual text or the like, and the distribution of the probability of belonging to a class of words is obtained from the clustering result. The similarity between words can be calculated from the distance between the probability distributions. For example, the following literature can be referred to for word clustering based on dependency.
References: Shinichi Kazama, Stijn De Saeger, Kentaro Torizawa, Maki Murata, “Creating a Large-scale Similar Words List Using Dependent Stochastic Clustering”, 15th Annual Conference of the Language Processing Society, pp. 84-87 , 2009

When it progresses to step S25, the related word filtering part 15 removes the related word selected in step S21 from exclusion object.
Next, in step S26, the related word filtering unit 15 determines whether or not the processing from steps S23 to S25 has been completed for all words in the program summary sentence. If all the words in the program summary sentence have been processed (step S26: YES), the process proceeds to the next step S27. If all the words in the program summary sentence have not been processed yet (step S26: NO), the process returns to step S23 to process the next word.

  Next, in step S27, the related word filtering unit 15 determines whether or not the processing from steps S21 to S26 has been completed for all the related words output from the related word extracting unit 14 regarding the program summary sentence. When the process for all related words is completed (step S27: YES), the process of the entire flowchart is terminated. If all the related words have not been processed and there are still related words (step S27: NO), the process returns to step S21 for processing the next related word.

  That is, when the related word selected in step S21 is determined to be equal to or higher than the predetermined threshold in step S24 for one or more of the words selected in step S23, the related word is It remains without being excluded from the list of related words. On the other hand, if the related word selected in step S21 does not have a degree of similarity equal to or higher than a predetermined threshold with any of the words selected in step S23, the related word is excluded from the list of related words. The As a result, related words that do not match the context of the original program summary sentence are excluded during amplification by related words. Therefore, the accuracy in calculating similarity is increased.

  As an example, the word “secret” included in the program summary sentence shown in Table 1 is used in the expression “secret taste” to attract the interest of why the unique taste of dried fish “Kusaya” occurs. It has been. On the other hand, in Table 3, words having a high-order subordinate relationship with this word “secret” are listed. Among the words “secret”, for example, related words such as “release work” and “opening commemoration program” are not less than a predetermined threshold with words other than “secret” included in the program summary sentence of Table 1. There is no similarity. Therefore, the related word filtering unit 15 excludes these related words such as “release work” and “opening commemorative program” as a result of the filtering process shown in FIG. That is, the excluded related words are not used for calculating the similarity between program summary sentences by the similarity calculating unit 16.

As described above, the similar program search device 2 inputs the program summary sentence text data (d ₀ ) related to the designated program to the similarity calculation device 1 as the first text data, and the program summary related to other programs. Each of the sentence text data (d _{1 to} d _M ) is input to the similarity calculation device 1 as second text data. Then, the similarity between the specified program and the other program calculated by the similarity calculation device 1 is acquired, and between the specified program calculated for a plurality of the other programs. Searching for another program similar to the specified program from among the other programs based on the similarity of

[Second Embodiment]
Next, a second embodiment will be described. In addition, about the function which is common in embodiment mentioned above, the same code | symbol is provided in drawing. Further, description of matters common to the above-described embodiment will be omitted, and description will be made centering on matters specific to the present embodiment.
FIG. 5 is a block diagram showing a configuration of a similar program search device according to the embodiment. As shown in the figure, the similar program search device 102 includes a similarity calculation device 101. The similarity calculation device 101 includes a document reading unit 12, a word extraction unit 13, a related word extraction unit 114, a related graph construction unit 119, and a similarity calculation unit 116.

The characteristic configuration of the similarity calculation apparatus 101 is as follows.
The related word extracting unit 14 in the first embodiment extracts only related words directly related to the word extracted by the word extracting unit 13. On the other hand, the related word extraction unit 114 of the present embodiment not only extracts related words directly related to the words extracted by the word extraction unit 13, but also related words related to those related words, Extract as related words. In other words, the related word extraction unit 114 also extracts related words having a plurality of levels of relationships with words included in the original document.
Further, the relation graph construction unit 119 in the present embodiment constructs a graph (relevance data) representing the relation between the word extracted by the word extraction unit 13 and the related word extracted by the related word extraction unit 114. In this graph, nodes are individual words to be extracted, and edges are associations (relationships) between words. Details of this graph will be described later.
In addition, the similarity calculation unit 16 in the first embodiment calculates the similarity from the amplified expression tendency information (a list of words including related words) output by the expression amplification unit 25. On the other hand, the similarity calculation unit 116 according to the present embodiment calculates the similarity between two documents based on the graph constructed by the related graph construction unit 119. This similarity calculation method will be described later.

  FIG. 6 is a schematic diagram illustrating a data example of a graph constructed by the related graph construction unit 119. As shown in the figure, the data generated by the relation graph construction unit 119 includes nodes and edges connecting the nodes.

There are three types of nodes. The first type is a node corresponding to a document. In the figure, the node that says d ₀ and d _m is the node corresponding to the document. The second type is a node corresponding to a word appearing in the document. In the _{figure, n 1, n 2, n} 3, ···, node labeled _{n i,} and _{n 1 ', n 2',} n 3 ', ···, n j' was labeled node It is. What is connected by an edge from the node _{d 0} corresponding to the document, the word _{n 1,} n _2, n 3 appearing in the document _{d 0,} · · ·, a node _{n i.} What is connected by an edge from the node _{d m} corresponding to the document, the word _n 1 appearing in the document _{d m ', n 2',} n 3 is a node _{', ···, n j'.} The third type is a node corresponding to a related word. Nodes corresponding to related words are indicated by circles in the figure.

  In the graph data constructed by the relation graph construction unit 119, an edge directly connecting the two and the node corresponding to the document and the node corresponding to the word appearing in the document exist. In addition, there is an edge between the word (including the related word) and the word (including the related word) when both words are related and only in that case. Note that this edge is an undirected edge. In other words, when one word A is related to another word B, conversely, word B is related to word A. When word A is not related to word B, word B is not related to word A.

For example, in the document d ₀ contains the word "lifestyle-related diseases", if it contains the word "tobacco" in the document d _m, related word extraction section 114 each word these Extract related words. Then, when the related word extraction unit 114 acquires the relationships “lifestyle-related diseases—hypertension”, “hypertension-smoking”, and “smoking-cigarette” as the relationship between the words, the related graph construction unit 119 An edge connecting the nodes in order is generated. That is, edges of “lifestyle-related diseases—hypertension”, “hypertension—smoking”, and “smoking—tobacco” are generated. In this way, a graph connecting both documents is constructed.

  There are various ways of expressing data representing a graph, but a relational database is used as an example. The first table includes columns of a node ID (primary key), a node type, and a word corresponding to the node. In the first table, there is a row for each node. The second table includes columns of an edge ID (primary key), an edge start node ID, and an edge end node ID. In the second table, there is a row for each edge.

In order to construct such graph data, the following processing is performed. That is, when a pair of documents (for example, documents d ₀ and d _m ) is given and the word extraction unit 13 extracts words included in each document, the related word extraction unit 114 extracts the words extracted by the word extraction unit 13. A related word (related expression) is extracted for a noun in the word. The related word extracting unit 114 extracts not only related words directly related to the words included in the document but also related words related to the related words. In other words, the related word extraction unit 114 extracts all related words directly or indirectly related to the words included in the document. However, depending on the setting or the like, an upper limit may be set on the number of related stages, and the related word extracting unit 114 may extract only related words within the range of the number of stages. The related graph construction unit 119 examines related words from the respective sides of both documents, and generates data of the graph shown in FIG.

That is, the expression amplifying unit 125 extracts a related expression related to the expression extracted by the word extracting unit 13 (expression extracting unit) using the relationship between the expression and the related expression related to the expression. The expression tendency information extracted by the word extraction unit 13 is amplified and the amplified expression tendency information is output. Here, the expression tendency information is a list of words extracted by the word extraction unit 13 (words indicated as “words appearing in the document” in FIG. 6). Further, the post-amplification expression tendency information is information of all nodes including related words in FIG.
Furthermore, expression amplifying unit 125 includes a representation contained in the first post-amplification expression trend information obtained by amplifying the first text data (corresponding to d ₀₎ based on the second text data (corresponding to d _m) With respect to the node corresponding to the expression included in the second amplified expression tendency information that was originally amplified, the node corresponding to the first text data, and the node corresponding to the second text data, Graph data (relevance data) representing the presence or absence of association is generated.
That is, the expression amplifying unit 25 in the first embodiment amplifies the expression tendency information based on the relevance of only one stage, whereas the expression amplifying part 125 of the present embodiment has a plurality of stages (particularly the upper limit on the number of stages Expression tendency information is amplified based on the relevance).

  Based on the graph generated by the related graph construction unit 119, the similarity calculation unit 116 calculates the similarity between the two documents by evaluating how strongly the two documents are connected. The procedure is described below. Note that the similarity calculation unit 116 uses, for example, an algorithm of Green Measures. The literature on the Green Measures algorithm is as follows. References Oliver Yann, Senellart Pierre, Finding Related Pages Using Green Measures: An Illustration with Wikipedia, Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence, pp. 1427-1433, 2007.

  When a graph representing the connection between documents is given, the probability of transition from one node to another is represented by a matrix M. Each row and column of the matrix M corresponds to a node in the graph. The matrix element in the row corresponding to the node p and the column corresponding to the node q is the value of the probability of transition from the node p to the node q. Using this matrix M, a Green Matrix is defined by the following equation (1).

In Equation (1), M ^t represents a transition matrix in the t-th random walk step. That is, elements of the matrix of M ^t, column corresponding and to node q in the row corresponding to the node p, after t time random walk is a value of the probability of transition from the node p to node q. ^M∞ is the limit value of the transition matrix that converges when the random walk is repeated infinitely. The element of the column corresponding to the node q of the row | line | column corresponding to the node p of the matrix G defined by Formula (1) is a value which shows how much the node p is related to the node q. The final relevance score between nodes is defined by the following formula (2) using Green Matrix.

In Expression (2), G _pq is an element of a row corresponding to the node p and a column corresponding to the node q in the matrix. Further, ν _q is an element corresponding to the node q in the vector ν. The vector ν is an equilibria measure, and is a vector that converges when an infinite number of transitions are repeated starting from an arbitrary vector μ. That is, μM ^∞ = ν, and ν is a vector that expresses as a probability value which node is likely to make a final transition. Log (1 / ν _q ) is a factor for correction. That is, the value of ν _q represents a high possibility of convergence after repeating the transition from an arbitrary vector μ. Therefore, the score related to the node is corrected according to such a high possibility. Since 0 <ν _q <1, it goes without saying that 0 <log (1 / ν _q ), and the larger the value of ν _q , the smaller the value of log (1 / ν _q ). In other words, this log (1 / ν _q ) is a coefficient that acts like an idf value in a tf-idf (term frequency-inverse document frequency) value used in information retrieval. That is, S ^p (q) is a value that indicates how easily the node p transitions to the node q, and can be regarded as indicating the strength of the relationship between the node p and the node q.

Then, using the above equation (2), the similarity calculating unit 116 will now be _described, using either _{S direct (d 0, d m} ) or _{S related (d 0, d m} ) calculates the similarity between the document _{d 0} and _{d m (1 ≦ m ≦ M} ).

[Similarity calculation method 1: use only the values given in the _{d m]}
In this approach, with respect to node _d 0, _{d m} in the graph, by using equation (2) directly calculates the two documents _d 0, _{d m} of the similarity _{S direct (d 0, d m} ). In other words, the similarity calculation unit 116 calculates the similarity between the documents _{d 0} and _{d m} by the equation below (3) (i.e., similarity between the nodes _{d 0} and _{d m} of the document).

  The weights given to the edges connecting the nodes are as follows. In other words, the weight from the node of the document to the word node is given by the following expression (4), and the weight between the edges of the word nodes is given by the following expression (5).

In the equation (4), _{d m} is a node of the document (0 ≦ m ≦ M), n p is the node of the words appearing in the document _{d m.} Further, tf _{(n p)} is the word frequency corresponding to the node _{n p} of the document _{d m,} idf _{(n p)} is the inverse document frequency of the word corresponding to the node _{n p} in all documents.

Further, in the equation (5), the Z _np, for example, a total number of edges has signed the word node n _p and the other nodes. Alternatively, the edge weight e (n _p , n _q ) may be determined as appropriate so that the sum total is 1 for the word node n _p .

  The edge weights given by the equations (4) and (5) are set as the values of the elements of the matrix M described above. However, if the nodes are not directly connected by edges, the value of the matrix element corresponding to the nodes is 0.

  That is, in this “method 1”, the similarity calculation unit 116 uses the equation (3) to calculate data representing the ease of transition from node to node based on the graph data, and the relevance score between the nodes. And the relevance score from the node corresponding to the first text data to the node corresponding to the second text data is calculated as the similarity.

[Similarity calculation method 2: Using the relevance scores of all nodes on the path from d ₀ to d _m]
In this method, by using the sum of the values of the relevance scores given to all nodes on the path from node _{d 0} in the graph to _{d m,} 2 single document _d 0, _{d m} of the similarity _{S related} ( d ₀ , d _m ) is calculated. In other words, the similarity calculation unit 116 calculates the similarity between the documents _{d 0} and _{d m} by the equation below (6).

In the formula _{(6), vertex (d 0} , d m) is the set of all nodes (including node _{d m)} in the node _{d 0} on a path to the node _{d m.} In this “method 2”, as in the case of “method 1”, the weight of the edge connecting the nodes is given by the equations (4) and (5). The value of the element of the matrix M corresponding to the node that is not directly connected by the edge is 0.

  That is, in this “method 2”, the similarity calculation unit 116 uses the equation (6) to calculate data representing the ease of transition from node to node based on the graph data, and the relevance score between the nodes. And the sum of the relevance scores from the node corresponding to the first text data to all the nodes included on the path from the node corresponding to the second text data is calculated as the similarity.

Figure 7 is the graph shown in FIG. 6 is a schematic diagram showing a range of relevance scores to be used for calculating the degree of similarity between the node d ₀ and d _m using method 1. The bold line frame in the figure indicates the range. That is, in the method 1, the similarity calculating unit 116 calculates the node d _0, using only relevance score to the node d _m included in the thick frame, a similarity between the node d ₀ and d _m To do.
Figure 8 is the graph shown in FIG. 6 is a schematic diagram showing a range of relevance scores to be used for calculating the degree of similarity between the node d ₀ and d _m using the technique 2. The bold line frame in the figure indicates the range. That is, in the method 2, the similarity calculating unit 116, from node d _0, with relevance scores to all nodes included in the thick frame, and calculates the degree of similarity between the node d ₀ and d _m .
In calculating the similarity, it is arbitrary whether Method 1 or Method 2 is used, and may be determined as appropriate.

  In addition, you may make it implement | achieve the function of all or one part of the similar program search apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, a “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included, and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

<Modification>
Although the embodiment has been described above, the present invention can also be implemented in the following modified example. In a possible combination, a plurality of modified examples listed below may be combined.

  Modification 1: In the above embodiment, words are extracted from the program summary sentence, and related words of those words are further extracted to calculate the similarity between the program summary sentences. In this modification, instead of a word, a phrase, a word N-gram (chain of N words), a character N-gram (chain of N characters), or the like is used to express the above embodiment and Similar calculations are performed.

  Modification 2: In the above embodiment, a union of a set of words extracted from the program summary sentence and a set of related words of these words (excluding related words excluded by the filtering process) is used. Thus, the similarity between program summary sentences was calculated. In this modification, a set of related words (excluding related words excluded by filtering processing) (which is also the expression tendency information after amplification) is not used, but a set of words extracted directly from the program summary sentence is not used. ) To calculate the similarity between program summary sentences.

  Modification 3: In the above embodiment, the similarity calculation device 1 or 101 is provided inside the similar program search device 2 or 102, respectively. In this modification, only the similarity calculation device 1 or 101 is configured. The text data input to the similarity calculation device 1 or 101 is not limited to the program summary sentence, and may be general text data. Even in such a case, the similarity calculation device 1 has the same operations and effects as those described in the above embodiment.

  Modification 4: In the above embodiment, the related word filtering unit 15 is provided as a part of the expression amplification unit 25. In this modification, the related word filtering unit 15 is not provided and the filtering process is not performed. That is, the related word output from the related word extracting unit 14 is used for the expression tendency information after amplification without filtering.

  Modification 5: In the second embodiment, Green Measures is used as a measure representing the strength of association between nodes. Instead, another measure representing the strength of association between nodes is used. Also good. For example, a page rank algorithm described in the following reference may be used. References Sergey Brin, Lawrence Page, The Anatomy of a Large-Scale Hypertextual Web Search Engine, [URL http://infolab.stanford.edu/~backrub/google.html], May 1, 2012 download

Modification 6: In the second embodiment, the similarity from the document d ₀ to the document d _m (1 ≦ m ≦ M) is calculated. In the method used in the second embodiment, as is clear from the definition of similarity, the similarity from node a to node b and the similarity from node b to node a can take different values. In Modification 6, using equation according to the second embodiment, except by reversing only the direction to calculate the similarity of the documents d ₀ from the document d _m, the degree of similarity between the two documents. Further, the similarity in both directions may be calculated to obtain an average value or a weighted average value thereof.

  Modification 7: In the second embodiment, the tf-idf value is used as the weight from the document node to the word node (formula (4)). In the modified example 7, instead of the tf-idf value, the importance score defined in advance for each word (or a value obtained by dividing each score by the total value of the scores) is converted from the document node to the word node. Used as a weight to

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The similar program search apparatus 2 according to the first embodiment is implemented, and the similarity of each comparison target program is calculated for the designated program and the comparison target program ranking is performed using the program summary sentence regarding the NHK television program. .

Of the specified program title is "parenting people of Planet-based beads Tanzania sculpture", the program summary statement (equivalent to d ₀₎ is, the sculptor of the family of traditional arts Makonde sculpture representing the "East Africa coverage. Era I will introduce the child-rearing of the sculpture people who want to keep the tradition while feeling the change. "

The degree of similarity and ranking calculated for each of the summary sentences of the eight types of comparison target programs were as follows. In addition, the summary sentences of the four programs were ranked in the fifth place, and the degree of similarity was 0.001.
The ranking first place. The similarity is 0.321. The program title is “New Sunday Museum Sculptor Katsura Funakoshi”. The outline of this program is: “Sculptor Katsura Funakoshi, who continues to carve people living in the modern age. Funakoshi's sculpture, which has a unique presence not found in traditional human sculptures, has been spoken to the viewers. Funakoshi engraves on the sculpture that everyone feels as a human being, and the world of sculptor Katsura Funakoshi through the appearance of himself. Follow. "
The ranking second place. The similarity is 0.141. The program title is “Walking around the world Ragusa”. The outline of this program is: “Ragusa has a new town and an old town spread across two mountains across the valley. Barraque-style buildings line the old city of Ibra, a labyrinthic city where you can see the mysterious veranda supported by human face sculptures and hear the bells of the baroque San Giorgio Cathedral It ’s a city walk with a mysterious sensation as if you were traveling back in time to the Middle Ages. ”

The ranking third place. The similarity is 0.092. The program title is “Beautiful Noh mask”. The summary of this program is “Noh mask, which has a seemingly frightening impression with no expression. However, if you look closely, you can see that the expression changes variously depending on the angle. The Noh mask contains various craftsmanship techniques, such as sculpture techniques to create emotions and coloring techniques to express depth. A Noh mask that has a mysterious character that surpasses that of a work of art. Introducing Noh masks that have been cultivated for over 600 years and the points to appreciate their beauty. Is.
4th in the ranking. The similarity is 0.037. The title of the program is “Nadaden Kazu no Kyokai 7th Chanoyu Urasenke”. The summary of this program is “Cha no Yu”, where the owner and guests can communicate with each other through a cup of tea. It is one of the representative players of the “Japanese” hobby, boasting a tradition of over 400 years. The 9th series, approaching the spirit of the tea ceremony of the Urasenke, from the 3 thousand families (Omotesen, Urasenke, and Samurai Senji Senya) that inherited the tradition of Senrikyu. The seventh will tell you how to make Japanese sweets easily at home. Is.

1st in the ranking 5th. The similarity is 0.001. The title of the program is “BS World Doc Part 1 Struggle for Oil Control OPEC 50 Years”. The summary of this program is: “The first part explores how OPEC took away the“ dominance of oil ”that the seven major oil companies (major) had in the background of the formation of OPEC in 1960. October 1973 , OPEC has the right to determine the price of oil and forms the world's strongest cartel, reaching its peak, but a sharp rise in prices will cause an oil shock in the world and change OPEC's subsequent fate. " .
2nd in the ranking. The similarity is 0.001. The program title is “Asian Smile Yotsuko Nurse Story Korea”. The summary of this program is: “Sisters of the four children born 21 years ago at the hospital in Incheon, South Korea passed the National Examination for Nurses 2010. All started working at the hospital where they were born. The quadruple was a child with very few cases.Since she was born small among the four, she was the only maid who was put in the intensive care unit. I was in charge of nursing, and I was in close contact with Sol ’s days struggling to protect the little new life. ”

3rd in the ranking. The similarity is 0.001. The program title is “High-Vision Special Japanese Inuit Parent and Child Three Generations”. The outline of this program is “Japanese, Ikuo Oshima (63) who lived in Greenland in the Arctic. This is a valuable video diary that recorded the hunting life and growth of grandchildren and their third generation living as an Inuit from the perspective of their family. Is.
4th in the ranking. The similarity is 0.001. The program title is “Appearance in the New York Wave Desert! The summary of the program is “Every summer, a huge piece of art appears in the desert of Nevada, USA. The world ’s largest art festival“ Burning Man ”is held outdoors. There are about 200 huge works of art such as dandelions 6 meters above the ground. Dance and music performances are held all night. A week of over 50,000 participants enjoying the latest art while living together in a tent. In such a strange world, program reporter Hana has infiltrated. Follow the midsummer dreams unfolded in the desert. Is.

The similar program search apparatus 102 according to the second embodiment was implemented and its evaluation experiment was performed. First, 352 programs were randomly sampled from 25,769 programs registered in “NHK On Demand” (program on-demand distribution service) from September 2010 to May 2011 under the following restrictions: did.
Constraint 1: A program with the same program title is not taken out (for example, “NHK Special” is a sample of only one program).
Constraint 2: Have two or more related programs.

  Next, for the related programs of 352 programs presented on NHK on-demand, the work of ranking the similarity between the sampled programs and the related programs was performed by three annotators. Related programs related to each program were extracted by a technique using the index of Okapi-BM25, and an average of 10.4 related programs were presented for one program. The ranking result given by the three annotators mentioned above had an average rank correlation (Spearman's rank correlation) of 0.565. This can be interpreted as a certain degree of coincidence. Finally, the ranks of similarity assigned by the three annotators were averaged, and the evaluation was made by comparing this data with the ranks of each method based on the data judged to be similar in ascending order of the average rank. I did it.

Baseline method: A word is expressed by a vector of words by giving a weight by tf-idf to a word n appearing in the document p. Similarity between two documents is evaluated by cosine similarity between vectors corresponding to these two documents.
Method according to the first embodiment: A word appearing in a document is expanded (a word set is amplified) using a word relationship, and the document is expressed by a vector having the amplified word as an element. Similarity between two documents is evaluated by cosine similarity between vectors corresponding to these two documents.
“Method 1” according to the second embodiment: Using the relevance score to the nodes in the range shown in FIG. 7, the similarity between documents is evaluated.
“Method 2” according to the second embodiment: The similarity between documents is evaluated using the sum of the relevance scores to all the nodes in the range shown in FIG.

  The above-mentioned methods are applied to the randomly sampled 352 programs and related programs, and related programs are reranked. The correlation between these results and the reference data generated by the annotator is calculated using Spearman's rank correlation. evaluated. As a result, the correlation value of the baseline method was 0.350. The correlation value of the method according to the first embodiment was 0.371. The correlation value of “Method 1” according to the second embodiment was 0.351. The correlation value of “Method 2” according to the second embodiment was 0.425. In any of the methods according to the embodiment, a result closer to the manual ranking result was obtained than in the baseline method. In particular, good results were obtained with “Method 2” according to the second embodiment.

  The present invention can be used to calculate the similarity between texts in general natural language processing. It can also be used to present information using text via communication such as the Internet. It can also be used to recommend program content. Such a program recommendation technique can be used for a program ranking process, a broadcast receiving apparatus, a video-on-demand service, and the like.

1,101 Similarity Calculation Device 2,102 Similar Program Search Device 11 Program Information Storage Unit 12 Document Reading Unit 13 Word Extraction Unit (Expression Extraction Unit)
14,114 Related Word Extraction Unit (Related Expression Extraction Unit)
15 Related Word Filtering Unit (Related Expression Filtering Unit)
16, 116 Similarity calculation unit 17 Ranking unit 18 Search result output unit 25, 125 Expression amplification unit 119 Related graph construction unit

Claims (8)

An expression extraction unit for obtaining expression tendency information representing an appearance tendency of the type of expression by extracting an expression from text data;
Amplifying the expression tendency information extracted by the expression extraction unit by extracting the related expression related to the expression extracted by the expression extraction unit using the relationship between the expression and the related expression related to the expression An expression amplifying unit that outputs the expression tendency information after amplification;
Based on the first post-amplification expression tendency information output from the expression amplifier based on the first text data input to the expression extractor and the second text data input to the expression extractor A similarity calculation unit that calculates a similarity between the first text data and the second text data based on the relationship between the second amplified expression tendency information output from the expression amplification unit When,
The similarity calculation apparatus characterized by comprising. The expression amplifying unit amplifies the expression tendency information by extracting a related expression related to the extracted related expression,
The expression amplifying unit includes an expression included in the first post-amplification expression tendency information amplified based on the first text data, and the second post-amplification expression amplified based on the second text data. Relevance data representing presence / absence of association between nodes with respect to a node corresponding to an expression included in trend information, a node corresponding to the first text data, and a node corresponding to the second text data Produces
The similarity calculation unit calculates the strength of the relationship between the first text data and the second text data calculated based on the relationship data as the similarity.
The similarity calculation apparatus according to claim 1, wherein: The similarity calculation unit sets data representing the ease of transition from node to node based on the relevance data as a relevance score between the nodes, and from a node corresponding to the first text data. Calculating the sum of the relevance scores up to all nodes included on the path to the node corresponding to the second text data as the similarity.
The similarity calculation device according to claim 2, wherein: The similarity calculation unit sets data representing the ease of transition from node to node based on the relevance data as a relevance score between the nodes, and from a node corresponding to the first text data. Calculating the relevance score up to a node corresponding to the second text data as the similarity;
The similarity calculation device according to claim 2, wherein: The similarity calculation unit includes first amplified expression tendency information output from the expression amplification unit based on first text data input to the expression extraction unit, and first input to the expression extraction unit. Calculating the degree of similarity between the second amplified expression tendency information output from the expression amplification unit based on the text data of 2;
The first post-amplification expression tendency information is information representing a distribution of expressions included in the first text data and related expressions directly related to expressions included in the first text data;
The second post-amplification expression tendency information is information representing a distribution of an expression included in the second text data and a related expression directly related to the expression included in the second text data.
The similarity calculation apparatus according to claim 1, wherein: The expression amplification unit includes:
The similarity according to claim 5, further comprising: a related expression filtering unit that excludes the related expressions whose similarity to the text data is less than a predetermined threshold among the extracted related expressions. Degree calculation device. The similarity calculation device according to any one of claims 1 to 6, comprising:
The program summary sentence text data relating to the designated program is input as the first text data to the similarity calculating device, and the program summary sentence text data relating to another program is inputted as the second text data to the similarity calculating device. The specified program calculated by the similarity calculating device, obtaining the similarity between the specified program and the other program, and calculating the plurality of other programs; Searching for another program similar to the designated program from among the other programs based on the similarity between
Similar program search device characterized by the above. Computer
An expression extraction unit for obtaining expression tendency information representing an appearance tendency of the type of expression by extracting an expression from text data;
Amplifying the expression tendency information extracted by the expression extraction unit by extracting the related expression related to the expression extracted by the expression extraction unit using the relationship between the expression and the related expression related to the expression An expression amplifying unit that outputs the expression tendency information after amplification;
Based on the first post-amplification expression tendency information output from the expression amplifier based on the first text data input to the expression extractor and the second text data input to the expression extractor A similarity calculation unit that calculates a similarity between the first text data and the second text data based on the relationship between the second amplified expression tendency information output from the expression amplification unit When,
A program for functioning as a similarity calculation device.

Images