JP4350026B2 - Topic scope extraction device, control method thereof, and program - Google Patents

Description

The present invention relates to a topic scope extraction device that extracts a set of topic scopes, which are message sets that make up a semantic group, from a message sequence in which a plurality of topics are mixed. A message sequence that is a flow and is composed of a plurality of topics.

  A portal site such as a bulletin board is a set of threads (words composed of original utterances and comments corresponding thereto), and each thread is a message string. This message sequence is a flow of conversations arranged in time series, and is composed of a plurality of topics.

  For such a thread set, there is a search process for inputting a natural language sentence and acquiring a message set that matches the input sentence.

  There can be considered a method of searching for a matching message based on the semantic similarity between each message and the input sentence.

  However, a message such as a bulletin board is characterized in that each message is extremely short and contains a small number of words. A message alone is not a complete topic. The amount of information is too small to consider the message as a document in a general document search.

  One topic in the message sequence is composed of a plurality of messages, and each message is only one of a group of messages that form a semantic group. Only when a group of messages forms a semantic group, it has the same amount of information as a document in a general document search.

  In the above method, simply searching for a message having high similarity with the input sentence cannot extract a message group including the message and forming a meaningful group. For this reason, many of the messages that should be searched for are leaked from the search results.

  Therefore, it is appropriate to extract a topic scope, which is a message set that makes a meaningful group from a message sequence, and to search for a topic scope that matches the input sentence.

As a technique for extracting a set of semantic messages, each message is represented by a feature vector with the word as the coordinate and the importance of the word message as the coordinate value. It is known that the message set is clustered so that the messages in the same cluster are the same (for example, see Non-Patent Document 1).
Akira Yoto, Takashi Ishikawa “Topic Extraction for Active Information Search in Community Web”, The 17th Annual Conference of the Japanese Society for Artificial Intelligence, pp. 1-4, 2003

  In the method described in Non-Patent Document 1, a message is represented by a feature vector. However, as described above, a message such as a bulletin board is generally short and contains a small number of words. Therefore, although it is a message of the same topic, there are few common parts of the included word set, and thus, a feature. There is a problem that cases where the vector is far away frequently occur.

  On the other hand, since the messages belong to different topics, there are many common parts of the included word sets, so there are many cases where the feature vectors are close, and for this reason, messages of the same topic are the same. There is a problem in that clustering that becomes a cluster cannot be executed properly.

An object of the present invention is to provide a topic scope extraction device capable of appropriately extracting a set of topic scopes, which are message sets that form a semantic group, from a message sequence in which a plurality of topics are mixed. .

  The invention described in claim 1 is a topic scope extraction device that extracts a set of topic scopes, which is a set of messages that make up a semantic group, from a message sequence in which a plurality of topics are mixed, and morphologically analyzes each message The message morpheme analyzing means for dividing into words, the concept base which is a storage means storing a pair of a word and a vector expressing the meaning of the word, and the concept base by searching the concept base Intra-message word vector acquisition means for acquiring a vector corresponding to each word obtained by the morpheme analysis means, and for any message pair, based on the word vector set in each message of the pair, between the pair of messages A message inter-vector set distance calculating means for calculating the inter-message vector set distance; For a message pair, based on the reference relationship between the messages in the pair, a reference distance calculation means for calculating a reference distance between the messages in the pair, and a word vector sequence obtained by the word vector acquisition means in the message Topic segmentation means for dividing the message sequence into a set of topic segments that are sections of the same topic, and for any topic segment pair, based on the word vector set in each topic segment of the pair, the pair of topics Between topic segment vector set distance calculating means for calculating the distance between topic segment vector sets between segments, and between the pair of messages obtained by the above-in-message vector set distance calculating means for any message pair. Distance between vector sets in message and reference distance calculation means Based on the obtained reference distance between the pair of messages and the distance between topic segment vector sets between the topic segments to which each pair of messages obtained by the topic segment vector set distance calculation means is obtained, Based on the inter-message distance calculation means for calculating the final distance between a pair of messages and the distance between any messages obtained by the above-mentioned inter-message distance calculation means, message strings are fuzzy clustered and obtained as a result. A topic scope extraction device comprising fuzzy clustering means for deriving a set of messages having a degree of belonging to a cluster equal to or greater than a threshold for each cluster as a topic scope.

The invention described in claim 2 is the method according to claim 1, wherein the in-topic vector information generating means for generating information on the word vector set in each topic scope obtained by the fuzzy clustering means, and the morphological analysis of the input sentence and the word Input sentence morpheme analyzing means for dividing into units, input sentence word vector obtaining means for obtaining a vector corresponding to each word obtained by the input sentence morpheme analyzing means by searching the concept base, and the input Input sentence vector information generating means for generating information related to a word vector set in the input sentence obtained by the in-sentence word vector acquiring means, and a word vector set in the input sentence obtained by the input sentence vector information generating means And the word vector set in each topic scope obtained by the topic scope vector information generating means. That by performing the distance calculation between the information, a topic scope extraction apparatus characterized by having an input sentence-topic scope distance calculation means for calculating the distance between the input sentence and the topic scope.

According to the present invention, it is possible to appropriately extract a set of topic scopes, which is a set of messages that form a semantic group, from a message string in which a plurality of topics are mixed.

  The best mode for carrying out the invention is the following examples.

  FIG. 1 is a block diagram illustrating a topic scope extraction apparatus 100 that is Embodiment 1 of the present invention.

  The topic scope extraction apparatus 100 is a topic scope extraction apparatus that extracts a set of topic scopes, which is a message set that forms a semantic group, from a message sequence in which a plurality of topics are mixed. Base 12, message word vector acquisition means 13, message vector set distance calculation means 14, reference distance calculation means 15, topic segmentation means 16, topic segment vector set distance calculation means 17, and message An inter-distance calculation means 18 and a fuzzy clustering means 19 are provided.

  FIG. 2 is a diagram illustrating an example of an input message string in the first embodiment.

The message is a posting unit, and the message with the number _i is represented as Mi. In FIG. 2, “>> 4” described in the message content of the message M ₉ indicates a link to the message M ₄ , that is, means that the message M ₉ refers to M ₄ . .

The set of topic scopes to be extracted from this message sequence is C ₁ = {M ₁ , M ₂ , M ₃ , M ₄ , M ₉ , M ₁₀ }, C ₂ = {M ₁ , M ₅ , M ₆ , M ₇ , M ₈ }, and M ₁ belongs to two topic scopes.

The message morpheme analyzing unit 11 is a unit that morphologically analyzes each message and divides the message into words. Among the words obtained as a result, the part of speech information or the like is referred to and only the content word is left. However, as in the ">>4" in M _9, links to other messages, leave intact. By dividing the word unit, each message becomes as shown in FIG.

  FIG. 3 is a diagram showing a result of morphological analysis of the message M shown in FIG.

  FIG. 4 is a diagram illustrating an example of the concept base 12 in the first embodiment.

  The concept base 12 is stored in a storage means such as a hard disk, and an f-dimensional vector value is assigned to each word. The words in the concept base 12 are independent words such as nouns, verbs, and adjectives. The word vectors in the concept base 12 are set such that the distance between words that are semantically similar is closer, and the distance between words that are not semantically similar is longer.

  Examples of concept bases are disclosed in “Similarity Discriminating Device” of Japanese Patent Laid-Open No. 6-103315 and “Similarity Discriminating Data Refinement Method and Device for Implementing this Method” of Japanese Patent Laid-Open No. 7-302265. Database.

  In Deerwester's paper (S. Deerwester, STDumais, GWFurnas, TKLandauer and R. Harshman, Indexing by Latent Semantic Analysis, Journal of the American Society for Information Science, pp.391-407, 1990.) The co-occurrence matrix between words and documents in which the frequency in the document is recorded is converted into a matrix in which the number of dimensions is reduced by singular value decomposition, and this converted matrix is also an example of a concept base.

  In Schutze's paper (H. Schutze, Dimensions of Meaning, Proc. Of Supercomputing '92, pp.786-796, 1992.), the frequency of co-occurrence between words in the corpus is recorded. The example is converted into a matrix with a reduced number of dimensions by singular value decomposition, and this converted matrix is also an example of a concept base.

  The in-message word vector acquisition means 13 is a means for acquiring a vector corresponding to each word obtained by the message morpheme analysis means 11 by searching the concept base 12.

  The intra-message vector set distance calculating means 14 is a means for calculating an inter-message vector set distance between the paired messages based on a word vector set in each message of the pair for an arbitrary message pair. .

A sequence of word vectors in the input text (vectors of words at different positions are different, even if they have the same value)
X = {x ₁ , x ₂ ,..., X _{| x |}
And

The word vector set in each message is a set of consecutive X elements. Henceforth, M _i shall mean the word vector set in message M _i simultaneously.

The center of gravity D (M _i ) of the message M _i is

と計算される。｜Ｍ_ｉ｜は、Ｍ_ｉ内の要素数である。

Is calculated. | M _i | is the number of elements in M _i .

The cost E (M _i ) of the message M _i

とする。

And

The intra-message vector set distance ΔE (M _i , M _j ) between any message M _i and M _j is an increase in cost when M _i and M _j are merged. ΔE (M _i , M _j ) is

となる。

It becomes.

  However, for a message having one word or less, the distance between the vector sets in the message with other messages is not calculated.

  According to the above equation (1), the distance between the message vector sets between arbitrary messages is as shown in FIG.

  FIG. 5 is a diagram illustrating a distance between message vector sets between messages M illustrated in FIG.

The distance between M _i and M _j (i <j) is described only for the component in the i-th row and j-th column.

  For any message, the distance from itself is 0.0.

For M ₃ and M ₆ having one word, the distance from other messages is not calculated. M ₃ and M ₆ have the same word set, but the distance is not 0.0, and it is avoided that M ₃ and M ₆ belong to the same topic.

The distance between M ₁ and M ₂ receiving the contents is 0.2, and the distance between M ₁ and M ₅ receiving the contents is also 0.2. The distance between the M ₉ and _{M 10} is 0.3. The distance between other different messages is 0.5.

  The reference distance calculation means 15 is a means for calculating the reference distance between the pair of messages for an arbitrary message pair based on the reference relationship between the pair of messages.

For two messages _{M i, M j (i <} j), _{when M j} refers to a _{M i, M i,} the reference distance between _{M j,} and 0.0. Even if M _i refers to M _j , there is not always topic identity with a message that appears later in time, so that the reference distance between M _i and M _{j is set} to 0. Do not set to zero.

  Thus, the reference distance between arbitrary messages is as shown in FIG.

  FIG. 6 is a diagram illustrating a reference distance between the messages M illustrated in FIG.

The distance between M _i and M _j (i <j) is described only for the component in the i-th row and j-th column.

Since M ₉ are referring to _{M 4,} reference distance between _{M 4} and _{M 9} is a 0.0. The reference distance between other messages is not calculated.

  The topic segmentation means 16 divides the message string from the word vector series obtained by the in-message word vector acquisition means 13 into a set of topic segments that are sections of the same topic.

  A topic segmentation method is disclosed in Japanese Patent Application Laid-Open Nos. 2002-342324 and 2004-234512.

  The method of topic segmentation described in Japanese Patent Laid-Open No. 2002-342324 takes a word string that is a set of a predetermined number of words before and after an arbitrary word boundary, and constructs the word string for each word string The centroid of the vector of the word to be calculated is calculated, and the cosine measure between the centroids corresponding to the preceding and following word strings is used as the cohesion degree of the word boundary, and the word boundary at which the cohesion degree is minimized is recognized as the boundary of the topic section. Is the method.

  In addition, the topic segmentation method described in Japanese Patent Application Laid-Open No. 2004-234512 has the sum of squares of the distance between each word vector and the centroid of the word vector in the section for an arbitrary section, This is a method for obtaining a cost as a topic section string, which is obtained as a cost, and the cost of an arbitrary section string is the sum of the costs of the sections included in the section string, and the section cost having the minimum cost under a certain condition. .

  FIG. 7 is a diagram showing the result of topic segmentation of the message sequence shown in FIG.

As shown in FIG. 7, the topic segmentation means 16 converts the message string into a set of topic segments T ₁ = {M ₁ , M ₂ , M ₃ , M ₄ }, T ₂ = {M ₅ , M ₆ , M ₇ , M ₈ }, T ₃ = {M ₉ , M ₁₀ }.

  Some message pairs in a topic segment have a distance as large as 0.5. However, topic segmentation makes it possible to detect topic identity between messages in a topic segment.

  The intra-topic-segment vector set distance calculating means 17 calculates, for an arbitrary topic segment pair, the distance between the topic-segment vector sets between the paired topic segments based on the word vector set in each topic segment of the pair. It is a means for calculating.

The set of word vectors in each topic segment is a continuous set of X elements. Henceforth, T _i shall mean the word vector set in T _i at the same time.

  The distance between topic segment vector sets is determined as follows in the same manner as the distance between message vector sets.

When the centroid of the topic segment T _i is D (T _i ) and the number of elements in T _i is | T _i |, the distance between the vector sets in the topic segment between any topic segments T _i and T _j ΔE (T _i , T _j )

とする。

And

  According to the above equation (2), the distance between topic segment vector sets between arbitrary topic segments is as shown in FIG.

  FIG. 8 is a diagram showing the distance between the topic segment vector sets between the topic segments T shown in FIG.

The distance between T _i and T _j (i <j) is described only for the component in the i-th row and j-th column.

  For any topic segment, the distance from itself is 0.0. The distance between different topic segments is 0.5.

  The inter-message distance calculation means 18 is configured to obtain the inter-message vector set distance between the pair of messages obtained by the intra-message vector set distance calculation means 14 and the reference distance calculation means 15 obtained for any message pair. Among the reference distance between the pair of messages and the distance between the topic segment vector sets between the topic segments to which each of the pair of messages obtained by the topic segment vector set distance calculation means 17 is obtained, the minimum value is set as above. It is a means to make the final distance between a pair of messages.

  Thus, the distance between arbitrary messages is as shown in FIG.

  FIG. 9 is a diagram showing the distance between the messages M shown in FIG.

The distance between M _i and M _j (i <j) is described only for the component in the i-th row and j-th column.

The distance between any messages in the same topic segment is 0.0. The distance between the message vector sets of M ₁ and M ₅ is 0.2, and the distance between the topic segments to which each message belongs is 0.5. Therefore, the distance between the messages is 0.2.

The distance between the vector sets in the message between M ₄ and M ₉ is 0.5, the reference distance is 0.0, and the distance between the topic segments to which each message belongs is 0.5. The inter-distance is 0.0.

  The distance between messages belonging to different topic segments other than the above is 0.5.

  The fuzzy clustering means 19 performs fuzzy clustering on the message string based on the distance between the messages obtained by the inter-message distance calculation means 18, and the degree of belonging to the cluster is determined for each cluster obtained as a result. This is means for deriving a set of messages that are equal to or greater than a predetermined threshold as a “topic scope”.

  The method of fuzzy clustering is a method described in “E.H. Ruspini, A new approach to clustering, Information and Control, Vol. 15, pp. 22-32, 1969.”. The Ruspini method will be described below.

Assume that a set of messages {M ₁ , M ₂ ,..., M _n } and a distance d (M _i , M _j ) between arbitrary messages M _i and M _j are given. Further, a set of message fuzzy clusters is represented as {C ₁ , C ₂ ,..., C _r }. Each component u _pi of U = (u _pi ) (1 ≦ p ≦ r, i ≦ i ≦ n) means the degree of attribution of M _i to C _p , and U = (u _pi ) is

の条件を満たすものとする。評価関数

It shall satisfy the conditions of Evaluation function

を考える。重みωも、実数値をとる変数である。

think of. The weight ω is also a variable that takes a real value.

In the Ruspini method, U, ω that minimizes the evaluation function J _R (U, ω), that is,

を求める。求めたＵ＝（ｕ_ｐｉ）の各成分ｕ_ｐｉが、Ｍ_ｉのＣ_ｐへの帰属度である。

Ask for. Each component u _{pi of the} obtained U = (u _pi ) is the degree of attribution of M _i to C _p .

Taking a predetermined threshold value α of the degree of membership, a message set {M _i : _upi ≧ α} is set as a corresponding topic scope for each fuzzy cluster C _p .

By inputting the distance information between arbitrary messages shown in FIG. 9 and executing the fuzzy clustering, a set of topic scopes C ₁ = {M ₁ , M ₂ , M ₃ , M ₄ , M ₉ , M ₁₀ }, C ₂ = {M ₁ , M ₅ , M ₆ , M ₇ , M ₈ }.

Since the inter-message distance between M ₁ and M ₅ is relatively small at 0.2, M ₁ belongs to the same topic scope as the message group in the topic segment T ₂ .

The message distance between _{M 4} and _{M 9} are, because it is 0.0, the distance between the topic segment _{T 1} and _{T 3,} despite the larger and 0.5, _{T 1} and _{T 3} Message groups belong to the same topic scope.

  FIG. 10 is a flowchart illustrating the operation of the first embodiment.

  First, in S1, the message morphological analysis unit 11 divides each message into words by morphological analysis and stores them in the storage device.

  In the present specification and claims, the “storage device” is a concept including not only a memory but also an external storage device such as a hard disk.

  In S2, the message word vector acquisition unit 13 searches the concept base 12 to acquire a vector corresponding to each word obtained by the message morpheme analysis unit 11, and stores it in the storage device.

  In S3, the inter-message vector set distance calculating means 14 calculates the inter-message vector set distance between the paired messages based on the word vector set in each paired message for an arbitrary message pair. And store it in the storage device.

  In S4, the reference distance calculation means 15 calculates the reference distance between the pair of messages for an arbitrary message pair based on the reference relationship between the pair of messages, and stores it in the storage device.

  In S5, the topic segmentation means 16 divides the message string from the word vector series obtained by the in-message word vector acquisition means 13 into a set of topic segments that are sections of the same topic, and stores them in the storage device.

  In S6, the inter-topic-segment vector-set distance calculating means 17 calculates, for any topic segment pair, the intra-topic-segment vector set between the paired topic segments based on the word vector set in each pair of topic segments. The distance is calculated and stored in the storage device.

  In S7, the inter-message distance calculating means 18 for any message pair, the inter-message vector set distance between the pair of messages obtained by the intra-message vector set distance calculating means 14 and the reference distance calculating means 15 Based on the reference distance between the pair of messages obtained in the above and the distance between topic segment vector sets between the topic segments to which each message of the pair belongs obtained by the distance calculation unit 17 within the topic segment vector set, The final distance between the pair of messages is calculated and stored in the storage device.

In S8, the fuzzy clustering means 19 performs fuzzy clustering on the message sequence based on the distance between the arbitrary messages obtained by the inter-message distance calculating means 18, and for each cluster obtained as a result, A set of messages whose degree of belonging is equal to or greater than a threshold is derived as a topic scope and stored in a storage device.

  FIG. 11 is a block diagram illustrating a configuration of a topic scope extraction apparatus 200 that is Embodiment 2 of the present invention.

  The topic scope extraction apparatus 200 is similar to the topic scope extraction apparatus 100 in that it includes topic scope vector information generation means 20, input sentence morpheme analysis means 21, input sentence word vector acquisition means 22, and input sentence vector information generation means 23. And an input sentence / topic scope distance calculation means 24 are added.

  That is, the topic scope extraction device 200 includes a message morpheme analysis unit 11, a concept base 12, an in-message word vector acquisition unit 13, an in-message vector set distance calculation unit 14, a reference distance calculation unit 15, a topic segmentation. Means 16, inter-topic segment vector set distance calculating means 17, inter-message distance calculating means 18, fuzzy clustering means 19, topic-scope vector information generating means 20, input sentence morpheme analyzing means 21, input sentence An internal word vector acquisition unit 22, an input sentence vector information generation unit 23, and an input sentence / topic scope distance calculation unit 24.

  The topic-scope vector information generating means 20 is means for generating information on the word vector set in each topic scope obtained by the fuzzy clustering means 19.

Word vector set in the topic scope C _p is the sum of the word vector set contained in the message in the C _p.

Thereafter, C _p shall also mean at the same time the word vector set in the C _p. For each topic scope _{C p,} the center of gravity D of _{C p (C p),} the number of elements in _{C p} | _{C p} | and is calculated and maintained. These _{C p} of the center of gravity D _{(C p),} the number of elements in _{C p} | _{C p} | and is information about the word vector set in the _{C p.}

  The input sentence morpheme analyzing unit 21 is a unit that performs morphological analysis on the input sentence and divides the input sentence into words. Among the words obtained as a result, the part of speech information or the like is referred to and only the content word is left.

  As an example, the input sentence Q: “delicious ramen store” becomes “delicious / ramen / store”.

  The input sentence word vector acquisition means 22 is a means for acquiring a vector corresponding to each word obtained by the input sentence morpheme analysis means 21 by searching the concept base 12.

  Henceforth, Q shall mean the word vector set in Q simultaneously.

  The input sentence vector information generating means 23 is means for generating information relating to a word vector set in the input sentence obtained by the input sentence word vector acquiring means 22. For the input sentence Q, the center of gravity D (Q) of Q and the number of elements | Q | in Q are calculated and held. The centroid D (Q) of Q and the number of elements | Q | in Q are information relating to the word vector set in the input sentence Q.

  The input sentence / topic scope distance calculation means 24 includes information on the word vector set in the input sentence obtained by the input sentence vector information generation means 23 and each topic scope obtained by the topic scope vector information generation means 20. It is a means for calculating the distance between the input sentence and each topic scope by calculating the distance between the information related to the word vector set.

The distance d (Q, C _p ) between the input sentence Q and the topic scope C _p is

として算出する。

Calculate as

For a set of topic scopes C ₁ = {M ₁ , M ₂ , M ₃ , M ₄ , M ₉ , M ₁₀ }, C ₂ = {M ₁ , M ₅ , M ₆ , M ₇ , M ₈ } When the scopes are sorted according to the distance from the input sentence Q, the order is C ₂ and C ₁ . The topic scope can be presented in this order for the input sentence Q. It is also possible to present only the topic scope having a distance equal to or less than a predetermined threshold. As a result, we are presenting the only topic scope C _2.

  It should be noted that the inter-message vector set distance calculation means 14 does not calculate the distance between the intra-message vector sets with other messages for messages whose number of words in the message is a certain number or less (hereinafter referred to as small messages). I made it. However, a message in a different topic segment and having a certain number of words or more in the message (hereinafter referred to as a large message) may receive the content of a small message without a reference relationship.

  As an example, a large message in a different topic segment may be received in response to a message of “Let's talk”.

  Therefore, only when the content of the small message is a content that evokes a new topic, the distance between the vector sets in the message between the small message and the large message that comes after the message is calculated. Also good. By this calculation, two messages can belong to the same topic.

  12 and 13 are diagrams illustrating examples of messages having links to a plurality of other messages according to the second embodiment.

  When there is a reference to a set N of two or more messages in one message M, as shown in FIGS. 12 and 13, the reference distance calculation means 15 determines between M and an arbitrary element of N. The reference distance may be set sufficiently small, for example, 0.0.

In the example illustrated in FIG. 12, the reference distance between each of M ₄ , M ₅ , and M ₆ and M ₉ is 0.0. In the example shown in FIG. _13, and _{M 13,} see the distance between the respective message _{M 8, M 10} is 0.0.

  FIG. 14 is a diagram showing a result of dividing a message having links to a plurality of other messages shown in FIG.

In particular, a message described for each link referred to as shown in FIG. 13 is divided into link information as shown in FIG. 14 based on format information such as empty lines in the message. You can also. As described above, a message division process may be performed in advance on a message string to be input, and then input to the topic scope extraction device 200. As in the example illustrated in FIG. 14, if a predetermined message has a link to the message M ₁₃ like “>> 13”, the predetermined message includes M _13.1 and M _13.2 . It means that you are referring.

  In the topic scope extraction apparatus 200, when there are a plurality of message contents in one message regardless of the presence or absence of link information, the message is divided and the divided message is input to the topic scope extraction apparatus 200. You may do it. As a message dividing method, the methods described in JP-A-2002-342324 and JP-A-2004-234512 can be employed.

  Further, instead of dividing each message, it is possible to divide the entire text of the message string and input a set of intersections between each topic segment and each message obtained as a result. In this case, the topic segmentation means 16 does not execute the topic segmentation, and the already obtained topic segment set is used in the processing after the topic segment vector set distance calculation means 17.

  FIG. 15 is a diagram illustrating an example of a message in which the number of the previous message is described in the poster name column in the second embodiment.

As shown in FIG. 15, there may be a case in which a message has a contributor name field, and the number of the message posted by the user is used as the contributor name of the newly posted message. As described above, when the number of the previous message is described in the contributor name field in the message, the topic identity may be recognized between the two messages. The reference distance between two messages may be set sufficiently small, for example, 0.0. In the example illustrated in FIG. 15, the reference distance between the messages M ₁₅ and M ₁₉ is 0.0.

  Next, the operation of the second embodiment will be described.

  FIG. 16 is a flowchart illustrating the operation of the second embodiment.

  Steps S1 to S8 are the same as steps S1 to S8 in the flowchart of FIG.

  Next to S8, in S9, the in-topic scope vector information generating unit 20 generates information on the word vector set in each topic scope obtained by the fuzzy clustering unit 19, and stores it in the storage device.

  In S11, the input sentence morpheme analyzing means 21 performs morphological analysis on the input sentence and divides it into word units.

  In S12, the input sentence word vector acquisition means 22 searches the concept base 12 to acquire a vector corresponding to each word obtained by the input sentence morpheme analysis means 21, and stores it in the storage device.

  In S13, the input sentence vector information generation unit 23 generates information on the word vector set in the input sentence obtained by the input sentence word vector acquisition unit 22, and stores the information in the storage device.

  In S 14, the input sentence / topic topic distance calculation unit 24 obtains the information related to the word vector set in the input sentence obtained by the input sentence vector information generation unit 23 and the topic scope in-scope vector information generation unit 20. By calculating the distance between the information related to the word vector set in each topic scope, the distance between the input sentence and each topic scope is calculated and stored in the storage device.

  By the way, when the distance between messages in one topic segment is measured, the number of words in the message is not sufficient, so the distance is often large. This means that even if the distance is large when measuring the distance in one message unit, the difference between the message sequence adjacent to the message sequence will be highlighted if the word vector set included in the continuous message sequence is taken Means there is. By dividing the message string into a set of topic segments by the topic segmentation means 16, it is possible to detect topic identity between messages in a topic segment that is difficult to detect with the similarity of one message unit.

  The reference distance calculation means 15 can unconditionally set a reference distance between messages having a reference relationship.

  The inter-message distance calculation means 18 uses the minimum values of the inter-message vector set distance, the reference distance, and the topic segment vector set distance as the final distance between messages.

  Consider a determinant for a message having a certain number of words or less in a message (hereinafter referred to as a small message) and other messages belonging to the same topic. Except for some cases where the content of a small message is different from the small message and the topic segment to which the small message belongs and a message with a certain number of words or more in the message (hereinafter referred to as a large message) is received without reference relationship. The presence or absence of a reference relationship between two messages and the distance between the topic segments to which they belong are the determining factors. The intra-message vector set distance calculation means 14 does not calculate the inter-message vector set distance with other messages for small messages. Thereby, the distance between the small message and the other message is determined by the reference distance and the distance between the vector sets in the topic segment.

On the other hand, determinants for causing two large messages to belong to the same topic are the presence / absence of the reference relationship, the distance between the topic segments to which they belong, and the distance between the vector sets in the message. In particular, when a large message A is received, a topic different from the topic to which A belongs may start without a reference relationship from a large message B having a different topic segment to which A belongs. In this case, the message A should belong to a plurality of topics. The inter-message vector set distance between A and B is smaller than the distance between topic segments to which A and B belong. As a result, although A and B belong to topic segments with different distances, A belongs to the same topic as the message set in the topic segment to which B belongs by fuzzy clustering.

The present invention can be applied to a technique for extracting a set of topic scopes, which is a set of messages that form a semantic group, from a message sequence in which a plurality of topics are mixed.

It is a block diagram which shows the topic scope extraction apparatus 100 which is Example 1 of this invention. In Example 1, it is a figure which shows an example of the message sequence used as input. It is a figure which shows the result of having performed the morphological analysis of the message M shown in FIG. It is a figure which shows the example of the concept base 12 in Example 1. FIG. It is a figure which shows the distance between the vector sets in a message between the messages M shown in FIG. It is a figure which shows the reference distance between the messages M shown in FIG. FIG. 3 is a diagram showing a result of topic segmentation of the message sequence shown in FIG. 2. It is a figure which shows the distance between the vector sets in a topic segment between the topic segments T shown in FIG. It is a figure which shows the distance between the messages M shown in FIG. 3 is a flowchart showing the operation of the first embodiment. It is a block diagram which shows the structure of the topic scope extraction apparatus 200 which is Example 2 of this invention. It is a figure which shows the example of the message which has the link to several other messages in Example 2. FIG. It is a figure which shows the example of the message which has the link to several other messages in Example 2. FIG. It is a figure which shows the result of having divided | segmented the message which has the link to other several messages shown in FIG. In Example 2, it is a figure which shows the example of the message in which the number of the previous message is described in the contributor name column. 10 is a flowchart showing the operation of the second embodiment.

Claims (6)

A topic scope extraction device that extracts a set of topic scopes, which are message sets that make up a semantic group, from a message sequence in which multiple topics are mixed,
A message morpheme analyzing means for dividing each message into words by performing morphological analysis;
A concept base which is a storage means in which a pair of a word and a vector expressing the meaning of the word is stored;
An in-message word vector acquisition means for acquiring a vector corresponding to each word obtained by the message morpheme analysis means by searching the concept base;
An intra-message vector set distance calculating means for calculating an inter-message vector set distance between the paired messages based on a word vector set in each pair of messages for an arbitrary message pair;
Reference distance calculation means for calculating a reference distance between the pair of messages based on a reference relationship between the pair of messages for an arbitrary message pair;
Topic segmentation means for dividing the message string into a set of topic segments that are sections of the same topic from the word vector series obtained by the word vector acquisition means in the message;
A topic segment vector set distance calculation means for calculating a distance between topic segment vector sets between the pair of topic segments based on a word vector set in each topic segment of the pair for an arbitrary topic segment pair; ;
For an arbitrary message pair, the inter-message vector set distance obtained by the intra-message vector set distance calculating means and the reference between the pair of messages obtained by the reference distance calculating means. Based on the distance and the distance between the topic segment vector sets between the topic segments to which each pair of messages belongs, obtained by the above-mentioned topic segment vector set distance calculation means, the final distance between the pair of messages is calculated. A message distance calculating means for calculating;
Based on the distance between any messages obtained by the message distance calculation means, fuzzy clustering is performed on the message string. Fuzzy clustering means to derive a set as a topic scope;
A topic scope extraction device characterized by comprising: In claim 1,
Topic-in-scope vector information generating means for generating information on a word vector set in each topic scope obtained by the fuzzy clustering means;
An input sentence morpheme analyzing means for analyzing the input sentence and dividing it into word units;
An input sentence word vector acquisition means for acquiring a vector corresponding to each word obtained by the input sentence morpheme analysis means by searching the concept base;
Input sentence vector information generating means for generating information on a word vector set in the input sentence obtained by the input sentence word vector acquiring means;
The distance between the information about the word vector set in the input sentence obtained by the in-input sentence vector information generating means and the information about the word vector set in each topic scope obtained by the in-topic scope vector information generating means An input sentence / topic scope distance calculating means for calculating a distance between the input sentence and each topic scope by performing a calculation;
A topic scope extraction device characterized by comprising: A method for controlling a topic scope extraction device that extracts a set of topic scopes, which are message sets that make up a semantic group, from a message sequence in which a plurality of topics are mixed,
A message morphological analysis step of dividing each message into words by storing each message in a storage device;
A vector corresponding to each word obtained in the message morpheme analysis step is acquired and stored by searching a concept base that is a storage means in which a pair of a word and a vector expressing the meaning of the word is stored. Obtaining a word vector in a message stored in the device;
For an arbitrary message pair, based on a word vector set in each message of the pair, a distance between the message vector sets between the pair of messages is calculated, and a distance between the message vector sets is stored in a storage device. When;
A reference distance calculation step of calculating a reference distance between the pair of messages for an arbitrary message pair based on a reference relationship between the pair of messages, and storing the reference distance in a storage device;
A topic segmentation step of dividing the message sequence into a set of topic segments that are sections of the same topic from the sequence of word vectors obtained in the word vector acquisition step in the message;
For an arbitrary topic segment pair, based on the word vector set in each topic segment of the pair, a distance between the topic segment vector sets between the paired topic segments is calculated, and stored in a storage device. An inter-set distance calculation step;
For an arbitrary message pair, the inter-message vector set distance between the pair of messages obtained in the intra-message vector set distance calculation step and the reference between the pair of messages obtained in the reference distance calculation step. Based on the distance and the distance between the topic segment vector sets between the topic segments to which each message of the pair belongs, obtained in the above-mentioned topic segment vector set distance calculation step, the final distance between the pair of messages is calculated. An inter-message distance calculating step of calculating and storing in a storage device;
Based on the distance between any messages obtained in the inter-message distance calculation step, the message sequence is fuzzy clustered, and for each cluster obtained as a result, the message belonging to the cluster has a certain degree of threshold or more. A fuzzy clustering step of deriving the set as a topic scope and storing it in a storage device;
A control method for a topic scope extraction device, comprising: In claim 3,
Generating information about word vector sets in each topic scope obtained in the fuzzy clustering step and storing the information in the topic scope vector information in a storage device;
An input sentence morpheme analysis step of analyzing the input sentence and dividing it into words;
An input sentence word vector acquisition step of acquiring a vector corresponding to each word obtained in the input sentence morpheme analysis step by searching the concept base;
An input sentence vector information generation step of generating information on a word vector set in the input sentence obtained in the input sentence word vector acquisition step and storing the information in a storage device;
The distance between the information about the word vector set in the input sentence obtained in the input sentence vector information generation step and the information about the word vector set in each topic scope obtained in the topic scope vector information generation step Calculating the distance between the input sentence and each topic scope by performing calculation, and storing the distance between the input sentence and the topic scope in the storage device;
A control method for a topic scope extraction device, comprising: A program that extracts a set of topic scopes, which are message sets that make up a semantic group, from a message sequence in which multiple topics are mixed,
A message morpheme analysis procedure in which each message is divided into words by morphological analysis and stored in a storage device;
A vector corresponding to each word obtained by the message morpheme analysis procedure is acquired and stored by searching a concept base that is a storage means in which a pair of a word and a vector expressing the meaning of the word is stored. Obtaining a word vector in a message stored in the device;
For an arbitrary message pair, based on the word vector set in each message of the pair, calculate the distance between the message vector sets between the paired messages and store the distance between the message vector sets in the storage device When;
A reference distance calculation procedure for calculating a reference distance between the pair of messages based on a reference relation between the pair of messages for an arbitrary message pair, and storing the reference distance in a storage device;
A topic segmentation procedure in which the message sequence is divided into a set of topic segments that are sections of the same topic from the sequence of word vectors obtained in the word vector acquisition procedure in the message, and stored in a storage device;
For an arbitrary topic segment pair, based on the word vector set in each topic segment of the pair, a distance between the topic segment vector sets between the paired topic segments is calculated, and stored in a storage device. A procedure for calculating the distance between sets;
For any message pair, the inter-message vector set distance between the pair of messages obtained by the above-intra-message vector set distance calculation procedure and the reference between the pair of messages obtained by the above-mentioned reference distance calculation procedure. Based on the distance and the distance between the topic segment vector sets between the topic segments to which each pair of messages belongs, obtained in the procedure for calculating the distance between the vector sets within the topic segment, the final distance between the pair of messages is calculated. A procedure for calculating a distance between messages to be calculated and stored in a storage device;
Based on the distance between the messages obtained in the inter-message distance calculation procedure, the message string is fuzzy clustered, and for each cluster obtained as a result, the message belonging to the cluster is more than a threshold value. A fuzzy clustering procedure for deriving the set as a topic scope and storing it in a storage device;
A program that causes a computer to execute. In claim 5,
A topic scope vector information generating procedure for generating information on a word vector set in each topic scope obtained by the fuzzy clustering procedure and storing the information in a storage device;
An input sentence morpheme analysis procedure for analyzing an input sentence and dividing it into words;
Obtaining a vector corresponding to each word obtained by the input sentence morpheme analysis procedure by searching the concept base, and storing the word vector in the input sentence in a storage device;
An input sentence vector information generation procedure for generating information on a word vector set in the input sentence obtained by the above-described input sentence word vector acquisition procedure and storing the information in a storage device;
The distance between the information related to the word vector set in the input sentence obtained in the above-mentioned input sentence vector information generation procedure and the information related to the word vector set in each topic scope obtained in the topic scope vector information generation procedure Calculating the distance between the input sentence and each topic scope by performing calculation, and storing the distance between the input sentence and the topic scope in the storage device;
A program that causes a computer to execute.

Images