JP2011123565A - Faq candidate extracting system and faq candidate extracting program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a FAQ candidate extracting system which has resistance against the characteristics of discourse data, such as spoken language or noise, and extracts a Q&A pair from the analysis result of a discourse data structure, without regulating the frame of a discourse text structure. <P>SOLUTION: The FAQ candidate extracting system 1 uses the discourse data 101 and discourse semantics 200 as inputs and extracts and outputs a question-answer pair to be a FAQ candidate 300 from the discourse data 101. The discourse semantics 200 includes flow information 21 of respective statement. The system includes a Q&A pair extracting portion 60 for identifying the first statement, wherein a flow indicating being a question sentence is set, identifying the second statement which appears first after the first statement, is different in speaker, and where a flow indicating stating a specific content on a matter intrinsic to the discourse is set, and then, extracting the first statement and the second statement as the Q&A pair. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a natural language processing technique, and more particularly, to a FAQ candidate extraction system and a FAQ candidate extraction program for extracting FAQ (Frequently Asked Questions) candidates from the result of analyzing the structure of discourse data obtained by speech recognition processing. It is related to effective technology when applied.

  For example, in a call center operation, an operator or a communicator needs to perform a series of operations such as creating a response record and storing it in a response management system after receiving a call. At this time, there is a problem that it takes time for the operator or the like to create a response record based on his / her memory.

  Further, quality (content, uniformity, completeness, conciseness, etc.) varies according to the skill of each operator. In this way, the quality of the response records varies or the quality is low, so it is not possible to accurately grasp the customer's voice, or from the accumulation of response records, new technologies such as text mining are used. Problems such as inability to obtain knowledge also occur.

  In order to solve such a problem, a solution using a speech recognition technology is being studied. By storing the voice information of the response as text using the speech recognition technology, it is possible to ensure the completeness of the response record. It is also possible to automatically create response records by applying natural language processing technology to the text data and summarizing the contents, reducing the cost of response record creation, uniformity, It is possible to improve the usability of response records by ensuring conciseness.

  As such a technique, for example, in Non-Patent Document 1, in voice conversation at a call center, voice data is converted into text, and rules are recursively obtained by inductive learning from past conversation contents and business daily reports. It describes a technology that automatically generates daily business reports by determining and extracting important points for unknown dialogue contents using rules and performing style conversion.

  On the other hand, there are various technologies that can be used to obtain new knowledge through analysis by text mining or the like from the accumulation of voice data that has been converted into text, depending on the type of voice data interaction. From the contents of the response, it extracts the question-answer pair (hereinafter sometimes referred to as “Q & A pair”) consisting of frequently asked questions and their answers, and automatically generates FAQ (Frequently Asked Questions) Technology is envisaged.

  Conventionally, it has been done to extract frequently asked questions by statistical processing from text data such as spoken dialogue, but it is statistically extracted together with answers to this It was difficult, and it was common practice to create separate Q & A pairs for answers.

  On the other hand, for example, in Non-Patent Document 2, a television program with closed captions that has been increasing in recent years is targeted, and a question sentence is used from among closed captions in the television program using CRF (Conditional Random Fields). A technique is described in which a question-answer pair is extracted by identifying and estimating the center of an answer sentence corresponding to this, and further identifying and identifying information that introduces and supplements the answer sentence using the CRF. .

Junji Yano, Kenji Araki, “Performance evaluation of daily report generation method using inductive learning for voice conversation in call center”, Information Processing Society of Japan Research Report 2007-NL-178, March 28, 2007, p. 21-28 Masahiro Kawano and 6 others, “Question-answer pair automatic extraction from TV program closed caption using CRF”, Proc. Of the 14th Annual Conference of the Language Processing Society, March 20, 2008, p. 915-918

  When using data obtained by converting a person's discourse into text by speech recognition (hereinafter sometimes referred to as “discourse data”), the sentences in the discourse data are usually recorded in spoken language, so follow the correct grammar. Japanese language sentences and existing natural language processing technology based on written language may not be able to cope well. Moreover, the accuracy of the current speech recognition engine cannot eliminate recognition errors, and there are many noises such as typographical errors and omissions due to erroneous recognition. Therefore, it is difficult to analyze and effectively use discourse data using natural language processing.

  On the other hand, as a technique for obtaining new knowledge, such as a Q & A pair that is a candidate for FAQ, from the accumulation of voice data that has been converted into text, the technique of Non-Patent Document 2 described above is based on a closed caption of a TV program. Here, the content of the target closed caption is annotated in the schema, that is, applied to the framework of the sentence structure prescribed in advance, and the question and answer are extracted based on this It is.

  However, it is not possible to extract a Q & A pair from discourse data that cannot be applied to the framework or schema of a predetermined sentence structure, such as data obtained by making a telephone response in a call center into text. The second technique is difficult, and such a technique has not existed in the past.

  SUMMARY OF THE INVENTION An object of the present invention is to provide FAQs that extract strong Q & A pairs as FAQ candidates from the results of analyzing the structure of discourse data without specifying the framework of the discourse sentence structure, and being strong in the characteristics of discourse data such as spoken words and noise. The object is to provide a candidate extraction system and a FAQ candidate extraction program.

  The above and other objects and novel features of the present invention will be apparent from the description of this specification and the accompanying drawings.

  Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.

  The FAQ candidate extraction system according to the representative embodiment of the present invention functions by the FAQ candidate extraction program, and analyzes the discourse data in which the content of the discourse is converted into text by voice recognition processing, and the discourse structure in the discourse data. It is a FAQ candidate extraction system that extracts and outputs a question-answer pair as a FAQ candidate from the discourse data, using as a result discourse semantics, which is information for grasping the semantic content of the entire discourse as a result, It has the following characteristics.

  That is, in the FAQ candidate extraction system, the discourse semantics includes flow information including flow information indicating the semantic content of each statement in the discourse data. The FAQ candidate extraction system includes the discourse semantics in which the statement is Identifying a first statement with the flow indicating that it is a question sentence for a speaker, and further appearing first after the first statement in the discourse data, and the first statement and the speaker Is different, and identifies a second statement with the flow indicating that the statement is specific to a matter specific to the discourse, and identifies the first statement and the second statement As a question-answer pair. Question - is characterized in that it has a reply pair extraction unit.

  Among the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows.

  According to a typical embodiment of the present invention, by using flow information about each statement of discourse data, which is a result of discourse structure analysis, the discourse text structure is not defined, and the discourse structure is not defined. Q & A pairs can be extracted from the data. In addition, since Q & A pairs are extracted by referring to only necessary parts from the contents of discourse data and discourse structure analysis results, even if other parts contain noise or erroneous analysis results due to noise, Q & A pairs can be extracted by minimizing the influence of.

It is the block diagram shown about the outline | summary of the structural example of the FAQ candidate extraction system which is one embodiment of this invention. It is the block diagram shown about the outline | summary of the structural example of the discourse structure analysis system in one embodiment of this invention. It is the figure shown about the example of the discourse in one embodiment of this invention, and the concept of discourse semantics. It is the figure shown about the example in the flow and context in discourse data in one embodiment of this invention. It is the figure shown about the example of the flow analysis rule of the discourse structure analysis system in one embodiment of this invention. It is the flowchart which showed the example of the flow of the flow analysis process in the flow analysis part of the discourse structure analysis system in one embodiment of this invention. It is the figure shown about the example of the specific expression analysis rule of the discourse structure analysis system in one embodiment of this invention. It is the flowchart which showed the example of the flow of the specific expression analysis process in the specific expression analysis part of the discourse structure analysis system in one embodiment of this invention. It is the figure shown about the example of cohesion analysis in the discourse structure analysis system in one embodiment of this invention. It is the flowchart which showed the example of the flow of the cohesiveness analysis process in the cohesiveness analysis part of the discourse structure analysis system in one embodiment of this invention. It is the figure shown about the example of Q & A pair extraction in the FAQ candidate extraction system in one embodiment of this invention. It is the flowchart which showed the example of the flow of the Q & A pair extraction process in the Q & A pair extraction part of the FAQ candidate extraction system in one embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

  The FAQ candidate extraction system according to one embodiment of the present invention is based on discourse semantics (information related to discourse structure for grasping the semantic content of the entire discourse), which is the analysis result of discourse structure for discourse data including spoken words. The Q & A pair is extracted from the discourse data. Furthermore, for these, clustering is performed by calculating and grouping the similarity based on the appearance frequency of the keyword set in advance, ranking is performed by scoring for each cluster, and the result is output as a FAQ candidate. System.

  Here, discourse semantics means, for example, a flow as described below for the entire discourse data by inferring the semantic content from character strings and morphemes in the discourse data, and also a discourse such as cohesiveness (discourse segment), proper expression, etc. It is the analysis result obtained by analyzing the structure. This discourse semantics may be generated from discourse data by using a discourse structure analysis system, which will be described later, or may be generated using a sentence structure analysis system using other natural language processing, etc. . It is also possible to manually create a tag by tagging the discourse data. In the present embodiment, description will be made assuming that the generation is performed using a discourse structure analysis system described later.

  The discourse semantics in this embodiment do not have to be the result of strict analysis of the content and structure of Japanese sentences. For example, the discourse semantics have a level of accuracy that does not impede practical use, such as creating a response record at a call center. I just need it. In this specification, “discussion” refers to an utterance action by one or more speakers, for example, a speech at a lecture where one person speaks, a call center operator as described above, and a customer. Includes communication, multi-person conference calls and video conferencing, and other multi-person conversations and dialogues.

  FIG. 3 is a diagram showing an example of discourse and the concept of discourse semantics in the present embodiment. The example of FIG. 3 shows an example of a discourse consisting of interaction between a call center operator and a customer. The utterance on the left side is the utterance of the operator, and the right side shows the utterance of the customer. In the present embodiment, one sentence of each speaker is called a “statement”. In addition, a group of consecutive statements for each speaker is referred to as a “block” (even if consecutive statements of the same speaker are separated in time, they become different blocks).

  In FIG. 3, “flow” is information that simply indicates the semantic content of each statement. By tracing this information, the flow of discourse can be grasped. For example, in the operator statement of FIG. 3, the first “Yes.” Indicates that the flow is “greeting”, while the next “Yes.” Indicates that the flow is “confusion”. In this way, depending on the discourse situation (context), the flow may be different even for statements of the same wording.

  In FIG. 3, “proprietary expression” is an expression that individually characterizes the content of the discourse. For example, in the discourse of exchange at the call center in FIG. 3, the speaker ID, person name, company name, telephone number, Applicable to proper nouns such as location. Further, the “segment” is a segmented set of common (united) contents of the discourse for each block. The information about the above “flow”, “specific expression”, and “segment” is a constituent element of “Discourse Semantics” as the analysis result of the discourse structure.

[System configuration (Discourse structure analysis system)]
FIG. 2 is a block diagram showing an outline of a configuration example of the discourse structure analyzing system according to the embodiment of the present invention. The discourse structure analysis system 2 is configured by a computer system such as a server or a PC (Personal Computer). For example, the preprocessing unit 10, the flow analysis unit 20, the morpheme analysis unit 30, and the proper expression analysis unit 40 are implemented by software programs. The discourse structure having each unit of the cohesiveness analysis unit 50, each table of the flow analysis rule 22 and the specific expression analysis rule 42, and the discourse data 100 in which the content of the discourse is converted into text by the speech recognition engine 3 is input. And the discourse semantics 200 is output.

  Note that the existing voice recognition engine 3 can be used. As described above, the discourse structure analysis system 2 according to the present embodiment is characterized by being resistant to noise (such as recognition errors by the speech recognition engine 3) of the discourse data 100, but the recognition rate of the speech recognition engine 3 is higher. Needless to say, it is possible to analyze the discourse structure with higher accuracy.

  The preprocessing unit 10 of the discourse structure analysis system 2 has a function of receiving input of the discourse data 100 and performing preprocessing such as data layout conversion so that processing can be performed in other units as necessary. . Here, for example, continuous statements for each speaker are grouped into blocks, sequence numbers (IDs) indicating the order are assigned to each block and each statement, and speaker data of each block is extracted and set. Perform the process. The discourse data 100 is represented as a data structure using, for example, XML (eXtended Markup Language).

  The flow analysis unit 20 has a function of analyzing a rule-based flow based on the flow analysis rule 22 on the discourse data 100 after the preprocessing by the preprocessing unit 10 and outputting the flow information 21 as an analysis result. . The flow information 21 holds the ID of each statement in the discourse data 100 and the flow information corresponding thereto, and is represented as a data structure using XML or the like, for example.

  This flow information 21 is included in the discourse semantics 200 as an output of the discourse structure analysis system 2 and is an essential component. That is, the discourse semantics 200 includes flow information 21 as a flow analysis result for at least the discourse data 100. Details of the flow analysis rule 22 and details of processing in the flow analysis unit 20 will be described later.

  The morpheme analysis unit 30 performs a so-called morpheme analysis on the discourse data 100 after the preprocessing by the preprocessing unit 10 and divides the discourse data into columns of morphemes (the smallest unit having meaning in the language) to determine the part of speech. As a result, the morpheme information 31 is output. The morpheme information 31 holds the ID of each statement in the discourse data 100 and morpheme string information corresponding thereto, and is represented as a data structure using, for example, XML.

  Similar to the flow information 21, the morpheme information 31 is included in the discourse semantics 200 as an output of the discourse structure analysis system 2, and becomes an input to the specific expression analysis unit 40 and the cohesiveness analysis unit 50 described later. In other words, when the specific expression analysis by the specific expression analysis unit 40 and the cohesiveness analysis (segmentation) by the cohesiveness analysis unit 50 are not performed, the morpheme analysis by the morpheme analysis unit 30 may be unnecessary. In the present embodiment, in the FAQ candidate extraction system to be described later, the analysis result by the cohesiveness analysis unit 50 and the morpheme information 31 itself may be used. Therefore, the morpheme information 31 is output as necessary. Also good. Note that an existing morpheme analysis engine (for example, MeCab (Japanese cloth candy) or ChaSen (tea candy)) can be used for the morpheme analysis unit 30.

  The specific expression analysis unit 40 is a rule-based specific expression based on the specific expression analysis rule 42 based on the discourse data 100 after the preprocessing by the preprocessing unit 10 and the morpheme information 31 output from the morpheme analysis unit 30. And the specific expression information 41 is output as an analysis result. That is, the analysis process in the specific expression analysis unit 40 is based on the premise that at least the morpheme analysis by the morpheme analysis unit 30 is performed. The specific expression information 41 holds information of each statement ID in the discourse data 100 and a list of specific expressions corresponding to the statement ID, and is represented as a data structure using, for example, XML.

  The specific expression information 41 is included in the discourse semantics 200 as an output of the discourse structure analysis system 2, as with the flow information 21, but the usage form by the application that uses the discourse semantics 200 thereafter and the required discourse semantics are included. It may be unnecessary depending on accuracy. Therefore, the output of the specific expression information 41 is not essential, and the user may be able to select whether or not to output. Details of the specific expression analysis rule 42 and details of the processing in the specific expression analysis unit 40 will be described later.

  The cohesiveness analysis unit 50 determines each block based on the discourse data 100 after the preprocessing by the preprocessing unit 10, the flow information 21 output from the flow analysis unit 20, and the morpheme information 31 output from the morpheme analysis unit 30. The cohesiveness is analyzed and segmented, and the cohesiveness information 51 is output as an analysis result. That is, the analysis processing in the specific expression analysis unit 40 is premised on that at least morpheme analysis by the morpheme analysis unit 30 and flow analysis by the flow analysis unit 20 are performed. The cohesiveness information 51 holds segment ID information corresponding to each block ID in the discourse data 100, and is represented as a data structure using XML or the like, for example.

  The cohesiveness information 51 is included in the discourse semantics 200 as an output of the discourse structure analysis system 2 as in the case of the specific expression information 41. However, the usage form by the application using the discourse semantics 200 after that and the required discourse semantics are included. Depending on the accuracy, etc., it may be unnecessary. Therefore, the output of the cohesiveness information 51 is not essential, and the user may be able to select whether or not to output. In the present embodiment, the cohesiveness information 51 may be used in the FAQ candidate extraction system described later, and the cohesiveness information 51 may be output according to the situation. Details of processing in the cohesiveness analysis unit 50 will be described later.

  As described above, the discourse semantics 200 output as the analysis result by the discourse structure analysis system 2 includes the flow information 21, the morpheme information 31, the unique expression information 41, and the cohesiveness information 51. At this time, a set of these data may simply be handled as the discourse semantics 200, or these data may be combined into one data structure using XML, for example.

[System configuration (FAQ candidate extraction system)]
FIG. 1 is a block diagram showing an outline of a configuration example of a FAQ candidate extraction system according to an embodiment of the present invention. The FAQ candidate extraction system 1 is configured by a computer system such as a server or a PC, similar to the discourse structure analysis system 2, and includes, for example, a Q & A pair extraction unit 60, a clustering unit 70, and a scoring unit 80 implemented by a software program. Each part has a table of the keyword dictionary 72, and the discourse data 101 in which the content of the discourse is converted into text, and all or part of the discourse semantics 200 that is the analysis result of the discourse structure of the discourse data 101 (particularly, the flow information 21 , The cohesiveness information 51 and the morpheme information 31) are input, and a Q & A pair is extracted from the discourse data 101 and output as a FAQ candidate 300.

  Note that the discourse data 101 is equivalent to that preprocessed by the preprocessing unit 10 of the discourse structure analysis system 2 described above, that is, for example, continuous statements for each speaker are collected as blocks. A sequence number (ID) indicating an order is assigned to each statement, and the speaker data of each block is set. The discourse semantics 200 is output as a result of the discourse structure (particularly flow, cohesiveness, and morpheme) being analyzed for the discourse data 101 by the discourse structure analysis system 2 described above.

  In the present embodiment, the analysis result of the discourse structure output by the discourse structure analysis system 2 is input to the FAQ candidate extraction system 1. For example, these systems are configured on one computer system. The processing from the discourse structure analysis processing to the FAQ candidate 300 extraction may be executed continuously.

  The Q & A pair extraction unit 60 of the FAQ candidate extraction system 1 extracts Q & A pairs in the discourse data 101 based on the discourse data 101, the flow information 21 and the cohesiveness information 51 of the discourse semantics 200, and Q & A pair information 61 is output. The Q & A pair information 61 holds information of a question statement or statement ID and a list of answer statements or statement IDs for the question. Details of the processing in the Q & A pair extraction unit 60 will be described later.

  For each Q & A pair in the Q & A pair information 61, the clustering unit 70 generates a word vector from the words (keywords) included in the keyword dictionary 72 based on the morpheme information 31 of the discourse semantics 200 and It has a function of calculating the similarity, and grouping each Q & A pair based on the calculated similarity.

  Further, the scoring unit 80 performs scoring for each cluster clustered by the clustering unit 70, extracts, for example, a predetermined number of upper clusters (Q & A pairs) based on the scoring result, and the FAQ candidate 300 is extracted. As an output function. Details of processing in the clustering unit 70 and the scoring unit 80 will be described later.

[Flow analysis]
Below, the detail of the process of the flow analysis by the flow analysis part 20 of the discourse structure analysis system 2 shown in FIG. 2 is demonstrated. FIG. 4 is a diagram illustrating an example of a flow and context in the discourse data 100. In the present embodiment, a context is set for each block of discourse. In the example of FIG. 4, the context of the first block (speaker: operator) is “start”, and the next block (speaker: customer) and the next block (speaker: operator) are both “declaration”. It has become.

  As shown in the example of FIG. 4, in this embodiment, in principle, the flow set in the last statement of the block immediately before a different speaker is set as the context of the target block. Since there is no immediately preceding block for the first block of the discourse data 100, “start” is always set as the context. By determining the context in this way, it is possible to analyze the flow along the discourse context even if the statements have the same wording.

  FIG. 5 is a diagram illustrating an example of the flow analysis rule 22 of the discourse structure analysis system 2. The flow analysis rule 22 is held in a file, database, or the like, and includes an expression for determining a flow in a statement (hereinafter sometimes referred to as “flow expression”) and an expression that matches the flow expression. It is a table that holds a list designated in advance with respect to correspondence with a flow pattern (hereinafter, sometimes referred to as “flow definition”) set in accordance with the context of a statement it has.

  As shown in the example of FIG. 5, the flow expression is specified by a regular expression of a character string (for example, “/Yes./” or “/ ?? /”) that serves as a clue for determining the flow. Accordingly, the flow analysis unit 20 or the discourse structure analysis system 2 needs to be able to perform processing necessary for regular expressions by introducing, for example, an existing regular expression library.

  Note that the character string specified as the flow expression should preferably specify, for example, a greeting at the beginning of a sentence, a reply, a particle, a conjunction, a sentence ending expression, etc. Specify in descending order. As for the priority, for example, the complexity of the regular expression may be calculated according to a predetermined condition, and the priority may be set in descending order of the complexity. Note that the flow expression pattern to be specified may differ depending on the type of discourse targeted by the discourse structure analysis system 2 (for example, exchange at a call center or regular meeting in a company).

  For each flow expression, for example, a flow definition is specified by a label “flow”. In the flow definition, set “greeting” as the flow if the context is “start” in the statement that matches the context of the target statement and the flow that is set in the statement (for example, “/Yes./”) Specify). A plurality of flow definitions by the label “flow” can be specified according to the type of context assumed in each flow expression.

  In addition, for example, a default flow (for example, a flow expression of “/Yes./”, which is a flow to be set when the context of the target statement does not correspond to any “flow” label by the label “default”. In the statement that matches, if the context of the statement does not match any of the “flow” labels, the flow is set to “contrast”. Note that one default flow is always set in each flow expression.

  For the flow definition, it is desirable to specify a main pattern as a general discourse flow pattern according to the type of discourse targeted by the discourse structure analysis system 2. For example, in the exchange at the call center, the flow definition can be specified based on a pattern such as “apology” if the customer makes a “complaint”.

  In the flow analysis rule 22, as described above, the flow expression to be specified and the content of the corresponding flow definition may differ depending on the type of discourse targeted by the discourse structure analysis system 2. A plurality of flow analysis rules 22 to be applied can be held so that the user can be switched, so that it is possible to flexibly cope with the characteristics of the discourse.

  As described above, in this embodiment, the part of speech is not specified in the flow analysis rule 22. That is, since the flow analysis unit 20 does not determine the part of speech, it is not necessary to perform the morpheme analysis by the morpheme analysis unit 30 in advance in the analysis process in the flow analysis unit 20. In order to improve accuracy, the morpheme information 31 is used, for example, a regular expression in the flow expression of the flow analysis rule 22 is applied to a morpheme sequence (including part-of-speech information) of the morpheme information 31 as described later. It is also possible to perform matching under more detailed conditions as applied.

  FIG. 6 is a flowchart showing an example of the flow of flow analysis processing in the flow analysis unit 20 of the discourse structure analysis system 2. When starting the flow analysis process, the flow analysis unit 20 first starts a loop process that repeats the process for each statement of the discourse data 100 after the preprocessing by the preprocessing unit 10 (S101). When loop processing for each statement is started, loop processing for repeating processing for each flow expression of the flow analysis rule 22 is started (S102).

  When the loop processing for each flow expression is started, it is determined whether or not the target statement matches the target flow expression (S103). If there is no match, the process proceeds to the next flow expression process (S108, S102). On the other hand, if there is a match, first, for the target statement, the flow set in the last statement in the block immediately before the block to which the statement belongs is set as a context (S104). At this time, as shown in FIG. 4, when the block to which the statement belongs is the first block of the discourse data 100, “start” is set in the context.

  Next, with respect to the context set in step S104, it is determined whether or not there is a context that matches the context specified by the “flow” label of the flow definition in the target flow expression (S105). If there is a match, the flow corresponding to the context in the matching flow definition is set as the flow of the target statement (S106). On the other hand, if there is no match, the default flow is set as the flow of the target statement (S107). Thereafter, the loop processing for each flow expression is exited.

  When the loop processing for each flow expression is completed, it is determined whether or not a flow is set in the target statement (S109). If the flow is set, nothing is performed. If the flow is not set, the “statement” flow is set in the target statement (S110), and the process proceeds to the next statement (S111). , S101).

  Here, the “statement” flow is set when the target statement does not include a word that matches the flow expression specified in the flow analysis rule 22. In other words, it is not a statement (statement that includes a flow expression) that corresponds to a predetermined pattern or a typical pattern in the framework of discourse, but generally a statement that describes specific details about matters specific to discourse. It is believed that there is.

  When all the loop processes for each statement are completed, the ID of each statement and the flow information corresponding thereto are output as the flow information 21 and the flow analysis process is terminated.

  In this way, flow analysis that considers spoken language is used to match each statement, and by setting an appropriate flow according to the context of the matched statement, flow analysis that is less susceptible to spoken language and noise for the entire discourse data 100 is performed. It can be carried out.

[Specific expression analysis]
Hereinafter, the details of the specific expression analysis processing by the specific expression analysis unit 40 of the discourse structure analysis system 2 shown in FIG. 2 will be described. FIG. 7 is a diagram showing an example of the specific expression analysis rule 42 of the discourse structure analysis system 2. The specific expression analysis rule 42 is held in a file, a database, or the like, and a regular expression for specifying a specific expression in a statement and a rule for specifying and extracting a specific expression from a portion that matches the regular expression It is a table that holds a list designated in advance for correspondence with (hereinafter may be described as “specific expression rules”).

  As shown in the example of FIG. 7, the specific expression in the statement is specified by a regular expression. Therefore, the specific expression analysis unit 40 or the discourse structure analysis system 2 needs to be able to perform processing necessary for regular expressions by introducing, for example, an existing regular expression library. In the specific expression analysis, it is necessary to determine not only the regular expression for the normal character string but also the expression of the morpheme string of each statement held in the morpheme information 31. It is expanded so that the information can be judged.

  For example, in the example of FIG. 7, the range of the morpheme string is specified by “()” in each regular expression, and this part is matched with the information of the morpheme string instead of the character string. In the first regular expression example, the part of “(<f: organization> +)” indicates one or more morpheme sequences analyzed that the part of speech (f) is “organization”. (?: Of)? "Indicates a part of speech of 0 or more and 1 or less analyzed that the surface shape is" no ". The part of “(<f: person name, surname>)” indicates one morpheme string that has been analyzed with the part of speech being “person name, surname”. This regular expression matches expressions such as “AAA Shoji Suzuki” and “AAA Shoji Suzuki”. Note that the representation of the morpheme string may vary depending on the output specifications of the morpheme analysis engine or the like in the morpheme analyzer 30.

  At this time, in the corresponding specific expression rule, the first element in the morpheme string (the morpheme string whose part of speech is “organization”, for example, “AAA Shoji”) is specified as a specific expression having the attribute “company name”. The second element in the column (the morpheme string whose part of speech is “person name, surname”, for example “Suzuki”) is specified to be specified as a specific expression having the attribute “person name”. It should be noted that a portion (for example, “no”) enclosed by “(?:)” Is treated as not included in the element of the target morpheme string.

  In the example of the second regular expression of FIG. 7, the part of speech is specified by a normal regular expression without particular consideration, and matches the expression of a telephone number such as “03-1234-5678”. At this time, in the corresponding specific expression rule, the first element in the morpheme string (in this example, there is only one element such as “03-1234-5678”) is defined as a specific expression having the attribute “phone number”. It is specified to be specified.

  It is desirable that the character string of the unique expression specified by the regular expression is, for example, a proper noun that individually characterizes the content of the discourse or a character string made up of numerical values. In addition, for example, a character string composed of expressions (for example, complaint expression or gratitude expression) indicating the emotion of a speaker (customer, etc.) is specified and specified as a specific expression having a corresponding emotion attribute. May be. In addition, as a result of analysis, a certain character string may be included in a plurality of unique expressions.

  In the specific expression analysis rule 42, as described above, a list of correspondence between the regular expression for specifying the specific expression and the specific expression rule is specified in advance. As with the flow analysis rule 22, the normal expression to be specified is specified. For the expression and the corresponding specific expression rule, different ones are set according to the type of discourse targeted by the discourse structure analysis system 2, or the user switches the specific expression analysis rule 42 to be applied according to the type of discourse. It is possible to respond flexibly according to the characteristics of discourse.

  FIG. 8 is a flowchart showing an example of the flow of the specific expression analysis process in the specific expression analysis unit 40 of the discourse structure analysis system 2. When the specific expression analysis unit 40 starts the specific expression analysis process, first, the specific expression analysis unit 40 starts a loop process that repeats the process for each statement of the discourse data 100 (S201). When the loop processing for each statement is started, loop processing for repeating the processing for each regular expression for specifying the specific expression of the specific expression analysis rule 42 is started (S202).

  When the loop processing for each regular expression is started, it is determined whether or not each morpheme sequence in the morpheme information 31 corresponding to the target statement matches the target regular expression (S203). If there is no match, the processing proceeds to the next regular expression (S205, S202). On the other hand, if a match is found, a specific expression is identified and extracted by applying a specific expression rule corresponding to the target regular expression (S204), and the process proceeds to the next regular expression process (S205, S202).

  When all the loop processes for each regular expression are completed, the process proceeds to the next statement (S206, S201). When all the loop processing for each statement is completed, the ID of each statement and the information of the list of specific expressions corresponding thereto are output as the specific expression information 41, and the specific expression analysis process is ended.

  In this way, by performing matching with a regular expression considering the part of speech of the morpheme string and extracting the morpheme string, it is possible to perform a specific expression analysis considering a grammar peculiar to spoken words such as omission of a particle.

[Cohesion analysis]
In the following, details of cohesiveness analysis processing by the cohesiveness analysis unit 50 of the discourse structure analysis system 2 shown in FIG. 2 will be described. FIG. 9 is a diagram showing an example of cohesiveness analysis in the discourse structure analysis system 2. In the cohesiveness analysis process, first, as shown in the upper diagram, all the blocks in the discourse data 100 are set as one segment.

  Thereafter, segment division processing is performed first. In the dividing process, it is determined for each statement that a flow of a specific type is set in the flow information 21, and a segment is divided in a block to which the statement belongs. Here, it is desirable that the type of the flow to be identified is a flow that is estimated to appear when the discontinuity or content changes.

  For example, in the case of multi-person discourse such as exchanges at a call center, it is often the case that the discourse progresses depending on the response of the question and the response to it, so the content is often switched in the “question” statement of the flow. It is done. Therefore, in the present embodiment, the statement of the flow “question” (hereinafter sometimes referred to as “question statement”) is determined based on the flow information 21, and as shown in the middle diagram of FIG. Each time a question statement appears, the segment shall be split.

  Here, the question statement is a statement that is identified in the flow analysis unit 20 of the discourse structure analysis system 1 as having a flow expression indicating a question sentence for the other speaker. Examples of the flow expression for specifying the question statement include an end sentence expression of a question sentence such as “/ ?? /” and an expression representing an action such as “/ I want to know /”.

  After dividing the segments, segment integration processing is performed. In the integration process, when there is a question statement corresponding to a predetermined integration condition for each segment, the segment having the question statement is integrated with the immediately preceding segment. Here, the integration condition is designation of an expression that causes the segment having the question statement to be inferred that the content continues from the previous segment (highly cohesive with the previous segment).

  Examples of the integration condition include a case where the question statement has a word (proprietary noun) common to the immediately previous question statement or a case where a directive is included. A segment having a question statement corresponding to these integration conditions is determined to be highly cohesive with the previous segment, and is integrated with the immediately preceding segment as shown in the lower diagram of FIG. It should be noted that the integration process is repeated until there are no segments that can be integrated.

  FIG. 10 is a flowchart showing an example of the flow of cohesiveness analysis processing in the cohesiveness analyzing unit 50 of the discourse structure analyzing system 2. When the cohesiveness analysis unit 50 starts cohesiveness analysis processing, first, all the blocks of the discourse data 100 are set as one segment (S301). Specifically, for example, the initial value of the segment ID is assigned to all the blocks. After that, as a segment dividing process, a loop process for repeating the process for each block of the discourse data 100 is started (S302).

  When the loop processing for each block is started, it is determined whether or not a question statement is included in the target block (whether or not the flow has a “question” statement) (S303). If a question statement is included, the segment is divided between the target block and the immediately preceding block (S304). Specifically, for example, the next segment ID is assigned to all blocks after the target block. Thereafter, the processing moves to the next block (S305, S302).

  When the loop processing (division processing) for each block is completed, next, loop processing that repeats the processing for each segment divided by the above-described division processing is started as segment combination processing (S306). When the loop processing for each segment is started, it is determined whether or not there is a question statement corresponding to the integration condition in the target segment (S307).

  If there is a question statement corresponding to the integration condition, the target segment and the previous segment are integrated (S308). Specifically, for example, the integration is performed by updating the segment ID of each block included in the target segment to the segment ID of the immediately preceding segment. At this time, the segment ID of each block included in each subsequent segment may be sequentially incremented. Thereafter, the process proceeds to the next segment (S309, S306). When the loop processing (integration processing) for each segment is completed, information on a list of segment IDs corresponding to the ID of each block (including each block) is output as the cohesiveness information 51 to perform cohesiveness analysis. The process ends.

  Thereby, each statement in the discourse data 100 can be segmented based on the flow information 21 obtained by the flow analysis, and can be used as useful information for various applications using the discourse semantics 200.

  The flow information 21, the unique expression information 41, and the cohesiveness information 51 included in the discourse semantics 200 output by the processes of the flow analysis, the specific expression analysis, and the cohesiveness analysis described above are described in the discourse structure analysis system 2. An interface that allows the user to refer to the contents and make corrections as appropriate may be provided.

[Q & A pair extraction]
Details of the Q & A pair extraction process by the Q & A pair extraction unit 60 of the FAQ candidate extraction system 1 shown in FIG. 1 will be described below. FIG. 11 is a diagram illustrating an example of Q & A pair extraction in the FAQ candidate extraction system 1. FIG. 11 shows an example of the result of the discourse structure analysis for the discourse data 101, and shows the contents of the flow information 21 in the discourse semantics 200 (the flow of each statement) for each statement of the discourse data 101. .

  In the FAQ candidate extraction system 1 of the present embodiment, a question statement is identified from the analysis result of the discourse structure as shown in the example of FIG. 11 and is identified as a “question”. Appearing, the flow is a statement of “statement” (sometimes referred to as “statement statement” below) and the speaker identifies a different statement and identifies it as an “answer” The “answer” statement is the Q & A pair.

  Here, as described in the processing of the flow analysis unit 20 of the discourse structure analysis system 2 in FIG. 6, the statement statement does not include a statement that matches the flow expression specified in the flow analysis rule 22 for the statement. In this case, a “statement” flow is set. In other words, it is not a statement that corresponds to a predetermined pattern or a typical pattern in the discourse framework, but is generally a statement that describes specific details about matters unique to discourse. Therefore, when such a statement statement is issued in the context of “question”, it can be determined that this is an “answer” to the “question”.

  FIG. 12 is a flowchart showing an example of the flow of Q & A pair extraction processing in the Q & A pair extraction unit 60 of the FAQ candidate extraction system 1. When starting the Q & A pair extraction process, the Q & A pair extraction unit 60 first starts a loop process that repeats the process for each statement of the discourse data 101 (S401). When the loop processing for each statement is started, it is determined based on the flow information 21 of the discourse semantics 200 whether or not the target statement is a question statement (S402). If it is not a question statement, the processing proceeds to the next statement (S407, S401).

  If the target statement is a question statement in step S402, loop processing for repeating the processing for each statement after the question statement is started (S403). When the loop processing for each statement after the question statement is started, it is determined based on the flow information 21 whether the target statement is a statement statement in which the question statement and the speaker are different (S404). If it is not the corresponding statement statement, the process proceeds to the next statement (S406, S403).

  If the target statement is a corresponding statement in step S404, the question statement and the statement statement are extracted as a Q & A pair (S405), the loop processing for each statement after the question statement is exited, and the statement The process proceeds to the next statement after the question statement (S407, S401). Even when the loop processing for each statement after the question statement is completed, the process proceeds to the processing of the next statement after the question statement (S407, S401). When the loop processing for each statement is completed, the list of Q & A pairs extracted in step S405 is output as Q & A pair information 61, and the Q & A pair extraction processing is terminated.

  In the example of FIG. 12, processing is performed for each statement of the discourse data 101 as a whole. However, based on the cohesiveness information 51 of the discourse semantics 200, each statement in a predetermined segment for each segment of the discourse data 101 is processed. By performing processing as a target, for example, the range of statements that are targets when identifying “answers” may be narrowed down. At this time, the “answer” statement may be identified for a segment different from the segment to which the question statement belongs.

  As described above, it is necessary to identify and extract the Q & A pair based on the flow information set in each statement of the discourse data 101, and to apply the discourse data 101 to the determined sentence structure framework or schema. In addition, Q & A pairs can be extracted from the discourse data 101 having various discourse flows.

[Clustering ~ Scoring]
In the following, details of clustering and scoring of Q & A pairs in the Q & A pair information 61 by the clustering unit 70 and scoring unit 80 of the FAQ candidate extraction system 1 shown in FIG. 1 will be described. In the present embodiment, for each Q & A pair in the Q & A pair information 61, first, the clustering unit 70 calculates a word vector based on a preset keyword, and resembles each Q & A pair based on the calculated word vector. Group things into clusters. Further, the scoring unit 80 scores each cluster (or a Q & A pair that represents the cluster) according to a predetermined criterion, and based on the scoring result, for example, assigns a predetermined upper number of Q & A pairs to the FAQ candidate 300. Output as.

  When calculating the above word vector, the clustering unit 70 calculates, for each Q & A pair in the Q & A information 61, an index indicating the degree of association with the keyword for each keyword preset in the keyword dictionary 72. Then, a word vector having each calculated index as an element is calculated. That is, when the number of keywords set in the keyword dictionary 72 is n, an n-dimensional word vector is calculated. As the keyword dictionary 72, for example, a product dictionary generated from a product master or the like in a company that uses the FAQ candidate extraction system 1 can be used.

  Here, as the index indicating the degree of association with each keyword for each Q & A pair, various indicators can be used as long as they can be evaluated numerically. In the present embodiment, for example, a commonly used index, a TF (Term Frequency) value that is an appearance frequency of a target word (keyword) in a sentence (Q & A pair), and all sentences (Q & A pair). ), A TF-IDF value that is a product of an IDF (Inverse Document Frequency) value that is the inverse of the number of sentences (Q & A pairs) in which the target word (keyword) appears is used. Thereby, it is possible to determine that a keyword having a large TF-IDF value is a word having high importance in the Q & A pair.

  When the clustering unit 70 further performs clustering by grouping into similar ones based on the word vectors calculated for each Q & A pair, for example, the clustering unit 70 is a technique commonly used, which is a method of each word vector in the vector space. A method is used in which the degree of similarity is determined based on an algebraic distance (or an angle formed between word vectors), and clustering is performed assuming that a distance closer than a predetermined distance (or a smaller angle than a predetermined angle) is similar. The reference for clustering is not limited to the distance between the word vectors, and other reference may be used. Further, for example, considering the temporal discreteness between Q & A pairs in the discourse data 101, the degree of similarity between the questions with a small ID difference (question statements that are close in time) of each Q & A pair is increased. You may weight such as.

  For each Q & A pair clustered by the above method, each cluster is scored by scoring processing by the scoring unit 80 of the FAQ candidate extraction system 1 shown in FIG. Of clusters (or Q & A pairs representing the clusters) are output as FAQ candidates 300.

  In scoring, for example, the algebraic size of each word vector of each Q & A pair in the cluster, the number of each keyword in the keyword dictionary 72, the size of the cluster (the number of Q & A pairs), and so on The score can be calculated based on the criteria. Also, the Q & A pair that represents each cluster can be determined by a method such as selecting a Q & A pair having the largest word vector size in the cluster, for example. The format of the FAQ candidate 300 is not particularly limited, and various formats such as a data structure using XML or a simple text format can be used.

  In this way, the Q & A pairs included in the Q & A pair information 61 output from the Q & A pair extraction unit 60 are subjected to clustering and scoring processing to narrow down and output the Q & A pairs that are the FAQ candidates 300. May be.

  As described above, according to the FAQ candidate extraction system 1 of the present embodiment, the sentence structure of the discourse is obtained by using the flow information 21 for each statement of the discourse data 101, which is the analysis result of the discourse structure. Q & A pairs can be extracted from the discourse data 101 without prescribing the framework. Further, by calculating the similarity based on the appearance frequency of the keyword from the extracted Q & A pair, clustering, and scoring each cluster, the FAQ candidate 300 can be effectively extracted from the Q & A pair and output. .

  Further, when extracting the Q & A pair, the Q & A pair is extracted by referring to only the necessary part from the contents of the discourse data 101 and the analysis result of the discourse structure, so that the other part includes noise and an erroneous analysis result due to noise. Even in such a case, the Q & A pair can be extracted while minimizing the influence from these.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

  The present invention can be used for a FAQ candidate extraction system and a FAQ candidate extraction program for extracting FAQ candidates from the result of analyzing the structure of discourse data obtained by voice recognition processing.

1 ... FAQ candidate extraction system, 2 ... Discourse structure analysis system, 3 ... Speech recognition engine,
10: Pre-processing unit,
20 ... Flow analysis unit, 21 ... Flow information, 22 ... Flow analysis rule,
30 ... Morphological analysis unit, 31 ... Morphological information 40 ... Specific expression analysis unit, 41 ... Specific expression information, 42 ... Specific expression analysis rule,
50: Cohesiveness analysis unit, 51 ... Cohesiveness information,
60 ... Q & A pair extraction unit, 61 ... Q & A pair information,
70 ... clustering unit, 72 ... keyword dictionary,
80 ... Scoring part,
100, 101 ... Discourse data, 200 ... Discourse semantics, 300 ... FAQ candidates.

Claims (8)

With the input of discourse semantics, which is information for grasping the semantic content of the whole discourse, which is the analysis result of the discourse structure in the discourse data, and the discourse data in which the content of the discourse is converted into text by voice recognition processing, A FAQ candidate extraction system for extracting and outputting question-answer pairs as FAQ candidates from the discourse data,
The discourse semantics includes flow information including flow information indicating the semantic content of each statement in the discourse data,
In the discourse semantics, identify the first statement with the flow indicating that the statement is a question sentence to the other speaker, and further appear first after the first statement in the discourse data And identifying a second statement with the flow indicating that the speaker is different from the first statement, and that the statement describes a specific content about matters unique to the discourse A FAQ candidate extraction system comprising: a question-answer pair extraction unit that extracts the first statement and the second statement as the question-answer pair. The FAQ candidate extraction system according to claim 1,
In addition, a keyword dictionary that holds multiple keywords,
For each question-answer pair output from the question-answer pair extraction unit, a word vector is calculated based on each keyword held in the keyword dictionary, and each of the above-described values is calculated based on the distance of the word vector. A clustering unit that determines the degree of similarity of the question-answer pairs and groups the question-answer pairs for each cluster;
A scoring unit that scores each of the clusters according to a predetermined criterion and outputs the question-answer pairs that are representative of the upper predetermined number of the clusters as FAQ candidates. FAQ candidate extraction system. The FAQ candidate extraction system according to claim 2,
When calculating the word vector, the clustering unit calculates a TF value and an IDF value for each keyword for each question-answer pair, and based on a product of the TF value and the IDF value. A FAQ candidate extraction system characterized by calculating a TF-IDF value and calculating the word vector having the TF-IDF value as an element. In the FAQ candidate extraction system according to any one of claims 1 to 3,
The discourse semantics further includes cohesiveness information including information on segments obtained by dividing the discourse data in units in which content switching is estimated,
The FAQ candidate extraction system, wherein the question-answer extraction unit identifies the first statement and the second statement for the statement included in the predetermined segment. With the input of discourse semantics, which is information for grasping the semantic content of the entire discourse, which is the analysis result of the discourse structure in the discourse data, and the discourse data in which the discourse content is converted into text by speech recognition processing, A FAQ candidate extraction program for causing a computer to function as a FAQ candidate extraction system that extracts and outputs FAQ candidate question-answer pairs from the discourse data,
The discourse semantics includes flow information including flow information indicating the semantic content of each statement in the discourse data,
In the discourse semantics, a first statement having a first flow indicating that the statement is a question sentence for the other speaker is identified, and further, the first statement is first after the first statement in the discourse data. And a second flow is set that indicates that the speaker is different from the first statement, and that the statement describes a specific content about matters unique to the discourse. A FAQ candidate extraction program comprising: a question-answer pair extraction unit that identifies a statement and extracts the first statement and the second statement as the question-answer pair. In the FAQ candidate extraction program according to claim 5,
In addition, a keyword dictionary that holds multiple keywords,
For each question-answer pair output from the question-answer pair extraction unit, a word vector is calculated based on each keyword held in the keyword dictionary, and each of the above-described values is calculated based on the distance of the word vector. A clustering unit that determines the degree of similarity of the question-answer pairs and groups the question-answer pairs for each cluster;
A scoring unit that scores each of the clusters according to a predetermined criterion and outputs the question-answer pairs that are representative of the upper predetermined number of the clusters as FAQ candidates. FAQ candidate extraction program. In the FAQ candidate extraction program according to claim 6,
When calculating the word vector, the clustering unit calculates a TF value and an IDF value for each keyword for each question-answer pair, and based on a product of the TF value and the IDF value. A FAQ candidate extraction program characterized by calculating a TF-IDF value and calculating the word vector having the TF-IDF value as an element. In the FAQ candidate extraction program according to any one of claims 5 to 7,
The discourse semantics further includes cohesiveness information including information on segments obtained by dividing the discourse data in units in which content switching is estimated,
The FAQ candidate extraction program, wherein the question-answer extraction unit identifies the first statement and the second statement for the statement included in the predetermined segment.

Images