JP4512826B2 - Question answering system - Google Patents

Description

  The present invention relates to a question answering system that answers an action as a solution to a question from a user in some troubled situation.

  Due to the spread of the Internet, there is a great variety of information on the Web (abbreviation of World Wide Web). For this reason, there are increasing opportunities to search for answers from information on the web using search engines for everyday questions. However, in order to use an existing search engine, it is necessary to accurately express the contents of the question with keywords. Furthermore, the user himself / herself has to search for a word or a sentence to be answered from a huge number of searched pages.

By the way, a question answering system has been studied in order to answer a question sentence in a natural language. Existing research is broadly divided into two types: those that make inferences based on limited domain knowledge and rules (inference type systems) and those that extract responses from unorganized document sets (extraction type systems). Is done. In recent years, the latter extraction system has been mainly studied.
Hereinafter, the above-described conventional technology will be described in detail.

  Research on question answering in the field of artificial intelligence dates back to the 1960s. Specifically, it is an expert system or a dialogue system, which is an “inference type system” that answers by inference. These have the feature of refining questions and answers through dialogue between the system and the user, and deriving answers from multiple facts by inference. However, since domain knowledge and inference rules need to be constructed manually, there are problems such as limited fields that can be answered and poor expandability.

  On the other hand, in the field of natural language processing and information retrieval, since the late 1990s, research on questions and answers has been conducted in TREC (Text Retrieval Conference: http://trec.nist.gov/) and NTCIR (Information Collection System Evaluation Test Collection Project). Became popular. Unlike inference systems in the field of artificial intelligence, “extraction systems” are the main research subjects here. Since an unorganized document set is used as an information source, it differs from an inference type system in that domain knowledge and inference rules need not be constructed.

  The answer target of the question answering system in TREC and NTCIR is an objective fact. Answers with nouns to WH question sentences centered on who, when, where, and what entered by the user. Examples of systems that use objective facts as answers include goo lab Japanese natural language search (http://labs.nttrd.com/) and MURAX (see Non-Patent Document 1 below). MURAX analyzes the input question sentence to make a hypothesis about what question it is and what the answer is. Based on the hypothesis, a query is constructed and searched, and the corresponding part is searched from the document set and answered.

Non-Patent Document 2 below proposes a knowledge acquisition method for the purpose of answering a “how type question” for asking behavior. By extracting important sentences from the text of question mails / answer mails posted on the mailing list of Linux users, knowledge for answering questions about methods and countermeasures is obtained. However, in this method, the target area is limited.
Julian Kupiec. MURAX: Finding and Organizing Answers from Text Search. In T. Strzalkowski, editor, Natural Language Information Retrieval, pp. 311-332. Kluwer Avademic Publishers, 1999. Yasuhiko Watanabe, Kazuya Sono and Yoshihiro Okada. “Acquisition of Knowledge for Question Answering System Using Mailing List”, Research Report of Information Processing Society of Japan, 2004-NL-162, pp. 131-138, 2004. Takenobu Tokunaga, Makoto Iwayama, Kentaro Inui, Hozumi Tanaka. "Japanese word order estimation model and its application", Information Processing Society of Japan, 91-NL-81, pp. 9-16, 1991.

The above-mentioned extraction type system mainly responds with nouns to WH question sentences centering on who, when, where, and what, and therefore it always matches the answer that the user wants to know (especially an expression that expresses behavior). Is not limited. In addition, the research corresponding to the how-type question that asks for behavior is limited to the usage of computers and the like.
In view of the above circumstances, the present invention provides a question answering system that can apply an extraction type question answering system to the web and answer a how type question that asks an action without limiting the area. That is, it provides a help desk-oriented question answering system that suggests an action to be taken in response to a question from a user in some troubled situation.

In order to achieve the above object, the present invention provides
In [1] question answering system, it is connected to the web, and the question extraction unit that extracts as a FAQ question representation you search frequently a web page, is connected to the question extractor, connected to the web, a search unit to exhaustively search the page associated with frequently asked questions extracted by the query extractor with search engine, connected to the search unit, answers to the frequently asked questions from the retrieved page a representation of the become action, an answer extraction unit for extracting a plurality of paragraphs including representing this action is connected to the answer extraction unit, a plurality of paragraphs extracted by the answer extraction section, the same action an answer organization unit for classifying for each group containing a representation of, our previously stored in the questions extracted well extracted by the unit questions and the answer pairs and classified answered organized unit And FAQ database unit, enter the question, and a user interface device reply is output, is connected to the FAQ database unit and the user interface device, often questions stored in the FAQ database unit, the user interface device A determination unit that determines whether there is a question that is the same as the question input to the answer, and when the determination unit determines that there is the same question, an answer corresponding to the question input to the user interface device, A question input to the user interface device when the expression representing the action or a paragraph including the expression representing the action is presented as a unit of an answer from the FAQ database unit, and the determination unit determines that there is no same question Japanese to the retrieval and answer extraction and replies organizing performed to generate the answer presented to To.

[2] In the above [1], wherein question answering system, a combination of threshold for number of characters in the extraction of the paragraph, and extracting the following 120 characters or more 200 characters as a paragraph.
[3] In the above [1], wherein the question answering systems, the answer extraction unit scored representation of the action included in the extracted multiple paragraphs, representation of the action contained in each paragraph The scores of the paragraphs are scored by summing the scores .

[4] In the above [3], wherein the question answering systems, the score of representation of the action, to select a representation of the appropriate action as an answer, the short distance of the receiving (a) relates, (b) Recommended It is accompanied by expressions and prohibited expressions, (c) high in rank in the search of the source page, (d) close to the expression representing the action included in the question, (e) the questioner (user) should a behavior, characterized Rukoto to be scored using the five criteria.

According to the present invention, the following effects can be achieved.
(A) It is possible to answer a how-type question without limiting the area. That is, a help desk-oriented question answering system that suggests an action to be taken is provided to a user in some troubled situation.
(B) Since the FAQ database unit for storing frequently asked questions and the answers to the frequently asked questions is provided in advance, quick answers can be presented to the frequently asked questions.

Question answering system of the present invention is connected to the web, and the question extraction unit that extracts as a FAQ question representation you search frequently a web page, is connected to the question extractor, connected to the web, a search unit to exhaustively search the page associated with frequently asked questions extracted by the query extractor with search engine, connected to the search unit, answers to the frequently asked questions from the retrieved page a representation of the become action, an answer extraction unit for extracting a plurality of paragraphs including representing this action is connected to the answer extraction unit, a plurality of paragraphs extracted by the answer extraction section, the same action and answers organized unit which classifies every group that contains expressions that describe and stored in advance in the question extracted well extracted by the unit questions and the answer pairs and classified answered organized unit F And Q database unit, enter the question, and a user interface device reply is output, is connected to the FAQ database unit and the user interface device, often questions stored in the FAQ database unit, the user interface device A determination unit that determines whether there is a question that is the same as the question input to the answer, and when the determination unit determines that there is the same question, an answer corresponding to the question input to the user interface device, A question input to the user interface device when the expression representing the action or a paragraph including the expression representing the action is presented as a unit of an answer from the FAQ database unit, and the determination unit determines that there is no same question In response to the above search, answer extraction and answer organization, an answer is generated and presented .

Hereinafter, embodiments of the present invention will be described in detail.
The system of the present invention searches a web page related to an inputted question sentence, classifies and presents the extracted answers. The questions are questions that ask what-if, what-to-do, and how-to-do behaviors. In order to answer these questions, we focus on “ expressions that represent actions (hereinafter simply referred to as action expressions) ” consisting of noun phrases and verbs. For example, to the question “What should I do if I am stung by a bee?”, The behavioral expressions such as “pull out the needle”, “wash with running water”, and “apply ointment” and the paragraph including the behavioral expression are answered.

FIG. 1 is a configuration diagram of a question answering system of the present invention.
In this figure, 1 is a user interface device, 2 is a question (question text: question information), 3 is an answer (answer text: answer information), 10 is a preprocessing section, 11 is a response section, 12 is a FAQ database section, 13 Is a web, 14 is a question extracting unit, 15 is a discriminating unit for discriminating whether or not it is a common question, 16 is a searching unit, 17 is an answer extracting unit, and 18 is an answer organizing unit.

In this system, since the answer is extracted by analyzing the searched page, it takes more time to respond than a normal web search. Therefore, in the present invention, a question that is likely to be asked and a response to the question are databased in advance as FAQ (Frequently Asked Questions), thereby enabling a quick answer.
First, pre-processing (offline processing) for constructing the FAQ database will be described. Here, "question extraction" by the question extraction unit 14, "search" by the search unit 16, "answer extraction" by the answer extraction unit 17, and "answer organization" by the answer organization unit 18 are performed step by step.

  First, the question extraction unit 14 extracts questions that frequently appear on the web 13. Next, the search unit 16 comprehensively searches for pages related to the question extracted by the question extraction unit 14 using an existing search engine. Next, the answer extraction unit 17 extracts a paragraph including the action expression from the page searched by the search unit 16. Next, the answer organizing unit 18 classifies the extracted paragraph based on the included action expression, and accumulates the classified answers and questions in pairs in the FAQ database unit 12.

  Next, online processing when the user uses the system will be described. When the user inputs the question 2 to the user interface device 1, the discrimination unit 15 searches for the same question in the FAQ database unit 12. If there is the same question, the corresponding answer is extracted from the FAQ database unit 12 and presented. If there is no same question, the search unit 16 searches, the answer extraction unit 17 extracts the answers, the answer organization unit 18 organizes the answers, and the dynamically created answers are sent to the answer unit 11 as in the pre-processing. Generate and present.

FIG. 2 is a screen example showing the answer.
The answer is presented on a screen as shown in FIG. This is an example of an answer screen for the question “What should I do if a bee stings?” The heading expression “going to the hospital” is an action expression that represents the group after organizing, and “pull out the needle” and “get a medical examination” below it are other action expressions included in this group. The paragraphs containing these behavioral expressions and a link to the page from which they are extracted are written together to make it easier to obtain detailed information.

That is, the “answer” of the present system is a phrase of action expression such as “go to hospital” and, at the same time, a sentence like a paragraph including the action expression. The user can select display contents such as a group list of action expressions and a list of paragraphs in the group depending on the usage environment.
Hereinafter, each process of question extraction, search, answer extraction, and answer organization will be described.

(1) Question extraction Since various question expressions are latent on the web 13, such as questions prepared on FAQ sites and simple questions written on personal diary sites and web logs, these are collected.
Expressions such as “What happens when XX is △?” And “What happens when XX is △△?” Are regularized with words and parts of speech, and the corresponding phrases and sentences are extracted. Specifically, an expression satisfying the rule that “noun is followed by a particle (ga.ni.)”, verb (˜after), followed by a question mark or “(,) what to do” is extracted.

  At this time, since morphemes are also used, it is possible to extract words that do not match the surface expression. On the other hand, expressions (noise) other than questions may be detected. However, even if the FAQ contains noise, there is no real harm unless the user asks a similar question including the noise. Therefore, in the question extraction, the recall (recall) is more important than the precision (precise).

NTCIR (http://research.nii.ac.jp/ntcir/index-ja.html) web collection was used as extraction source data. This is a collection of 11,038,720 pages (about 100 GB) collected from the JP domain. As a result, 6961 phrases were extracted.
(2) Search The search site Google is used for the web 13 search, and the inputted question sentence is used as it is as a query. However, in order to expand the scale of the experiment, it is necessary to independently develop a search engine suitable for question answering.

(3) Answer Extraction The answer extraction unit 17 extracts a paragraph that becomes an answer including an action expression from the page group searched by the search unit 16 and performs scoring. Although there are PDF files and the like on the actual web 13, in the present invention, since it is necessary to perform a layout analysis of the text, only HTML files are targeted.

Hereinafter, a detailed extraction procedure will be described.
First, a paragraph having a certain length is extracted from each retrieved page. This is because the answers extracted in units of words and sentences have little information for explaining the behavior, and conversely, unnecessary information is likely to be included in units of pages. For distinguishing paragraphs, <P> and <BR> were used among HTML tags.

  The usage of tags varies depending on the page creator, and variations in the length of paragraphs occur only by using tags for discrimination. For example, in a page using <P> indicating a paragraph instead of a line feed, each sentence becomes a paragraph and information is insufficient. Conversely, reading very long paragraphs is burdensome for the user. Therefore, a threshold value regarding the number of characters is used in combination, and 120 to 200 characters are defined as a paragraph.

Next, the paragraph is scored according to its appropriateness as an answer. Since the action expression is the minimum unit answers the system of the present invention, paragraph Line Dohyo current is contained more is important as the answer. Therefore, the action expression was formalized and used for scoring.
In the scoring of a paragraph, first, a score is assigned to each behavioral expression appearing on the searched page, and the scores of the behavioral expressions included in the paragraph are summed up.

Here, behavioral expressions will be described.

Is used to perform dependency analysis on the text portion of the retrieved page. From this, a dependency structure of “noun + particle + verb” is extracted and set as the minimum unit of action expression. For example, in the case of the sentence “Apply medicine to the affected area”, two phrases “Apply + drug +” and “Apply + affected area +” are obtained. Here, if “drug is applied”, the particle is omitted as in “drug: apply” and this is used as an action expression.

Even if the particle changes due to alternation, the action content of “who will do what” does not change, and the same action expression is omitted, such as “a person stings a bee” and “a bee stings a person” Because it is considered.
This method can also treat an expression in which a plurality of noun phrases (nouns + particles) are related to a verb, such as “paint medicine on the affected area”, as an action expression. However, when there are a plurality of noun phrases, there is a problem that “painting medicine” and “painting medicine on the affected part” are regarded as different behavior expressions. Therefore, the experiment limited the noun phrase to one.

In scoring action expressions, the following five criteria are used to select an appropriate action expression as an answer.
(A) The dependency distance is short.
The dependency distance is the number of morphemes between the noun phrase and the verb. In the case of Japanese, an element closer to the end of the sentence has the property of limiting the meaning of the verb (see Non-Patent Document 3 above). In addition, there is an advantage that the shorter the distance, the fewer errors in dependency analysis.

(B) It is accompanied by a recommended expression such as “to do” and “to do” and a prohibited expression such as “do not do”.
Recommended expressions are used to describe effective countermeasures for problem solving. In addition, because prohibited expressions are used when describing measures that should not be performed, they are as useful as recommended expressions. For example, in the example of the question “If you are stung by a bee?”, “Let's apply medicine”, “Go to the hospital”, etc. are recommended expressions, “Do not put ammonia”, “Do not crush poison bags” "Please" is a prohibited expression. Such expressions are collated and detected by regular expressions.

(C) The rank in the search of the extraction source page is high.
The extraction source page is a page from which action expressions are extracted. Since Google is currently used for the search, the rank of the extraction source page means the search rank of Google. Google uses PageRank, and pages that are frequently linked from other pages are ranked high. Therefore, it is possible to partially incorporate reliability. This score is calculated by the following formula.

(Number of search pages-rank) / number of search pages (d) The distance from the behavioral expression included in the question within the paragraph is short.
This is judged from the number of morphemes between the verb in the action expression and the verb in the question. The closer the verb in the behavioral expression is to the verb in the question, the stronger the relationship with the question, and the more important behavioral expression appears as a coping method.

(E) Actions to be taken by the questioner (user).
In order for the questioner to extract the behavior that is the main actor, the behavioral expression in which the subject is clearly indicated by the case particle “ga” (hereinafter “ga”) lowers the score. For example, if the sentence “bee protects the nest”, the actor is “bee”. The behavioral expressions “protecting bees” and “protecting the nest” appearing in this sentence are both behavioral expressions performed by the bees and are not appropriate as answers to the questioner.

  The score of each action expression is calculated by combining the above criteria (a) to (e) by the following formula (1).

Here, x is an action expression that appears in a paragraph. ax is the dependency distance, and dx is the distance to the question. bx takes 1 or 0 depending on the presence / absence of recommended / prohibited expressions. cx is a value calculated in (c) above. ex takes 1 or 0 depending on the presence or absence of the case.

  The score of paragraph P is obtained by summing up the scores of behavioral expressions included in P by the following equation (2).

(4) Answer organization The answer organization unit 18 classifies a plurality of paragraphs extracted by the above-described answer extraction method. Here, paragraphs containing the same action expression are grouped into the same group. However, behavioral expressions with low scores are noisy and are deleted in advance. At present, classification is performed based on behavioral expressions having a score of 2 or more. When presenting answers to the user, it is possible to save the trouble of reading multiple similar paragraphs by presenting the highest score paragraph and action expression for each group. It becomes easy to do.

In addition, action expressions included in the same group are in a dependency relationship, and conversely, action expressions not belonging to the same group are in an exclusive relationship. Such a relationship between behavior expressions is a clue for extracting an order relationship or dependency relationship of behavior.
(Evaluation experiment)
Hereinafter, an evaluation experiment of the question answering system according to the present invention will be described.

Since a how-type question answering system has hardly been studied, an evaluation method has not been established. Therefore, a test collection was created manually, and the evaluation was focused on the answer extraction function, which is the most important function in the system of the present invention, and in particular, the scoring of behavioral expressions.
This is because the system of the present invention groups paragraphs based on high-scoring behavioral expressions and presents the highest-scoring behavioral expression within the group to the user as representative (ie, “go to hospital” in FIG. This is because it is important that the appropriate action expression score is higher than others, so the five criteria (a) to (e) previously proposed are effective for scoring action expressions. Evaluation was performed from the viewpoint of whether or not there was.

The question used the following 13 phrases with high question frequency among the phrases extracted by question extraction.
(1) If you forget your password, (2) If you have an accident, (3) If you get lost, (4) If you have a sudden illness, (5) If you are stolen, (6) If you are stung by a bee, (7 ) If you get injured, (8) If you have acne, (9) If you have a seizure, (10) If you lose your passport, (11) If you get sexual harassment, (12) If you get burned, (13) Become a fire When these questions were asked, the appropriateness of the extracted behavioral expression was determined manually. In addition, appropriate expressions were also judged as prohibited expressions and treated as correct answers.

For the behavioral expressions ranked by score, F values were calculated using the accuracy and recall shown below.
Here, the F value (F-measure) is a harmonic average of a recall and a precision. This is an evaluation scale that takes into account both the recall and accuracy. The larger the recall and accuracy, the greater the F value.

Now, for all possible behavioral expressions for a question,
The following contingency table can be created by classifying from the viewpoint of “whether it is an action expression output by the system as an answer?” Or “whether it is a correct action expression for the question?”.

Here, A to D are the number of action expressions in the corresponding cell, and A + B + C + D is the total number of action expressions. The cells A to C correspond to the following cases.
A: Success, B: Failure (extraction failure), C: Failure (extraction error)
Based on the above contingency table, the recall and accuracy are calculated by the following equations, respectively.

Reproducibility = A / (A + B)
Accuracy = A / (A + C)
Since the F value is a harmonic average of recall (R) and accuracy (P), it is calculated below.
First, an arithmetic average is calculated for the reciprocals of R and P.

Further, taking the reciprocal, F value = 2A / (2A + B + C)
That is, the F value represents the ratio of “success” to “success and failure (extraction failure, extraction error)”.

Summary,
Accuracy = Number of correct answers output / Number of all action expressions output Recall rate = Number of correct answers output / Number of correct answers in all action expressions F value = 2 x Recall rate x Accuracy / (Recall rate + Accuracy)
(Results and discussion)
In order to examine the transition of the F value according to the number of responses, the behavioral expressions were increased step by step, and the F values were calculated for the top 200 behavioral expressions. For all questions used in the evaluation, F values at each point where the number of behavioral expressions was changed were calculated, and the average values were used for comparative evaluation.

The results are shown in FIG. The vertical axis of the graph represents the average F value, and the horizontal axis represents the number of behavior expressions. Each series uses the value evaluated under the following conditions, and averages all questions.
all: ranking by score using all of criteria (a) to (e) Pa: ranking by score using criteria (a) “dependency distance” Pb: criteria (b) “presence / absence of recommended / prohibited expressions” Pc: Ranking by score using criteria (c) “ranking in search” Pd: Ranking by score using criteria (d) “distance from behavioral expression included in question” Pe: Ranking by score using the standard (e) “behavioral expression with ga nouns” For comparison,
freq: Indicates the result of evaluation under the condition of ranking by appearance frequency.

First, when comparing freq and others, the average F value is increased by using any of the criteria except for the criterion (d), and the result is improved. It can be said that it is a standard.
Further, when all was compared with Pa to Pe, all showed better results regardless of the number of behavioral expressions to be output. That is, it was found that there is a synergistic effect by the combination of standards.

In the above evaluation experiment, a survey on a how-type question was shown. However, according to the present invention, in principle, it can be applied to a where-to-do question and a where-to-do question. Can be answered from a broad perspective.
In addition, this invention is not limited to the said Example, Based on the meaning of this invention, a various deformation | transformation is possible and these are not excluded from the scope of the present invention.

  The question answering system of the present invention can be used as a question answering system that can answer a question asking for an action without limiting the area.

It is a block diagram of the question answering system which shows the Example of this invention. It is a figure which shows the example of an answer screen in the question answering system which shows the Example of this invention. It is a F value graph by each condition in the example of an experiment of the present invention.

1 User Interface Device 2 Question (Question Text: Question Information)
3 answers (answer text: answer information)
DESCRIPTION OF SYMBOLS 10 Pre-processing part 11 Answer part 12 FAQ database part 13 Web 14 Question extraction part 15 Discrimination part 16 Search part 17 Answer extraction part 18 Answer organization part

Claims (4)

Is connected to the (a) web, and the question extraction unit that extracts as frequently asked questions the question expression you searched frequent the web page,
(B) is connected to the question extractor, connected to the web, a search unit comprehensively searches for pages related to frequently asked questions extracted by the query extractor with search engine,
( C ) connected to the search unit, an expression representing an action that is an answer to the frequently asked question from the searched page , and an answer extraction unit that extracts a plurality of paragraphs including the expression representing the action ;
( D ) an answer organizing unit that is connected to the answer extracting unit and classifies a plurality of paragraphs extracted by the answer extracting unit into groups each including an expression representing the same action ;
( E ) a FAQ database unit that stores in advance a pair of frequently asked questions extracted by the question extraction unit and the answers classified by the answer organization unit;
( F ) a user interface device for inputting a question and outputting an answer;
( G ) a determination unit that is connected to the FAQ database unit and the user interface device and determines whether the common question stored in the FAQ database unit has the same question as the question input to the user interface device ; Comprising
(H) When it is determined by the determination unit that there is the same question, an answer corresponding to the question input to the user interface device is set to an expression representing the action or a paragraph including an expression representing the action. If the determination unit determines that there is no same question, the search, answer extraction, and response organization are performed on the question input to the user interface device to generate an answer. A question answering system characterized by being presented . According to claim 1, wherein the question answering systems, question answering system that a combination of the threshold for number of characters in the extraction of the paragraph, and extracting the following 120 characters or more 200 characters as a paragraph. 2. The question answering system according to claim 1, wherein the answer extraction unit scores the expressions representing the actions included in the extracted plurality of paragraphs, and sums the scores of the expressions representing the actions included in the respective paragraphs. By doing so , scoring of each stage is performed . In claim 3, wherein the question answering systems, the score of representation of the action, to select a representation of the appropriate action as an answer,
(A) The dependency distance is close (b) The recommended expression and the prohibited expression are accompanied (c) The rank in the search of the extraction source page is high (d) The distance from the expression representing the action included in the question is close ( question answering system characterized Rukoto to be scored using the five criteria are actions to be e) question (user).