WO2021130964A1 - 情報提供方法 - Google Patents

情報提供方法 Download PDF

Info

Publication number
WO2021130964A1
WO2021130964A1 PCT/JP2019/051143 JP2019051143W WO2021130964A1 WO 2021130964 A1 WO2021130964 A1 WO 2021130964A1 JP 2019051143 W JP2019051143 W JP 2019051143W WO 2021130964 A1 WO2021130964 A1 WO 2021130964A1
Authority
WO
WIPO (PCT)
Prior art keywords
question
sentence
user
candidate sentence
behavior
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/051143
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
直美 河村
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to US17/637,175 priority Critical patent/US20220309942A1/en
Priority to JP2021566686A priority patent/JP7131720B2/ja
Priority to PCT/JP2019/051143 priority patent/WO2021130964A1/ja
Publication of WO2021130964A1 publication Critical patent/WO2021130964A1/ja
Anticipated expiration legal-status Critical
Priority to JP2022131632A priority patent/JP7343014B2/ja
Priority to JP2023138146A priority patent/JP7722420B2/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers
    • G09B7/02—Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers
    • G09B7/06—Electrically-operated teaching apparatus or devices working with questions and answers of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers
    • G09B7/07—Electrically-operated teaching apparatus or devices working with questions and answers of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers providing for individual presentation of questions to a plurality of student stations
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers
    • G09B7/06—Electrically-operated teaching apparatus or devices working with questions and answers of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers
    • G09B7/10—Electrically-operated teaching apparatus or devices working with questions and answers of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers wherein a set of answers is common to a plurality of questions
    • G09B7/12—Electrically-operated teaching apparatus or devices working with questions and answers of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers wherein a set of answers is common to a plurality of questions characterised by modifying the teaching program in response to a wrong answer, e.g. repeating the question or supplying further information

Definitions

  • the present invention relates to an information providing method, an information providing system, and a program that provide information in response to a question from a user.
  • chatbot system As a system that automatically answers questions entered by users using a web server on the Internet or a computer installed in a store.
  • Q & A data consisting of a combination of a question candidate sentence and an answer sentence is stored in advance, the question sentence input by the user is analyzed, and the question candidate sentence corresponding to the question sentence is extracted.
  • the chatbot asks the user to select the question candidate sentence closest to the content that the user wants to ask from the question candidate sentences, and displays the answer sentence associated with the selected question candidate sentence.
  • Patent Document 1 there is one described in Patent Document 1.
  • the chatbot has introduced a function to get feedback from the user whether the answer finally presented is correct. For example, after displaying an answer, the chatbot asks the user for input to evaluate whether the answer is "correct,” “wrong,” or "unresolved.” As a result, the chatbot improves the accuracy of the subsequent response to the user's question by analyzing and learning the user's evaluation of the answer.
  • an object of the present invention is to provide an information providing method capable of solving the above-mentioned problem that the accuracy of the response to a question in a chatbot cannot be improved.
  • the information providing method which is one embodiment of the present invention, is According to the question text input by the user, the question candidate text corresponding to the question text is output to the user, and the question text is output to the user. Detects the user's behavior with respect to the question candidate sentence and The question candidate sentence is evaluated for the question sentence according to the behavior. It takes the configuration.
  • the information providing system which is one form of the present invention, is A question-and-answer section that outputs a question candidate sentence corresponding to the question sentence to the user according to the question sentence input from the user.
  • a detection unit that detects the user's behavior with respect to the question candidate sentence
  • An evaluation unit that evaluates the question candidate sentence for the question sentence according to the behavior, and an evaluation unit. With, It takes the configuration.
  • the program which is one form of the present invention is For information processing equipment A question-and-answer section that outputs a question candidate sentence corresponding to the question sentence to the user according to the question sentence input from the user. A detection unit that detects the user's behavior with respect to the question candidate sentence, An evaluation unit that evaluates the question candidate sentence for the question sentence according to the behavior, and an evaluation unit. To realize, It takes the configuration.
  • the present invention is configured as described above, so that the accuracy of the response to the question in the chatbot can be improved.
  • FIG. It is a block diagram which shows the structure of the chatbot in this invention. It is a figure which shows an example of the data stored in the chatbot disclosed in FIG. It is a figure which shows an example of the data stored in the chatbot disclosed in FIG. It is a figure which shows an example of the data stored in the chatbot disclosed in FIG. It is a figure which shows an example of the data stored in the chatbot disclosed in FIG. It is a figure which shows an example of the data stored in the chatbot disclosed in FIG. It is a figure which shows the state at the time of the question answer operation by the chatbot and the user terminal disclosed in FIG. It is a figure which shows the state at the time of the question answer operation by the chatbot and the user terminal disclosed in FIG. It is a figure which shows the state at the time of the question answer operation by the chatbot and the user terminal disclosed in FIG.
  • FIG. It is a figure which shows the state at the time of the question answer operation by the chatbot and the user terminal disclosed in FIG. It is a figure which shows the state at the time of the question answer operation by the chatbot and the user terminal disclosed in FIG. It is a figure which shows the state at the time of the question answer operation by the chatbot and the user terminal disclosed in FIG. It is a figure which shows the state at the time of the question answer operation by the chatbot and the user terminal disclosed in FIG. It is a figure which shows the state at the time of the question answer operation by the chatbot and the user terminal disclosed in FIG. It is a flowchart which shows the operation of the chatbot disclosed in FIG. It is a block diagram which shows the hardware structure of the information providing system in Embodiment 2 of this invention. It is a block diagram which shows the structure of the information providing system in Embodiment 2 of this invention. It is a flowchart which shows the operation of the information providing system in Embodiment 3 of this invention.
  • FIGS. 1 to 6 are diagrams for explaining the configuration of the chatbot
  • FIGS. 5 to 6 are diagrams for explaining the processing operation of the chatbot.
  • the chatbot 10 in the present invention is composed of a web server connected to a network, receives a question from a user terminal 20 (information processing device) operated by the user U, and automatically answers the question. It functions as an information providing system that provides information processing.
  • the chatbot 10 is managed by a predetermined company and provides an answer to a question from an employee (user) in the company, or is managed by a business operator that provides a predetermined product or service. It automatically provides answers to questions about products and services from users who have accessed via the network.
  • the chatbot 10 in the present invention may be used in any situation and may provide any information.
  • the chatbot 10 is not necessarily limited to being configured by an information processing system that receives a question from the user terminal 20 via a network and provides an answer.
  • the chatbot 10 is an information processing system composed of installed terminals installed in a store or the like, and is configured to directly receive a question or provide an answer from a user via text information or voice information. You may.
  • the chatbot 10 in the present embodiment is composed of one or a plurality of information processing devices including an arithmetic unit and a storage device. Then, as shown in FIG. 1, the chatbot 10 includes a question-and-answer unit 11, a detection unit 12, and an evaluation unit 13, which are constructed by the arithmetic unit executing a program. Further, the chatbot 10 includes a Q & A data storage unit 14, a correct answer rate data storage unit 15, and an evaluation data storage unit 16 formed in the storage device.
  • a question-and-answer unit 11 a detection unit 12
  • an evaluation unit 13 which are constructed by the arithmetic unit executing a program.
  • the chatbot 10 includes a Q & A data storage unit 14, a correct answer rate data storage unit 15, and an evaluation data storage unit 16 formed in the storage device.
  • the question-and-answer unit 11 outputs the chat screen A1, the message input field A2, and the send button A3 to the display screen of the accessing user terminal 20 as shown in FIG. 5A.
  • the chat screen A1 is a screen that displays the exchange of messages between the user U and the operator P.
  • the message input field A2 is an input field in which the user U inputs a question text via the user terminal 20, and when the send button A3 is pressed, the question text is transmitted to the chatbot 10 and sent to the chatbot 10. Accepted.
  • the question answering unit 11 receives the question text from the user terminal 20, as shown in FIG. 5B, the question answering unit 11 outputs to display the question text from the user U in the question column U1 of the chat screen A1, and responds accordingly. , Output so that the question candidate sentence is displayed in the response column P1 of the operator P on the chat screen A1.
  • the question answering unit 11 searches and extracts the question candidate sentences corresponding to the question sentences from the user U in the Q & A data storage unit 14, and extracts the extracted question candidate sentences in the response column P1 of the chat screen A1. Display in.
  • the Q & A data storage unit 14 stores Q & A data composed of a combination of a question candidate sentence and an answer sentence prepared in advance.
  • Q & A data in the Q & A data of QAID "QA1", the question candidate sentence "Please tell me the procedure for maternity leave” and the answer sentence "Procedure for maternity leave can be found at the following site ... ⁇ ⁇ (URL) ”is registered.
  • the Q & A data may be a combination of question candidate sentences and answer sentences of any content.
  • the question answering unit 11 extracts the question candidate sentences corresponding to the question sentences from the user U from the question candidate sentences in the Q & A data storage unit 14.
  • the question answering unit 11 stores a model in which the question candidate sentences (Q & A data) corresponding to the question sentences are machine-learned in advance, and by inputting the question sentences from the user U into such a model, one question / answer unit 11 is used.
  • a plurality of question candidate sentences are output, and the output question candidate sentences are extracted.
  • the extraction of the question candidate sentence corresponding to the question sentence by the question answering unit 11 may be performed by another known method, or may be performed by any method.
  • the question-and-answer unit 11 displays a list of the question-and-answer sentences extracted as described above in the response column P1 from the operator P, as shown in FIG. 5B.
  • the question answering unit 11 since the question answering unit 11 has extracted a plurality of question candidate sentences (1, 2, 3), all of them are displayed in the response column P1.
  • the question answering unit 11 displays and outputs each question candidate sentence so that the user U can select it.
  • the number of question candidate sentences displayed by the question answering unit 11 in the response column P1 is arbitrary. It should be noted that the question answering unit 11 is inappropriate for the question text itself because it can determine that the question text itself is not appropriate when no question candidate text is extracted for the question text. An evaluation to that effect may be given.
  • the question-and-answer unit 11 provides a response column with a list of the above-mentioned question candidate sentences (1, 2, 3) and an option such as "See more questions” that the user U wants to output other question candidate sentences. Displayed on P1. At this time, the question answering unit 11 displays and outputs the option of "more questions” so that the user U can select it.
  • the phrase "see more questions” may be another phrase, as long as the user U wants to output another question candidate sentence in addition to the displayed question candidate sentence.
  • the question and answer unit 11 provides an option such as "not in this” in the response column P1.
  • the option of "not in this” is an option for telling that the question candidate sentence desired by the user U does not exist in the question candidate sentences already displayed, and the wording is another wording. You may. Then, the question-and-answer unit 11 displays and outputs the option “not in this” so that the user U can select it.
  • the question-and-answer unit 11 further displays and provides various information to the user U on the chat screen A1 according to the behavior of the user U detected by the detection unit 12, which will be described later, and details thereof will be described later. To do.
  • the detection unit 12 detects the behavior of the user U after displaying the question candidate sentence or the like in the response column P1 of the chat screen A1. For example, the detection unit 12 detects the selection status by the user U for each option including the question candidate sentence displayed in the response column P1 as the behavior of the user U.
  • the user U is the user terminal among the question candidate sentences (1, 2, 3) displayed in the response column P1, "more questions", and "not in this”. By operating 20, the one selected on the chat screen A1 displayed on the display screen of the user terminal 20 is detected.
  • the detection unit 12 displays a question candidate sentence or the like in the response field P1, and then another question sentence is further input from the user U in the message input field A2, and the send button A3 is pressed. Is detected.
  • the other question texts input at this time are accepted by the question answering unit 11 described above, and the question candidate texts and the like corresponding to the other question texts similarly described above are output to the user terminal 20.
  • the detection unit 12 ends the question such as closing the chat screen A1 without selecting the question candidate sentence or the like in the response column P1 or inputting another question sentence as described above. Also detect.
  • the detection unit 12 outputs the answer sentence corresponding to the question candidate sentence in the question answer unit 11, and the answer sentence corresponds to the question candidate sentence.
  • the behavior of the user U with respect to the (second behavior) is also detected. For example, when the detection unit 12 includes a link (address information) to another web page in which a more detailed answer is posted in the response text as the behavior of the user U, the link is the user U. Detects whether or not it was selected by. Further, the detection unit 12 detects the degree of evaluation when the evaluation of the answer is input by the user U following the answer sentence. For example, when a button indicating that the answer was useful to the user U and a button indicating that the answer was not useful are displayed, which button is selected is detected.
  • the explanation returns to the above-mentioned question-and-answer section 11.
  • the question answering unit 11 displays the question candidate sentences in response to the question from the user U, and then further provides information to the user U according to the behavior of the user U detected by the detection unit 12. Take various measures such as providing. In the following, the behavior of the user U will be described by adding a "pattern number".
  • the question is asked.
  • the answering unit 11 ends the answering process without continuing to answer the question.
  • the question answering unit 11 described above. As described above, other question candidate sentences corresponding to the first question sentence are further extracted from the Q & A data storage unit 14. Then, as shown in FIG. 5C, the question answering unit 11 displays "more questions” in the question column U1 of the user U on the chat screen A1, and further extracts it in the response column P1 of the operator P. Display another question candidate sentence (4,5,6) and other options. Even when the user U selects "not in this" for the question candidate sentence or the like displayed in the response column P1 shown in FIG. 5B (Pattern 6), the question answering unit 11 is the same as described above. Further extract other question candidate sentences corresponding to the first question sentence, or end the answer process without continuing the answer to the question.
  • the question answering unit 11 asks.
  • the Q & A data including the selected question candidate sentence is specified from the Q & A data storage unit 14, and the answer sentence associated with the selected question candidate sentence in the specified Q & A data is read out.
  • the question answering unit 11 displays "question candidate 1" in the question column U1 of the user U on the chat screen A1 and is selected in the response column P1 of the operator P. Display the "answer text" corresponding to the question candidate text.
  • the question answering unit 11 considers that the answer to the question is completed and ends the answer processing.
  • the question / answer unit 11 shows it in FIG. 5D. As such, a link to another web page is also displayed in the response field P1 of the chat screen A1. Then, when the link included in the answer text is selected by the user U, the question-and-answer unit 11 displays and provides the linked web page on the user terminal 20 (pattern 8). Further, as shown in FIG. 5E, the question answering unit 11 displays an input means for inputting the degree of evaluation by the user for the answer following the answer sentence.
  • address information for example, URL
  • an input means for example, for the user U, a button indicating that the answer was useful and a button indicating that the answer was not useful are displayed. Then, when any button is selected by the user U (pattern 9), the question-and-answer unit 11 ends the response corresponding to the question from the user.
  • the question-and-answer unit 11 may display not only the above-mentioned two buttons but also an input means capable of inputting an evaluation in three or more stages as an input means for inputting the degree of evaluation by the user for the answer. Good.
  • the answer sentence corresponding to the question candidate sentence is already displayed, but the user U is still another question candidate for the question candidate sentence or the like displayed in the response column P1.
  • the question answering unit 11 displays another selected question candidate sentence in the question column U1 of the user U as shown in FIG. 5F. .. Then, although not shown, the question-and-answer unit 11 displays the answer sentence corresponding to another question candidate sentence in the response column P1 as described above.
  • the question answering unit 11 receives a new question sentence and displays it in the question column U1 of the user U of the chat screen A1 in the same manner as described above.
  • FIG. 5B even when the user U does not select the question candidate sentence displayed in the response field P1 and the user U inputs a new question sentence in the message input field A2 (pattern 7). ), The question answering unit 11 receives a new question sentence as described above, and displays it in the question column U1 of the user U of the chat screen A1 as described above. Then, the question answering unit 11 extracts a new question candidate sentence corresponding to the new question sentence and displays it in the response column P1 in the same manner as described above.
  • the evaluation unit 13 evaluates the question candidate sentence for the question sentence according to the behavior of the user U detected by the detection unit 12 as described above.
  • the correct answer rate is set for each pattern number corresponding to the behavior of the user U described above, and the evaluation unit 13 corresponds to the actually input question sentence and the question sentence.
  • the question candidate sentences extracted in the above process are associated with the evaluation data containing as a pair.
  • the evaluation unit 13 determines the user ID of the user U who input the question, the input question sentence, the QAID of the Q & A data including the extracted question candidate sentence, the correct answer rate, and the correct answer rate. Are associated with each other and stored in the evaluation data storage unit 16 as evaluation data.
  • the correct answer rate is a value indicating the degree to which the question candidate sentence output corresponding to the question sentence input from the user U is considered to be correct, and as shown in the correct answer rate table of FIG. It is assumed that it is set for each pattern number according to the behavior.
  • the evaluation unit 13 calculates the correct answer rate of the question candidate sentence extracted corresponding to the question sentence according to the behavior of the user U detected by the detection unit 12 as follows. First, consider the case of pattern 1, that is, the case where the user U finishes the question such as closing the chat screen A1 without selecting anything for the displayed question candidate sentence or the like. In this case, since it can be judged that all the question candidate sentences extracted corresponding to the question sentences have a high probability of being "wrong answers", the correct answer rate is set to "-0.5", and that value. Is calculated as the correct answer rate of all question candidate sentences for the question sentences.
  • pattern 3 that is, the user U selects one question candidate sentence (for example, question candidate 1) for the displayed question candidate sentence, and the answer sentence corresponding to the question candidate sentence is
  • question candidate sentence for example, question candidate 1
  • the answer sentence corresponding to the question candidate sentence is
  • the correct answer rate is set to "0.5"
  • the value is set for the question sentence. Calculated as the correct answer rate of the selected question candidate sentences.
  • the user U selects one question candidate sentence (for example, question candidate sentence 1) for the question candidate sentence or the like, and the answer sentence corresponding to the question candidate sentence. Is displayed, but then consider the case where the user U selects yet another question candidate sentence (for example, question candidate 2).
  • the question candidate sentence (for example, question candidate sentence 1) first selected by the user U is considered to have a high probability of being an "wrong answer", but since it has been selected once, the correct answer rate is "-0". .4 ”is set, and the value is calculated as the correct answer rate of the question candidate sentence selected first for the question sentence.
  • a question candidate sentence or the like is displayed corresponding to the first question sentence from the user U, and the user U selects one question candidate sentence for the question candidate sentence or the like. Then, the answer sentence corresponding to the selected question candidate sentence is displayed, but after that, consider the case where the user U inputs a new question sentence.
  • the evaluation unit 13 analyzes the similarity between the first question sentence and the new question sentence. For example, the evaluation unit 13 determines whether or not the first question sentence and the new question sentence are similar by a known method.
  • the evaluation unit 13 morphologically analyzes the first question sentence and the new question sentence, digitizes them into vectors, calculates the similarity between them, and determines that they are similar when the similarity is equal to or higher than a predetermined value.
  • the analysis of the similarity between the first question text and the new question text may be performed by any method. Then, when the evaluation unit 13 determines that the first question sentence and the new question sentence are similar, it can determine that the selected question candidate sentence for the first question sentence has a high probability of being an "wrong answer". A negative value is set for the correct answer rate, and that value is calculated as the correct answer rate of the question candidate sentence selected for the first question sentence.
  • the evaluation unit 13 determines that the first question sentence and the new question sentence do not resemble each other, it can determine that the selected question candidate sentence for the first question sentence has a high probability of being a "correct answer". A positive value is set for the correct answer rate, and that value is calculated as the correct answer rate of the question candidate sentence selected for the first question sentence.
  • the question candidate sentence or the like is displayed corresponding to the first question sentence from the user U, but the user U does not select anything for the question candidate sentence or the like.
  • the evaluation unit 13 analyzes the similarity between the first question sentence and the new question sentence in the same manner as described above. Then, when the evaluation unit 13 determines that the first question sentence and the new question sentence are similar, there is a probability that all the question candidate sentences displayed for the first question sentence are "wrong answers".
  • a question candidate sentence or the like is displayed corresponding to the first question sentence from the user U, and the user U selects one question candidate sentence for the question candidate sentence or the like. Then, suppose that the answer text corresponding to the selected question candidate text is displayed, and the answer text contains a link to another web page containing a detailed answer.
  • the evaluation unit 13 can determine that the selected question candidate sentence for the question sentence has a high probability of being a "correct answer", and the correct answer rate is set to "1". The value is calculated as the correct answer rate of the question candidate sentence selected for the question sentence.
  • the evaluation unit 13 can determine that the selected question candidate sentence for the question sentence has a high probability of being an "wrong answer", and the correct answer rate is "-0. 5 ”is set, and the value is calculated as the correct answer rate of the question candidate sentence selected for the question sentence.
  • a question candidate sentence or the like is displayed corresponding to the first question sentence from the user U, and the user U selects one question candidate sentence for the question candidate sentence or the like. Then, consider the case where the answer text corresponding to the selected question candidate text is displayed, and a selection button (a button for selecting "whether or not the answer was useful") indicating the degree of evaluation by the user for the answer is displayed. ..
  • the evaluation unit 13 can determine that the selected question candidate sentence for the question sentence has a high probability of being a "correct answer” when the button "the answer was useful" is selected by the user U.
  • the correct answer rate is set to "1", and the value is calculated as the correct answer rate of the question candidate sentence selected for the question sentence.
  • the evaluation unit 13 can determine that when the button "The answer was not useful" is selected by the user U, the selected question candidate sentence for the question sentence has a high probability of being "wrong answer”. , The correct answer rate is set to "-1", and the value is calculated as the correct answer rate of the question candidate sentence selected for the question sentence.
  • the evaluation unit 13 has a function of correcting the correct answer rate associated with the question candidate sentence for the question sentence.
  • the evaluation unit 13 calculates the similarity indicating the degree of similarity between the evaluation data, and corrects the correct answer rate included in each of the evaluation data according to the similarity.
  • the evaluation unit 13 first calculates the degree of similarity between the two evaluation data based on the question text and the question candidate text included in each evaluation data. For example, the evaluation unit 13 determines the similarity of the question candidate sentences according to whether or not the QAIDs of the Q & A data including the question candidate sentences are the same, and the question sentences are subjected to morphological analysis to perform morphological analysis between the question sentences. Calculate the similarity.
  • the evaluation unit 13 comprehensively judges the similarity of the question candidate sentences and the similarity of the question sentences, and when it is judged that the two evaluation data are similar to each other, the other correct answer is added to its own correct answer rate. By adding the rates, the correct answer rate is corrected. For example, as shown in FIG. 4B, when it is determined that the evaluation data corresponding to the question sentences by the user IDs A and E are similar to each other, the correct answer rate is added to each other for correction.
  • the above-mentioned calculation of the correct answer rate by the evaluation unit 13 is an example, and the correct answer rate is calculated according to the behavior of the user U by other criteria and methods, and the correct answer rate of the question candidate sentence for the question sentence is set. You may.
  • the above-mentioned evaluation data is stored in the evaluation data storage unit 16 and later used as learning data for machine learning to generate a model to be used when extracting a question candidate sentence from the question sentence. Then, the correct answer rate included in the evaluation data is used as a weight when learning the model.
  • the evaluation data described above is not necessarily limited to being used as learning data for machine learning, and may be used in any situation.
  • the chatbot 10 displays the chat screen A1 as shown in FIG. 5A on the user terminal 20 and accepts the question text input in the message input field A2 (step S1 in FIG. 6). ..
  • the chatbot 10 searches and extracts the question candidate sentence corresponding to the question sentence from the user U from the Q & A data storage unit 14 (step S2 in FIG. 6), and as shown in the chat screen A1 in FIG. 5B.
  • the question text from the user U is displayed in the question column U1, and the list of the extracted question candidate texts is displayed in the response field P1 of the chat screen A1 (step S3 in FIG. 6).
  • the chatbot 10 has a list of question candidate sentences (question candidate sentences 1, 2, 3) and options such as "more questions” and "not in this". It is displayed in the response column P1.
  • the chatbot 10 detects the behavior of the user U after displaying the question candidate sentence or the like in the response column P1 of the chat screen A1 (step S4 in FIG. 6). For example, in the chatbot 10, as the behavior of the user U, the question candidate sentence displayed in the response column P1, the selection status by the user U for each option, the operation by the user U after the answer sentence is displayed, and the user. The input of another question text into the further message input field A2 by U is detected.
  • the chatbot 10 outputs various displays to the question column U1 and the response column P1 of the chat screen A1 as shown in FIGS. 5C to 5G according to the behavior of the user U, and is a question candidate for the question text.
  • Calculate the correct answer rate of the sentence step S5 in FIG. 6).
  • the chatbot 10 refers to the preset correct answer rate table shown in FIG. 3 and determines the correct answer rate of the question candidate sentence with respect to the question sentence according to the behavior of the user U, as shown in FIG. 4A.
  • the question text and the question candidate text are associated with each other and stored as evaluation data (step S6 in FIG. 6).
  • the chatbot 10 later corrects the correct answer rate included in the evaluation data at an arbitrary timing. For example, if the similarity between the evaluation data, that is, the similarity between the question sentences and the question candidate sentences included in each evaluation data is calculated, and it is judged that the evaluation data are similar to each other, the correct answer rate of the other is added to the own correct answer rate. Make corrections by adding the correct answer rate.
  • the chatbot 10 can use the evaluation data later as learning data for generating a model to be used when extracting the question candidate sentence from the question sentence.
  • the chatbot 10 uses the correct answer rate included in the evaluation data as a weight when learning the model.
  • the chatbot 10 outputs a question candidate sentence corresponding to the question sentence input by the user to the user, and the question sentence corresponds to the behavior of the user with respect to the question candidate sentence. Evaluate the question candidate sentences for. Therefore, the chatbot 10 can detect the behavior of the user in the process from the user asking a question to obtaining an answer, and can obtain an evaluation of the question candidate text for the question text according to the behavior. .. As a result, the chatbot 10 can obtain an evaluation of the response to the question from the user U, and can improve the accuracy of the response to the question.
  • chatbot 10 in the above-described embodiment is described as having a configuration in which the chatbot 10 communicates with the user U using sentences
  • the chatbot in the present invention may communicate with the user by voice. That is, the chatbot may accept the question text from the user via voice, output the question candidate text to the user, and detect the behavior of the user.
  • FIGS. 7 to 9 are block diagrams showing the configuration of the information providing system according to the second embodiment, and FIGS. 7 to 8 are flowcharts showing the operation of the information providing system.
  • the outline of the configuration of the chatbot and the information providing method described in the first embodiment is shown.
  • the information providing system 100 is composed of a general information processing device, and is equipped with the following hardware configuration as an example.
  • -CPU Central Processing Unit
  • -ROM Read Only Memory
  • RAM Random Access Memory
  • 103 storage device
  • -Program group 104 loaded into RAM 303
  • a storage device 105 that stores the program group 304.
  • a drive device 106 that reads and writes the storage medium 110 external to the information processing device.
  • -Communication interface 107 that connects to the communication network 111 outside the information processing device -I / O interface 108 for inputting / outputting data -Bus 109 connecting each component
  • the information providing system 100 can construct and equip the question-and-answer unit 121, the detection unit 122, and the evaluation unit 123 shown in FIG. 8 by acquiring the program group 104 by the CPU 101 and executing the program group 104.
  • the program group 104 is stored in, for example, a storage device 105 or a ROM 102 in advance, and the CPU 101 loads the program group 104 into the RAM 103 and executes the program group 104 as needed. Further, the program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in the storage medium 110 in advance, and the drive device 106 may read the program and supply the program to the CPU 101.
  • the question-and-answer unit 121, the detection unit 122, and the evaluation unit 123 described above may be constructed by an electronic circuit.
  • FIG. 7 shows an example of the hardware configuration of the information processing device constituting the information providing system 100, and the hardware configuration is not limited to the above-mentioned case.
  • the information processing device may be configured from a part of the above-described configuration, such as not having the drive device 106.
  • the information providing system 100 executes the information providing method shown in the flowchart of FIG. 9 by the functions of the question answering unit 121, the detecting unit 122, and the evaluation unit 123 constructed by the program as described above.
  • the information providing system 100 is In response to the question text input by the user, the question candidate text corresponding to the question text is output to the user (step S101).
  • the user's behavior with respect to the question candidate sentence is detected (step S102),
  • the question candidate sentence is evaluated with respect to the question sentence according to the behavior (step S103).
  • the information providing system 100 outputs a question candidate sentence corresponding to the question sentence input by the user to the user, and the user's behavior with respect to the question candidate sentence.
  • the question candidate sentences are evaluated for the question sentences according to the above. Therefore, the information providing system 100 can detect the behavior of the user in the process from the user asking a question to obtaining an answer, and can obtain an evaluation of the question candidate text for the question text according to the behavior. it can. As a result, the information providing system 100 can obtain an evaluation of the response to the question from the user U, and can improve the accuracy of the response to the question.
  • Appendix 2 The information provision method described in Appendix 1.
  • the question answering unit outputs the question candidate sentence to the display screen of the information processing device operated by the user.
  • the detection unit detects an operation by the user on the display screen displaying the question candidate sentence.
  • the evaluation unit evaluates the question candidate sentence with respect to the question sentence according to the operation. Information provision method.
  • Appendix 3 The information providing method described in Appendix 1 or 2.
  • the detection unit detects the selection status of the question candidate sentence output to the user by the user.
  • the evaluation unit evaluates the question candidate sentence for the question sentence according to the user's selection status for the question candidate sentence. Information provision method.
  • the question-and-answer unit outputs the question candidate sentence to the user, and outputs other options for which the question candidate sentence is desired to be output to the user.
  • the detection unit detects the question candidate sentence and the user's selection status for the option as the behavior.
  • the evaluation unit evaluates the question candidate sentence for the question sentence according to the user's selection status for the question candidate sentence and the option. Information provision method.
  • Appendix 5 The information providing method described in either Appendix 1 or 4.
  • the detection unit detects that after the question candidate sentence is output to the user, another question sentence is further input from the user, and the detection unit detects that another question sentence is input from the user.
  • the evaluation unit evaluates the question candidate sentence with respect to the question sentence based on the question sentence and the other question sentence. Information provision method.
  • Appendix 6 The information provision method described in Appendix 5, The evaluation unit analyzes the similarity between the question text and the other question text, and evaluates the question candidate text with respect to the question text based on the analysis result. Information provision method.
  • the question-and-answer unit outputs an answer sentence corresponding to the question candidate sentence to the user according to the user's behavior with respect to the question candidate sentence.
  • the detection unit detects the user's second behavior with respect to the answer text,
  • the evaluation unit evaluates the question candidate sentence with respect to the question sentence according to the second behavior. Information provision method.
  • Appendix 9 The information provision method described in Appendix 8
  • the evaluation unit calculates a degree of similarity indicating the degree to which the data in which the question sentence and the question candidate sentence are paired are similar to each other, and determines the correct answer rate associated with each of the data according to the degree of similarity.
  • Information provision method To fix, Information provision method.
  • a question-and-answer section that outputs a question candidate sentence corresponding to the question sentence to the user according to the question sentence input from the user.
  • a detection unit that detects the user's behavior with respect to the question candidate sentence
  • An evaluation unit that evaluates the question candidate sentence for the question sentence according to the behavior, and an evaluation unit.
  • Information provision system equipped with.
  • the question answering unit outputs the question candidate sentence to the display screen of the information processing device operated by the user.
  • the detection unit detects an operation by the user on the display screen displaying the question candidate sentence.
  • the evaluation unit evaluates the question candidate sentence with respect to the question sentence according to the operation. Information provision system.
  • Appendix 12 The information providing system according to Appendix 10 or 11.
  • the detection unit detects the selection status of the question candidate sentence output to the user by the user.
  • the evaluation unit evaluates the question candidate sentence for the question sentence according to the user's selection status for the question candidate sentence. Information provision system.
  • the question-and-answer unit outputs the question candidate sentence to the user, and outputs other options for which the question candidate sentence is desired to be output to the user.
  • the detection unit detects the question candidate sentence and the user's selection status for the option as the behavior.
  • the evaluation unit evaluates the question candidate sentence for the question sentence according to the user's selection status for the question candidate sentence and the option. Information provision system.
  • Appendix 14 The information providing system according to any one of Appendix 10 or 13.
  • the detection unit detects that after the question candidate sentence is output to the user, another question sentence is further input from the user, and the detection unit detects that another question sentence is input from the user.
  • the evaluation unit evaluates the question candidate sentence with respect to the question sentence based on the question sentence and the other question sentence. Information provision system.
  • Appendix 15 The information providing system described in Appendix 14, The evaluation unit analyzes the similarity between the question text and the other question text, and evaluates the question candidate text with respect to the question text based on the analysis result. Information provision system.
  • the information providing system according to any one of Appendix 10 to 15.
  • the question-and-answer unit outputs an answer sentence corresponding to the question candidate sentence to the user according to the user's behavior with respect to the question candidate sentence.
  • the detection unit detects the user's second behavior with respect to the answer text,
  • the evaluation unit evaluates the question candidate sentence with respect to the question sentence according to the second behavior.
  • Information provision system
  • Appendix 17 The information providing system according to any one of Appendix 10 to 16.
  • the evaluation unit calculates a correct answer rate indicating the degree to which the question candidate sentence output corresponding to the question sentence is correct according to the behavior.
  • the correct answer rate is associated and stored in the data in which the question sentence and the question candidate sentence are paired. Information provision system.
  • Appendix 18 The information providing system described in Appendix 17, The evaluation unit calculates a degree of similarity indicating the degree to which the data in which the question sentence and the question candidate sentence are paired are similar to each other, and determines the correct answer rate associated with each of the data according to the degree of similarity. To fix, Information provision system.
  • a question-and-answer section that outputs a question candidate sentence corresponding to the question sentence to the user according to the question sentence input from the user.
  • a detection unit that detects the user's behavior with respect to the question candidate sentence,
  • An evaluation unit that evaluates the question candidate sentence for the question sentence according to the behavior, and an evaluation unit.
  • Non-temporary computer-readable media include various types of tangible storage media.
  • Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, It includes a CD-R / W and a semiconductor memory (for example, a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, and a RAM (RandomAccessMemory)).
  • a semiconductor memory for example, a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, and a RAM (RandomAccessMemory)
  • the program may also be supplied to the computer by various types of temporary computer readable medium.
  • temporary computer-readable media include electrical, optical, and electromagnetic waves.
  • the temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
PCT/JP2019/051143 2019-12-26 2019-12-26 情報提供方法 Ceased WO2021130964A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US17/637,175 US20220309942A1 (en) 2019-12-26 2019-12-26 Information providing method
JP2021566686A JP7131720B2 (ja) 2019-12-26 2019-12-26 情報提供方法
PCT/JP2019/051143 WO2021130964A1 (ja) 2019-12-26 2019-12-26 情報提供方法
JP2022131632A JP7343014B2 (ja) 2019-12-26 2022-08-22 情報提供方法
JP2023138146A JP7722420B2 (ja) 2019-12-26 2023-08-28 情報提供方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/051143 WO2021130964A1 (ja) 2019-12-26 2019-12-26 情報提供方法

Publications (1)

Publication Number Publication Date
WO2021130964A1 true WO2021130964A1 (ja) 2021-07-01

Family

ID=76575834

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/051143 Ceased WO2021130964A1 (ja) 2019-12-26 2019-12-26 情報提供方法

Country Status (3)

Country Link
US (1) US20220309942A1 (https=)
JP (3) JP7131720B2 (https=)
WO (1) WO2021130964A1 (https=)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023281672A1 (ja) * 2021-07-07 2023-01-12 日本電気株式会社 システム、サーバ装置、情報提供方法及び記憶媒体
JP2024064873A (ja) * 2022-10-28 2024-05-14 株式会社リコー 情報提供装置、情報提供システム、情報提供方法、及びプログラム
JP7788500B1 (ja) * 2024-07-03 2025-12-18 株式会社三菱Ｕｆｊ銀行 情報処理装置、方法、プログラム、および情報処理システム

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2024007230A (ja) * 2022-07-05 2024-01-18 富士通株式会社 問題集作成プログラム、装置、及び方法
WO2025143646A1 (ko) * 2023-12-26 2025-07-03 삼성전자 주식회사 전자 장치 및 이를 이용한 카테고리에 기반한 사용자 쿼리에 대한 사용자 인터페이스 제공 방법
JP7780166B1 (ja) * 2025-09-22 2025-12-04 匠技研工業株式会社 情報処理システム、情報処理方法、及びプログラム

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150339590A1 (en) * 2014-05-21 2015-11-26 Yahoo! Inc. Synthetic question formulation
JP2016045652A (ja) * 2014-08-21 2016-04-04 国立研究開発法人情報通信研究機構 質問文生成装置及びコンピュータプログラム
JP2017027233A (ja) * 2015-07-17 2017-02-02 日本電信電話株式会社 質問文生成装置、方法、及びプログラム

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7295965B2 (en) * 2001-06-29 2007-11-13 Honeywell International Inc. Method and apparatus for determining a measure of similarity between natural language sentences
US8666983B2 (en) * 2003-06-13 2014-03-04 Microsoft Corporation Architecture for generating responses to search engine queries
US8346701B2 (en) * 2009-01-23 2013-01-01 Microsoft Corporation Answer ranking in community question-answering sites
US20110004588A1 (en) * 2009-05-11 2011-01-06 iMedix Inc. Method for enhancing the performance of a medical search engine based on semantic analysis and user feedback
US20120116982A1 (en) * 2010-06-02 2012-05-10 Salesforce. com. Inc. Method and system for escalating content of discussions to particular memory locations
US20120130910A1 (en) * 2010-11-19 2012-05-24 Salesforce.Com, Inc. Customer support flow
US10691698B2 (en) * 2014-11-06 2020-06-23 International Business Machines Corporation Automatic near-real-time prediction, classification, and notification of events in natural language systems
US9996604B2 (en) * 2015-02-09 2018-06-12 International Business Machines Corporation Generating usage report in a question answering system based on question categorization
JP6649124B2 (ja) * 2015-05-25 2020-02-19 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ 機械翻訳方法、機械翻訳装置及びプログラム
US20170212872A1 (en) * 2016-01-22 2017-07-27 International Business Machines Corporation Duplicate post handling with natural language processing
JP6871799B2 (ja) * 2017-04-28 2021-05-12 キヤノンマーケティングジャパン株式会社 情報処理装置、情報処理システム、その制御方法及びプログラム
JP6968662B2 (ja) * 2017-10-31 2021-11-17 エヌ・ティ・ティ・コムウェア株式会社 情報提供システム、情報提供装置、情報分析装置、及び情報提供方法
US11409996B2 (en) * 2017-11-29 2022-08-09 International Business Machines Corporation Clustering subject matter experts based on question and answer ratings
EP3742305A4 (en) * 2018-01-15 2020-11-25 Fujitsu Limited OUTPUT CONTROL PROGRAM, OUTPUT CONTROL PROCESS AND OUTPUT CONTROL DEVICE
WO2019160152A1 (ja) * 2018-02-19 2019-08-22 Ａｒｉｔｈｍｅｒ株式会社 対話管理サーバ、対話管理方法、及びプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150339590A1 (en) * 2014-05-21 2015-11-26 Yahoo! Inc. Synthetic question formulation
JP2016045652A (ja) * 2014-08-21 2016-04-04 国立研究開発法人情報通信研究機構 質問文生成装置及びコンピュータプログラム
JP2017027233A (ja) * 2015-07-17 2017-02-02 日本電信電話株式会社 質問文生成装置、方法、及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
MATSUDA, TORU.: " An Expert System based on Dempster-Schafer Theory. ", RICOH TECHNICAL REPORT, no. 18, 25 October 1988 (1988-10-25), pages 32 - 38 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023281672A1 (ja) * 2021-07-07 2023-01-12 日本電気株式会社 システム、サーバ装置、情報提供方法及び記憶媒体
JPWO2023281672A1 (https=) * 2021-07-07 2023-01-12
JP7683958B2 (ja) 2021-07-07 2025-05-27 日本電気株式会社 システム、サーバ装置、情報提供方法及びプログラム
JP2024064873A (ja) * 2022-10-28 2024-05-14 株式会社リコー 情報提供装置、情報提供システム、情報提供方法、及びプログラム
JP7729311B2 (ja) 2022-10-28 2025-08-26 株式会社リコー 情報提供装置、情報提供システム、情報提供方法、及びプログラム
JP7788500B1 (ja) * 2024-07-03 2025-12-18 株式会社三菱Ｕｆｊ銀行 情報処理装置、方法、プログラム、および情報処理システム

Also Published As

Publication number Publication date
JP2022166258A (ja) 2022-11-01
JP7722420B2 (ja) 2025-08-13
JPWO2021130964A1 (https=) 2021-07-01
JP7343014B2 (ja) 2023-09-12
JP2023162332A (ja) 2023-11-08
US20220309942A1 (en) 2022-09-29
JP7131720B2 (ja) 2022-09-06

Similar Documents

Publication Publication Date Title
JP7343014B2 (ja) 情報提供方法
JP2017111190A (ja) 対話テキスト要約装置及び方法
US20170161370A1 (en) Control method, processing apparatus, and recording medium
JP6787973B2 (ja) 質問に対する回答を支援するためのシステム、方法、及びプログラム
JP2018195298A (ja) 対話シナリオの表示制御プログラム、対話シナリオの表示制御方法および情報処理装置
JP2019219737A (ja) 対話サーバ、対話方法及び対話プログラム
US20220261527A1 (en) Information processing apparatus and non-transitory computer readable medium
CN117971661A (zh) 大模型测试方法、装置、电子设备及存储介质
JP2017097488A (ja) 情報処理装置、情報処理方法および情報処理プログラム
JP6824547B1 (ja) アクティブラーニングシステム及びアクティブラーニングプログラム
JPWO2014045546A1 (ja) メンタルヘルスケア支援装置、システム、方法およびプログラム
JP6555757B2 (ja) 応対支援装置、方法、及びプログラム
US9684917B2 (en) Product interoperability determination
US11776546B1 (en) Intelligent agent for interactive service environments
US20150364051A1 (en) Generating a comprehension indicator that indicates how well an individual understood the subject matter covered by a test
Cranford et al. Generating predictive inferences when multiple alternatives are available
JP7081671B2 (ja) 評価プログラム、評価方法および情報処理装置
JP2022032278A (ja) 質問回答装置
JP2022062249A (ja) アンケート処理装置、アンケート処理方法、およびプログラム
JP7058142B2 (ja) アンケート処理装置、アンケート処理方法、およびプログラム
JP7724937B1 (ja) 情報処理装置、システム、及びシステムの動作方法
JP7359349B1 (ja) 評価支援システム、情報処理装置の制御方法、及び情報処理装置の制御プログラム
KR101580159B1 (ko) 설문 모집단 설정 방법, 이를 수행하는 설문 모집단 설정 서버 및 이를 저장하는 기록매체
JP7734115B2 (ja) 応対フロー作成支援装置、及び応対フロー作成支援方法
KR101674646B1 (ko) 이미지와 연관된 텍스트를 결정하기 위한 사용자 인증 서비스 제공 장치 및 그 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19957694

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021566686

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19957694

Country of ref document: EP

Kind code of ref document: A1