JP2024031780A - Program, method and system - Google Patents

Abstract

To provide a system capable of efficiently improving identification performances of a machine learning model.SOLUTION: A program causes a processor to execute the steps of: calculating, based on a question sentence input by a user from a user terminal and question patterns stored in a storage part, a first likelihood to be estimated from the question sentence and representing a degree of relation of each of a plurality of categories stored in the storage part and associated with answer patterns, with the question sentence; selecting a category corresponding to the question sentence based on the calculated first likelihood and an action log of the user; and displaying an answer pattern associated with the selected category on the user terminal.SELECTED DRAWING: Figure 8

Description

The present disclosure relates to a program, method, and system for executing processing as a response system using a text input interface.

2. Description of the Related Art Conventionally, automatic response systems have been known that automatically respond to inquiries based on text input.
For example, Patent Document 1 discloses an automatic response system that includes a machine learning model that is machine learned using a plurality of question patterns prepared for each category as learning data.
In this automatic response system, the question text input by the user is classified into a specific category by a machine learning model, and the answer pattern linked to the classified category is output to the user as an answer text. Ru.

Patent No. 6442807

In the system described in Patent Document 1, for example, the number of categories to be classified may increase due to long-term operation, and the classification processing performed by the machine learning model becomes more advanced, resulting in continuously high identification performance. was difficult to maintain.

Therefore, the present disclosure aims to provide a system that can efficiently improve the identification performance of a machine learning model.

A program according to an aspect of the present disclosure is stored in a storage unit and associated with an answer pattern in a processor based on a question text input by a user from a user terminal and a question pattern stored in the storage unit. For each of the plurality of categories, calculating a first likelihood indicating the degree of association with the question text estimated from the question text, and based on the calculated first likelihood and the user's behavior log, A step of selecting a category to which the sentence falls, and a step of displaying an answer pattern associated with the selected category on a user terminal are executed.

According to the present disclosure, it is possible to efficiently improve the identification performance of a machine learning model.

1 is a diagram illustrating an overview of an automatic response system of the present invention. FIG. 2 is a diagram showing a functional configuration of a terminal device that constitutes an automatic response system. FIG. 2 is a diagram showing a functional configuration of a question and answer server that constitutes an automatic response system. It is a diagram showing a specific example of a user database stored in a storage unit. It is a figure which shows the specific example of the question and answer database which a memory|storage part memorize|stores. FIG. 3 is a diagram illustrating a specific example of an operation log database stored in a storage unit. It is a diagram showing a specific example of a learning data database stored in a storage unit. FIG. 2 is a diagram illustrating the entire flow of processing by the automatic response system. FIG. 3 is a diagram illustrating the flow of category selection processing by the question and answer server. It is a figure explaining the flow of the process of selection of learning data by a question-and-answer server. It is a figure which shows the functional structure of the question-and-answer server of the automatic response system based on a modification.

Hereinafter, an automatic response system 1 according to an embodiment of the present disclosure will be described with reference to the drawings. Note that the embodiments described below do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all configurations described in the embodiments are essential configuration requirements of the present disclosure. In addition, in all the figures explaining the embodiments, common components are denoted by the same reference numerals, and repeated explanations will be omitted.

<1. Overview＞
FIG. 1 is a diagram illustrating an overview of an automatic response system 1 of the present invention.
As shown in FIG. 1, an automatic response system 1 according to the present embodiment is a system that executes processing as a response system using an interactive text input interface using a database in which question patterns and answer patterns are stored. .

Particularly, in the automatic response system 1, in addition to the question text from the user, the content (category and answer pattern) to be answered to the question text from the user is selected using the user's action log. The user behavior log specifically refers to the user's operation log.
For example, even for questions with similar question text, it is assumed that the answers sought by the user may differ depending on the operation (site visited) that the user performed immediately before asking the question. Therefore, the automatic response system 1 aims to improve the identification performance of the automatic response system 1 by using the user's operation log as input information.
In the following description, first, the configuration of the automatic response system 1 will be explained, and then response processing using the automatic response system 1 will be explained in detail.

<2. Overall configuration>
The overall configuration of the automatic response system 1 according to this embodiment will be described.
As shown in FIG. 1, the automatic response system 1 includes a plurality of questioners (in FIG. 1, user terminals 10A and 10B used by the questioners, hereinafter collectively referred to as "user terminals 10"), and a question answering server 20. and, including. The plurality of user terminals 10 and the question and answer server 20 are connected via a network 80 so that they can communicate with each other. The network 80 is configured by a wired or wireless network.

The user terminal 10 is a device operated by each user who is a questioner. In addition, the user terminal 10 may be realized by, for example, a tablet compatible with a mobile communication system or a mobile terminal such as a smartphone; the user terminal 10 may be a stationary PC (Personal Computer), a laptop PC, etc. Good too.

The user terminal 10 is communicably connected to the question and answer server 20 via the network 80. The user terminal 10 has a wireless base station 81 that is compatible with communication standards such as 5G and LTE (Long Term Evolution), and a wireless LAN (Local Area Network) that is compatible with wireless LAN (Local Area Network) standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11. It is connected to the network 80 by communicating with a communication device such as a wireless LAN router 82 .
As shown in FIG. 1, the user terminal 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage section 16, and a processor 19.

The communication IF 12 is an interface for inputting and outputting signals so that the user terminal 10 communicates with an external device.
The input device 13 is a device (for example, a keyboard, a touch panel, a touch pad, a pointing device such as a mouse, etc.) for receiving input operations from a user.

The output device 14 is a device (display, speaker, etc.) for presenting information to the user.
The memory 15 is for temporarily storing programs and data processed by the programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory).

The storage unit 16 is a storage device for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive).
The processor 19 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

The question-and-answer server 20 is a device that answers questions from questioners by referring to a question-and-answer database 2022, which will be described later.
The question and answer server 20 is a computer connected to the network 80. The question and answer server 20 includes a communication IF 22, an input device 23, an output device 24, a processor 25, a memory 26, and a storage 27.

The communication IF 22 is an interface for inputting and outputting signals so that the question and answer server 20 communicates with external devices.
The input device 23 is a device (for example, a keyboard, a touch panel, a touch pad, a pointing device such as a mouse, etc.) for receiving input operations from an administrator.
The output device 24 is a device (display, speaker, etc.) for presenting information to the administrator.

The processor 25 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.
The memory 26 is for temporarily storing programs and data processed by the programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory).
The storage 27 is a storage device for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive).

<3. Configuration of user terminal 10>
FIG. 2 is a block diagram showing the functional configuration of the user terminal 10 configuring the automatic response system 1. As shown in FIG. The user terminal 10 is assumed to be, for example, a terminal handled by a general user who wishes to make inquiries about unclear points in user support.

As shown in FIG. 2, the user terminal 10 includes a plurality of antennas (an antenna 111, an antenna 112), a wireless communication section (a first wireless communication section 121, a second wireless communication section 122) corresponding to each antenna, and an input terminal. It includes a reception section 130 (including a keyboard 131 and a display 132), an audio processing section 140, a microphone 141, a speaker 142, a camera 150, a storage section 160, and a control section 170.

The user terminal 10 also has functions and configurations that are not particularly illustrated in FIG. 2 (for example, a battery for holding power, a power supply circuit that controls the supply of power from the battery to each circuit, etc.). As shown in FIG. 2, each block included in the user terminal 10 is electrically connected by a bus or the like.

The antenna 111 radiates a signal emitted by the user terminal 10 as a radio wave. Further, the antenna 111 receives radio waves from space and provides a received signal to the first wireless communication unit 121.

Antenna 112 radiates signals emitted by user terminal 10 as radio waves. Further, the antenna 112 receives radio waves from space and provides a received signal to the second wireless communication unit 122.

The first wireless communication unit 121 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 111 so that the user terminal 10 communicates with other wireless devices. The second wireless communication unit 122 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 112 so that the user terminal 10 communicates with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high frequency circuit, and the like. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation/demodulation and frequency conversion of wireless signals transmitted and received by the user terminal 10 and provide received signals to the control unit 170.

The input receiving unit 130 has a mechanism for receiving input operations from the user. Specifically, input reception unit 130 includes a keyboard 131 and a display 132. Note that the input receiving unit 130 may be configured as a touch screen that detects the position of the user's touch on the touch panel by using, for example, a capacitive touch panel.

The keyboard 131 receives input operations from the user of the user terminal 10 . The keyboard 131 is a device for inputting characters, and outputs input character information to the control unit 170 as an input signal.

The display 132 displays data such as images, videos, and text under the control of the control unit 170. That is, the display 132 also functions as an output unit that outputs information to the user. The display 132 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

The audio processing unit 140 modulates and demodulates the audio signal. The audio processing unit 140 modulates the signal provided from the microphone 141 and provides the modulated signal to the control unit 170. The audio processing unit 140 also provides an audio signal to the speaker 142.
The audio processing unit 140 is realized, for example, by a processor for audio processing. The microphone 141 receives a voice input and provides a voice signal corresponding to the voice input to the voice processing unit 140 . The speaker 142 converts the audio signal provided from the audio processing unit 140 into audio and outputs the audio to the outside of the user terminal 10 .

The camera 150 is a device that receives light with a light receiving element and outputs it as a photographed image. The camera 150 is, for example, a depth camera that can detect the distance from the camera 150 to the object to be photographed.

The storage unit 160 is configured with, for example, a flash memory, and stores data and programs used by the user terminal 10. In one aspect, storage unit 160 stores user information 161.

The user information 161 is information about a user who uses the automatic response system 1 using the user terminal 10. The user information 161 includes information that identifies the user (user ID), the user's name, information on the company to which the user belongs, account information for the message app used by the user, and contact information such as the user's email address. Contains information, etc.

The control unit 170 controls the operation of the user terminal 10 by reading a program stored in the storage unit 160 and executing instructions included in the program. The control unit 170 is, for example, an application program installed in the user terminal 10 in advance. The control unit 170 functions as an operation reception unit 171, a transmission/reception unit 172, a data processing unit 173, and a display processing unit 174 by operating according to a program.

The operation reception unit 171 performs a process of accepting a user's input operation on an input device such as the keyboard 131.

The transmitting/receiving unit 172 performs processing for the user terminal 10 to transmit and receive data with an external device such as the question and answer server 20 according to a communication protocol.

The data processing unit 173 performs calculations according to a program on the data input by the user terminal 10, and outputs the calculation results to a memory or the like.

The display processing unit 174 performs a process of presenting information to the user. The display processing unit 174 performs processes such as displaying a display image on the display 132, outputting audio to the speaker 142, and generating vibrations in the camera 150.

<4. Functional configuration of question and answer server 20>
FIG. 3 is a diagram showing the functional configuration of the question and answer server 20 that constitutes the automatic response system 1. As shown in FIG. The question and answer server 20 is assumed to be, for example, a terminal handled by a support manager who handles inquiries in user support. As shown in FIG. 3, the question and answer server 20 functions as a communication section 201, a storage section 202, and a control section 203.

The communication unit 201 performs processing for the question and answer server 20 to communicate with an external device.

The storage unit 202 stores data and programs used by the question and answer server 20. The storage unit 202 stores, for example, the following data.
・User database 2021 (User DB2021)
・Question and answer database 2022 (Question and answer DB2022)
・Operation log database 2023 (operation log DB 2023)
・Learning data database 2024 (Learning data DB2024)
・Question classification model 2025
・Behavior classification model 2026

<4-1. Database structure>
Each database will be explained in detail. Note that details of the question classification model 2025 and the behavior classification model 2026 will be described later.

<4-1-1. User DB2021>
User DB 2021 is a database that stores information regarding users. FIG. 4 is a diagram showing a specific example of the user DB 2021 stored in the storage unit 202.
As shown in FIG. 4, the user DB 2021 includes the item "user ID", the item "name", the item "personal information", the item "member information", and the item "terminal information". For example, a new record is recorded in the user DB 2021 when a user registers as a member.

The item "user ID" stores user identification information that can identify individual users.

The item "Name" stores information on the name of the user corresponding to the user ID.

The item "personal information" stores attribute information regarding the individual user corresponding to the user ID. Attribute information regarding individuals includes, for example, gender, age, place of residence, nationality, occupation, family structure, place of birth, and native language.

The item "member information" stores attribute information of the user corresponding to the user ID as a member who uses the automatic response system 1. Attribute information as a member includes, for example, membership type, usage history, contract details, and the like.

The item "terminal information" stores attribute information regarding the user terminal 10 used by the user corresponding to the user ID. The attribute information as a terminal includes terminal identification information for identifying a tablet or a smartphone.
Note that the structure of the user DB 2021 shown in FIG. 4 is merely an example, and other items may be managed. Furthermore, the storage unit 202 does not need to store the user DB 2021.

<4-1-2. Question and answer DB2022>
The question and answer DB 2022 shown in FIG. 3 is a database that stores question patterns and answer patterns in association with each other. FIG. 5 is a diagram showing a specific example of the question and answer DB 2022 stored in the storage unit 202.
As shown in FIG. 5, the question and answer DB 2022 associates a title with one answer pattern for one category, and stores one or more question patterns assumed for one category. ing. That is, answer patterns and question patterns are linked to categories. The question and answer DB 2022 is stored in the storage unit 202 in advance, for example, before the automatic response system 1 starts operating.

The question and answer DB 2022 includes an item "category," an item "answer pattern," and an item "question pattern."
The item "Category" stores a title indicating the category of the question. The category title is, for example, information that clearly indicates the content of the question.

Furthermore, it is not necessary that there is only one question pattern for one answer pattern. For example, when making a purchase on an EC (Electronic Commerce) site, if a user wants to know how to change his or her address, he or she may receive various messages such as ``I want to change my address.'' or ``I entered the address incorrectly.'' Since there are many ways to ask questions, it is useful to register multiple question patterns for one category (title: How to change your address).
Note that the question and answer DB 2022 may have a structure in which category titles are managed hierarchically.

<4-1-3. Operation log DB2023>
The operation log DB 2023 shown in FIG. 3 is a database that stores user operation logs. FIG. 6 is a diagram showing a specific example of the operation log DB 2023 stored in the storage unit 202.
As shown in FIG. 6, the operation log DB 2023 includes the item "operation log ID", the item "user ID", the item "operation information", the item "operation target", and the item "operation date and time". ing. The operation log DB 2023 records a new record every time an operation from a user is accepted.

The item "operation log ID" stores identification information for identifying the user's operation log acquired by the question and answer server 20.

The item "user ID" stores the user ID of the user who performed the operation corresponding to the operation log ID.

The item “operation information” stores operation information corresponding to the operation log ID. The operation information is time-series information, and describes, for example, a login operation, a page transition operation, a page stay operation, a click-based product selection operation, and the like.

The item "operation target" stores the URL of the site where the operation corresponding to the operation log ID was performed.

The item “operation date and time” stores the date and time of the operation corresponding to the operation log ID.
Note that the structure of the operation log DB 2023 shown in FIG. 6 is merely an example, and other items may be managed.

<4-1-4. Learning data DB2024>
The learning data DB 2024 shown in FIG. 3 is a database that stores learning data for learning user behavior characteristics. FIG. 7 is a diagram showing a specific example of the learning data DB 2024 stored in the storage unit 202.
As shown in FIG. 7, the learning data DB 2024 includes the item "data ID", the item "operation log ID", the item "answer category", and the item "evaluation score".

The item "data ID" stores identification information as learning data.

The item "operation log ID" stores a series of operation logs related to past user inquiries as learning data corresponding to the data ID. The series of operation logs related to inquiries include the input of a question into the automatic response system 1 and the operation log performed a predetermined time before that input.

The item "answer category" stores information regarding the category to which the answer text answered by the automatic response system 1 corresponds in a series of operation logs related to the inquiry corresponding to the data ID.

The item "evaluation score" stores information regarding the score of the questionnaire result input by the user afterward in response to the response text answered by the automatic response system 1 in a series of operation logs related to the question corresponding to the data ID. Ru.
Note that the structure of the learning data DB 2024 shown in FIG. 7 is merely an example, and other items may be managed.

<4-2. Control unit functions>
The control unit 203 shown in FIG. 3 functions as various functional modules when the processor 29 of the question and answer server 20 performs processing according to a program.
Various functional modules include a transmission/reception control module 2031, a question text acquisition module 2032, a first likelihood calculation module 2033, a second likelihood calculation module 2034, a category selection module 2035, an answer output module 2036, a questionnaire acquisition module 2037, and Includes a learning data selection module 2038.

The transmission/reception control module 2031 controls processing in which the question and answer server 20 transmits and receives signals from an external device according to a communication protocol.

Further, the question text acquisition module 2032 acquires text data related to a question input by the user into the user terminal 10. By allowing questions to be asked in natural text format, users can easily ask questions as if they were asking a person.

The first likelihood calculation module 2033 calculates the first likelihood for each of the plurality of categories stored in the question and answer DB 2022 from text data related to the question. The first likelihood is an index indicating the degree of association with the question text estimated from the question text. In other words, the first likelihood is an index indicating the likelihood that the question text falls under a certain category, which is estimated from the question text.
The first likelihood calculation module 2033 calculates the first likelihood of each of the plurality of categories by inputting the question text to the question classification model 2025.

The question classification model 2025 is obtained by having a machine learning model perform machine learning in accordance with a model learning program based on the learning data. In this embodiment, the question classification model 2025 is trained to, for example, output a first likelihood for each category in response to input question sentences. At this time, the learning data uses a plurality of question patterns as input data, and uses categories to which each of the plurality of question patterns corresponds as correct answer output data.

The question classification model 2025 is, for example, a composite function with parameters that is a composite of multiple functions. A parameterized composite function is defined by a combination of multiple tunable functions and parameters. The question classification model 2025 may be any parameterized composite function that satisfies the above requirements, but is assumed to be a multilayer network model (hereinafter referred to as a multilayer network). The question classification model 2025 using a multilayer network includes an input layer for inputting questions, an output layer for outputting the first likelihood, and at least one intermediate layer or hidden layer provided between the input layer and the output layer. and has. The question classification model 2025 is assumed to be used as a program module that is part of artificial intelligence software.

As the multilayer network according to this embodiment, for example, a deep neural network (DNN), which is a multilayer neural network that is a target of deep learning, can be used. As the DNN, for example, a recurrent neural network (RNN) that targets time-series information or the like may be used. Alternatively, a network with a self-attention mechanism that targets natural language may be used.

In calculating the first likelihood, the question classification model 2025 compares the question sentence with the question pattern stored in the question and answer DB 2022, and calculates the first likelihood based on the similarity.
The degree of similarity can be calculated by, for example, natural language processing. Specifically, it is conceivable to extract keywords from the question pattern and the question text, respectively, and use the degree of matching of the keywords as the degree of similarity using a weight for each keyword. Alternatively, it is also possible to vectorize each question pattern and question text and calculate the similarity using cosine similarity. In this way, by calculating which question pattern group a question text is similar to using similarity, it is possible to select an answer pattern even if it is not the same as the question pattern stored in the question and answer DB 2022. Become.

The second likelihood calculation module 2034 calculates the second likelihood for each of the plurality of categories stored in the question and answer DB 2022 based on the action log (operation log). The second likelihood is an index indicating the degree of association between the operation log estimated from the action log and the category. In other words, the second likelihood is an index indicating the likelihood that a question from a user who takes the behavior log falls under a certain category, which is estimated from the user's behavior log.

The second likelihood calculation module 2034 calculates the second likelihood of each of the plurality of categories by inputting the behavior log to the behavior classification model 2026. As the action log, at least any information of each item stored in the operation log DB 2023 can be used. For example, information on the website page that was opened at the time when the question was entered or immediately before that time can be used as an action log. Information about the page of the website that is open at the time of inputting the question text or immediately before that time can be obtained from the item "operation target" in the operation log DB 2023.

The behavior classification model 2026 is obtained by having a machine learning model perform machine learning in accordance with a model learning program based on the learning data. In this embodiment, the behavior classification model 2026 is trained to output the second likelihood for each category, for example, in response to input of behavior logs. At this time, the learning data uses action logs of a plurality of users as input data, and uses categories corresponding to each action log as correct answer output data. A category corresponding to an action log refers to a category that is considered to be an appropriate answer to a question asked in the action log.

The behavior classification model 2026 is, for example, a composite function with parameters that is a composite of multiple functions. A parameterized composite function is defined by a combination of multiple tunable functions and parameters. The behavior classification model 2026 may be any parameterized composite function that satisfies the above requirements, but is assumed to be a multilayer network model (hereinafter referred to as a multilayer network). The behavior classification model 2026 using a multilayer network includes an input layer for inputting operation logs, an output layer for outputting the second likelihood, and at least one intermediate layer or hidden layer provided between the input layer and the output layer. It has a layer. The behavior classification model 2026 is assumed to be used as a program module that is part of artificial intelligence software.

As the multilayer network according to this embodiment, for example, a deep neural network (DNN), which is a multilayer neural network that is a target of deep learning, can be used. As the DNN, for example, a recurrent neural network (RNN) that targets time-series information or the like may be used. Alternatively, a network with a self-attention mechanism that targets natural language may be used.

A behavior classification model 2026 can be created using the operation logs stored in the learning data DB 2024. In this case, in calculating the second likelihood, the behavior classification model 2026 compares the input user's behavior log (operation log) with the operation log stored in the learning data DB 2024, and determines the similarity. Based on this, a second likelihood is calculated.

In addition, in calculating the second likelihood, in addition to the method of using the behavior classification model 2026 as a machine learning model that outputs the second likelihood described above, the behavior classification model 2026 uses statistical information based on past response results. Alternatively, a method of calculating the second likelihood may be adopted.
Specifically, for example, when using information about a page of a website that is open at the time of inputting a question or immediately before that as a behavior log, the behavior classification model 2026 uses information about the user who is opening the page. For questions asked in the past, a probability distribution for each category of subsequent answers can be calculated and used as a second likelihood for each category corresponding to the page of the website.

Moreover, the behavior classification model 2026 may adopt a method of calculating the second likelihood by statistically processing the learning data accumulated in the learning data DB 2024.
In this case, for example, when using the information of the page of the website that was open at the time when the question was entered or immediately before that as the behavior log, the user who opened the page may be asked to , for the percentage of positive evaluations obtained in the post-answer questionnaire, a probability distribution is calculated using each answered category as a variable, and it is also used as the second likelihood for each category corresponding to the website page. good.

The category selection module 2035 selects a category to which the question text falls based on the first likelihood and the user's behavior log. That is, the question-and-answer server 20 selects the category to be answered using the user's behavior log, which is information with a different nature from the question text.

When selecting a category, the category selection module 2035 selects a category based on the first likelihood calculated from the question classification model 2025 and the behavior log without making any changes to the question classification model 2025. That is, the category selection module 2035 separately evaluates the category to be answered estimated from the question text and the category to be answered estimated from the user's behavior log, and then selects the respective evaluation results. The categories that should be answered for the question text are finally selected. According to this judgment method, the question classification model 2025 that estimates the category to be answered from the question text that has been conventionally adopted can be followed without changing.

The category selection module 2035 selects a category to which the question text falls based on the first likelihood and the second likelihood. Specifically, the category selection module 2035 calculates an evaluation index by multiplying the first likelihood by the second likelihood, and selects a category based on the calculated evaluation index. Note that the category selection module 2035 may perform any preset normalization on the value obtained by multiplying the value of the first likelihood and the value of the second likelihood and use it as the value of the evaluation index. .
The category selection module 2035 may select the top n categories based on the evaluation index.

The answer output module 2036 outputs the answer pattern linked to the selected category to the user terminal 10. The answer output module 2036 refers to the question and answer DB 2022, identifies the answer pattern associated with the category selected by the category selection module 2035, and outputs the text data of the answer pattern to the user terminal 10 as an answer to the question text. Output for.
The answer output module 2036 may output the top n answer sentences based on the evaluation index to the user terminal 10 along with the categories, and allow the user to select the most appropriate answer.

In addition, if the evaluation index of the selected category does not exceed a preset threshold, the answer output module 2036 estimates that the expression of the question is unclear, and does not display the answer pattern. It is also possible to display a message prompting another question, such as "Please change the question text." This provides the user with an opportunity to correct the question, and also avoids displaying an answer pattern different from the intended answer to the question, thereby avoiding unnecessary confusion.

The questionnaire acquisition module 2037 outputs a questionnaire to the user terminal 10. The questionnaire is about evaluating the validity of the answers output to the user. For example, the following questions are set in the questionnaire.
・Whether the problem faced by the user was resolved ・Whether the categories and answers answered were appropriate The response format to the questionnaire may be an alternative response format such as Yes/No, or a 5-point evaluation format. The answer may be in a quantitative answer format.

The learning data selection module 2038 selects learning data for the behavior classification model 2026 to perform machine learning. The learning data selection module 2038 uses a series of behavior logs related to question text input from past users and evaluation results from users regarding the categories selected for the question text. Select the corresponding answer categories as training data.
In other words, the learning data selection module 2038 selects as learning data the categories to which answers that received high ratings from users are associated, and the series of action logs that led to the question at that time. Store it in DB2024.

<5. Processing flow>
Next, the processing of the automatic response system 1 will be explained. The automatic response system 1 executes processing as a response system using an interactive text input interface. That is, the automatic response system 1 plays the role of a chatbot on the user support inquiry screen using an interactive text input interface.

An example of a response system using an interactive text input interface is a chatbot that pops up at the bottom right of an inquiry screen on an EC site.
In a response system using an interactive text input interface, first, in preparation for system settings, a conversation card to be used as a response system using an interactive text input interface is registered. The conversation card includes a guide message for prompting the user to input a question text. Thereby, preparations are made to provide message guidance to the user who has accessed the inquiry screen.

FIG. 8 is a diagram illustrating the entire flow of processing by the automatic response system 1. As shown in FIG.
As shown in FIG. 8, in the process of the automatic response system 1, first, the user inputs the content of the question in text using the keyboard 131 of the user terminal 10 (step S101). The control unit 170 of the user terminal 10 transmits the input text information to the question-and-answer server 20 via the transmission/reception unit 172 according to the processing of the data processing unit 173.

After step S101, the question and answer server 20 acquires the question text (step S201).
Specifically, the question text acquisition module 2032 of the question answering server 20 acquires text information regarding the question transmitted from the user terminal 10.

After step S201, the question and answer server 20 calculates the first likelihood in each category (step S202).
Specifically, the first likelihood calculation module 2033 of the question and answer server 20 inputs text information related to the question into the question classification model 2025, thereby calculating the first likelihood in each of the plurality of categories stored in the question and answer DB 2022. Calculate the likelihood.

After step S202, the question and answer server 20 selects a category (step S203). FIG. 9 is a diagram illustrating the flow of category selection processing by the question-and-answer server 20.
As shown in FIG. 9, in the category selection process, the question and answer server 20 calculates the second likelihood for each category (step S2031).
Specifically, the second likelihood calculation module 2034 of the question and answer server 20 calculates the second likelihood by inputting the user's behavior log to the behavior classification model 2026.

After step S2031, the question and answer server 20 calculates an evaluation index using the first likelihood and the second likelihood (step S2032).
Specifically, the category selection module 2035 of the question and answer server 20 calculates the evaluation index for each category by multiplying the value of the first likelihood and the value of the second likelihood. Here, the category selection module 2035 performs preset arbitrary normalization on the value obtained by multiplying the value of the first likelihood and the value of the second likelihood, and also uses it as an evaluation index value. good.
Note that the method of calculating the evaluation index is not limited to this example. For example, the evaluation index may be calculated by addition or multiplication after weighting at least one of the first likelihood and the second likelihood. Further, a function for calculating an evaluation index using the first likelihood and the second likelihood as variables may be set in advance.

After step S2032, the question and answer server 20 selects a category based on the calculated evaluation index (step S2033).
Specifically, the category selection module 2035 of the question and answer server 20 selects the category with the highest evaluation index from among the plurality of categories set in the question and answer DB 2022. At this time, the category selection module 2035 may select the top n categories based on the evaluation index.
This completes the category selection process (step S203) shown in FIG.

As shown in FIG. 8, after step S203, the question and answer server 20 outputs an answer (step S204).
Specifically, the answer output module 2036 of the question and answer server 20 refers to the question and answer DB 2022, identifies the answer pattern associated with the category selected by the category selection module 2035, and outputs it to the user terminal 10 as text data. Answer.

After step S204, the user terminal 10 displays the received answer (step S102).
Specifically, the control unit 170 of the user terminal 10 outputs text information regarding the answer to the display 132. As a result, the answer from the question answering server 20 to the input question is presented to the user.

After step S102, the question and answer server 20 outputs a questionnaire (step S205).
Specifically, the questionnaire acquisition module 2037 of the question-and-answer server 20 transmits a questionnaire to the user terminal 10 that accepts input of evaluation regarding the validity of the answers.

After step S205, the user terminal 10 displays the received questionnaire (step S103).
Specifically, the control unit 170 of the user terminal 10 outputs a questionnaire regarding the validity of the answers to the display 132. As a result, a questionnaire regarding the validity of the answers is presented to the user.

After step S103, the user terminal 10 receives responses to the questionnaire (step S104).
Specifically, the control unit 170 of the user terminal 10 receives input of answers to the questionnaire from the user, and transmits them to the question-and-answer server 20 via the transmission/reception unit 172 according to processing by the data processing unit 173.

After step S104, the question and answer server 20 obtains responses to the questionnaire (step S206).
Specifically, the transmission/reception control module 2031 of the question and answer server 20 acquires information regarding responses to the questionnaire transmitted from the user terminal 10.
With the above steps, the automatic response processing using the interactive text input interface is completed.

Next, the process of selecting learning data by the question and answer server 20 will be explained. FIG. 10 is a diagram illustrating a flow of processing for selecting learning data by the question-and-answer server 20.
A learning data selection module 2038 in the question and answer server 20 receives an operation from the management and selects learning data to be learned by the behavior classification model 2026. The administrator prepares data to be retrained by the behavior classification model 2026 in order to maintain and improve the discrimination performance (validity of the second likelihood) of the behavior classification model 2026.

Specifically, in order to select training data for retraining the behavior classification model 2026, the administrator asks questions and answers about instructions regarding the selection of training data from the accumulated operation logs and evaluation results for the answers. input to the server 20.
Note that the training data for the behavior classification model 2026 selected by the question and answer server 20 is not limited to use for relearning, and may be used for machine learning of the behavior classification model 2026 that is newly generated thereafter. .

As shown in FIG. 10, in the learning data selection process, the question answering server 20 extracts an action log related to the question (step S211).
Specifically, the learning data selection module 2038 of the question answering server 20 refers to the operation log DB 2023 and selects, from among the accumulated operation logs, operation logs related to the operation of asking a question using the automatic response system 1; Extract the operation log related to the previous operation.

Here, the immediately preceding operation log refers to an operation log regarding operations performed within a predetermined time period from the time when the operation related to the question was performed, based on the operation date and time. In this description, the operation log related to the operation of asking a question using the automatic response system 1 and the operation log related to the previous operation are collectively referred to as a series of operation logs related to inputting the question text.

For example, it is assumed that a user who has used the automatic response system 1 to inquire about login (password reissue) has attempted to log in immediately beforehand, but an error occurs. Therefore, there may be a certain relationship between the content of the question and the operation immediately before the question. Therefore, the learning data selection module 2038 extracts the operation log related to the input of the question text and the operation log related to the operation immediately before that as a series of operation logs related to the input of the question text.

After step S211, the question-and-answer server 20 extracts the results of the questionnaire in response to the extracted series of operation logs (step S212).
Specifically, the learning data selection module 2038 of the question and answer server 20 extracts the following information.
- The question text input in the question based on the extracted operation - The category selected by the question answering server 20 and the answer text outputted for the question text - The results of a questionnaire from users regarding the answers from the question answering server 20

After step S212, the question and answer server 20 identifies operation logs related to answers that received positive evaluations (step S213).
Specifically, the learning data selection module 2038 of the question and answer server 20 refers to the extracted questionnaire results and specifies the following information.
・Operation logs related to questions entered to obtain answers that received positive evaluations from users ・Contents of answers that received positive evaluations (answer text and categories)

As a result, the learning data selection module 2038 identifies the question for which the user has obtained the desired answer among the questions for which some operation was performed immediately before asking the question, and Operation logs can be identified. In other words, the learning data selection module 2038 can identify answer contents (categories, answer sentences) that are highly relevant to a series of operation logs related to inputting question sentences.

After step S213, the question and answer server 20 selects the specified operation log and the corresponding category (step S214).
Specifically, the learning data selection module 2038 of the question answering server 20 selects the identified operation log and the category selected for the question related to the operation log as the learning data. The learning data selection module 2038 records the operation log ID related to the selected operation log in the item "operation log ID" in the learning data DB 2024, and records the selected category in the item "answer category" in the learning data DB 2024. . At this time, a new data ID is assigned and a new record in the learning data DB 2024 is recorded.

The learning data selection module 2038 may extract the value of the questionnaire result regarding the answer to the question related to the operation log as an evaluation score, and record it in the learning data DB 2024. By referring to the evaluation score, you can understand the degree of relationship between the operation log included in the training data and the category to which the operation log should be answered, and select the training data that should be prioritized for learning. Can be sorted.
With the above, the learning data selection process by the question and answer server 20 is completed.

<6. Summary＞
As described above, in the automatic response system 1, the question and answer server 20 selects the category to which the question corresponds based on the first likelihood calculated from the question and the user's action log. That is, the automatic response system 1 selects a category to which a question should be answered, not only by comparing the question with a preset question pattern as in the past, but also by considering the user's behavior log. Therefore, in addition to the content of the question text, categories can be selected taking into consideration the user's behavioral tendencies, and the identification performance of the machine learning model can be efficiently improved.

Further, in the automatic response system 1, the question answering server 20 calculates the first likelihood for each of the plurality of categories by inputting the question text into the question classification model 2025. The question classification model 2025 is generated by machine learning using learning data in which a plurality of question patterns are input data, and a plurality of categories to which answer patterns corresponding to each of the plurality of question patterns are linked are correct answer output data. has been created. Therefore, the first likelihood can be calculated with high accuracy.

In addition, in the automatic response system 1, the question and answer server 20 selects a category based on the first likelihood calculated from the question classification model 2025 and the behavior log without making any changes to the question classification model 2025. do. Therefore, by extending the functionality of the response system using the existing question classification model 2025, the functions of the automatic response system 1 can be realized, and the automatic response system 1 can be smoothly introduced.

Furthermore, in the automatic response system 1, the question classification model 2025 calculates the first likelihood based on the degree of similarity between the question text and the question pattern. Therefore, the first likelihood can be quantitatively calculated using the similarity value.

Furthermore, in the automatic response system 1, the question and answer server 20 calculates the second likelihood for each of the plurality of categories, and selects the category to which the question sentence falls based on the first likelihood and the second likelihood. . Therefore, by using the second likelihood evaluated from the action log independent of the first likelihood, it is possible to select an appropriate category to which the question corresponds through multifaceted evaluation.

Furthermore, in the automatic response system 1, the question and answer server 20 calculates the second likelihood for each of the plurality of categories by inputting the behavior log into the behavior classification model 2026.
The behavior classification model 2026 uses learning data in which the behavior logs of multiple users are input data, and the correct answer output data is multiple categories to which answer patterns corresponding to each of the behavior logs of the multiple users are linked. It was created using machine learning. Therefore, the second likelihood can be calculated with high accuracy.

In addition, in the automatic response system 1, the question and answer server 20 uses a series of action logs related to input of questions from past users and evaluation results from users regarding answers to the questions, to determine a series of actions. Select answer categories corresponding to the logs as learning data.
Therefore, the behavior classification model 2026 can be retrained using the operation logs and questionnaire results accumulated through the user's use of the automatic response system 1, and the identification performance of the behavior classification model 2026 can be efficiently improved. Can be done.

Further, in the automatic response system 1, the question answering server 20 selects a category based on an evaluation index obtained by multiplying the first likelihood by the second likelihood. Therefore, the evaluation index can be calculated quantitatively.

<7. Modified example>
The automatic response system 1 may perform other processing.
For example, in the process of selecting a category, the category selection module 2035 may select a category based on the user's attributes in addition to the first likelihood and the user's behavior log. The configuration and processing of the automatic response system 1 according to such a modification will be described in detail. FIG. 11 is a diagram showing a functional configuration of the question answering server 20B of the automatic response system 1 according to a modification.

As shown in FIG. 11, the control unit 203 of the question and answer server 20B further includes a third likelihood calculation module 2039. Furthermore, the storage unit 202 of the question and answer server 20B further includes an attribute classification model 2027.

The third likelihood calculation module 2039 calculates the third likelihood for each of the plurality of categories stored in the question and answer DB 2022 based on the attributes of the user who asked the question. The third likelihood is an index indicating the degree of association between the question asked by the user and the category to which the question falls, which is estimated from the attributes of the user who asked the question. In other words, the third likelihood is an index indicating the likelihood that the question asked by the user falls under a certain category, which is estimated from the user's attributes.

The third likelihood calculation module 2039 calculates the third likelihood of each of the plurality of categories by inputting the user attributes of the user who asked the question to the attribute classification model 2027. As the user attribute, at least any one of personal information, member information, and terminal information stored in the user DB 2021 can be used.

The attribute classification model 2027 is obtained by having a machine learning model perform machine learning in accordance with a model learning program based on the learning data. In this embodiment, the attribute classification model 2027 is trained to output the third likelihood for each category, for example, in response to user attribute input. At this time, the learning data uses a plurality of user attributes as input data, and uses a category to which a question asked by a user of each user attribute corresponds as correct answer output data.

The attribute classification model 2027 is, for example, a composite function with parameters that is a composite of multiple functions. A parameterized composite function is defined by a combination of multiple tunable functions and parameters. The attribute classification model 2027 may be any parameterized composite function that satisfies the above requirements, but is assumed to be a multilayer network model (hereinafter referred to as a multilayer network). The attribute classification model 2027 using a multilayer network includes an input layer for inputting user attributes, an output layer for outputting the third likelihood, and at least one intermediate layer or hidden layer provided between the input layer and the output layer. It has a layer. The attribute classification model 2027 is assumed to be used as a program module that is part of artificial intelligence software.

As the multilayer network according to this embodiment, for example, a deep neural network (DNN), which is a multilayer neural network that is a target of deep learning, can be used. As the DNN, for example, a recurrent neural network (RNN) that targets time-series information or the like may be used. Alternatively, a network with a self-attention mechanism that targets natural language may be used.

In calculating the third likelihood, the attribute classification model 2027 can statistically calculate the third likelihood from the results of past answer results.
Specifically, the attribute classification model 2027 calculates the user attributes of users who have asked questions in the past and probability distributions for each of the answered categories, and calculates the third likelihood and the third likelihood for each category corresponding to the user attributes. can do.
In addition, the attribute classification model 2027 calculates a probability distribution with each answered category as a variable for the percentage of questions asked by the user in the past that received a positive evaluation in the questionnaire after answering, It may also be a third likelihood for each category corresponding to the user attribute.

The third likelihood calculation module 2039 calculates the third likelihood at least before step S2032 shown in FIG. Then, in step S2032 shown in FIG. 9, the category selection module 2035 calculates the evaluation index of the category using the third likelihood in addition to the first likelihood and the second likelihood. In step S2033 shown in FIG. 9, the category selection module 2035 selects a category based on the evaluation index that also uses the third likelihood.

In the question-and-answer server 20 according to the modified example, the learning data selection module 2038 selects the learning data to be trained on the behavior classification model 2026, and also selects the learning data to be trained on the attribute classification model 2027. You may also perform a process of selecting.
In this case, the learning data selection module 2038 may select as learning data a combination of the user attributes of users who have asked questions in the past and the answer categories corresponding to the user attributes that have been highly evaluated. good.

Further, in the above-mentioned modification, the question answering server 20 includes a second likelihood calculation module 2034 that calculates the second likelihood, and a third likelihood calculation module 2039 that calculates the third likelihood, and each module Although the process of calculating the second likelihood and the third likelihood individually has been described, the present invention is not limited to such an embodiment.
That is, the question-and-answer server 20 calculates a fourth likelihood indicating the likelihood that the question falls under a certain category using both the behavior log and user attributes, and calculates the fourth likelihood based on the first likelihood and the fourth likelihood. You may also select the category to which you should respond.
In this case, as the classification model (behavior/attribute classification model) that calculates the fourth likelihood, a machine learning model that outputs the fourth likelihood in response to the input of the behavior log and user attributes can be adopted. . The behavior/attribute classification model is created by machine learning using behavior logs and user attributes as input data, and appropriate answer categories for questions corresponding to the behavior logs and user attributes as midsummer learning data.

<8. Other variations>
As another modification, the question and answer DB 2022 may be realized by a server that implements a program on a computer. Each part constituting the question-and-answer server 20 may be realized in a distributed manner on a plurality of computers.

Further, in addition to selecting a category based on a question text, the question and answer server 20 may select a category based on a user's attributes without selecting a category based on a user's behavior log. That is, the question-and-answer server 20 may calculate an evaluation index for selecting a category using the first likelihood and the third likelihood without calculating the second likelihood.

Further, the question and answer server 20 may select the category to which the question text falls based on the first likelihood and the user's action log, without calculating the second likelihood.
Alternatively, the question and answer server 20 may select the category to which the question corresponds based on the first likelihood and the user attributes of the user, without calculating the third likelihood.

Although the embodiments of the present disclosure have been described above, various modifications and combinations that may become necessary due to design convenience or other factors are not limited to the specific inventions described in the claims or the embodiments of the invention. It shall be within the scope of the invention corresponding to the examples.

<9. Additional notes>
The contents of the present invention are additionally described below.

(Additional note 1)
A program for causing a computer including a processor and a memory to execute, the program causing the processor to:
Based on the question text input by the user from the user terminal and the question pattern stored in the storage unit, estimation is made from the question text for each of the plurality of categories stored in the storage unit and associated with answer patterns. a step of calculating a first likelihood indicating the degree of association with the question text (step S202);
a step of selecting a category to which the question falls, based on the calculated first likelihood and the user's action log (step S203);
A program that executes a step (step S205) of outputting an answer pattern linked to a selected category to a user terminal.

(Additional note 2)
In the step of calculating the first likelihood (step S203),
Inputting a question sentence into a question classification model 2025 created by machine learning using learning data in which a plurality of question patterns are used as input data and categories corresponding to each of the plurality of question patterns are used as correct answer output data. The program according to appendix 1, which calculates the first likelihood for each of a plurality of categories.

(Additional note 3)
In the step of selecting a category (step S203),
The program according to appendix 2, which selects a category based on the first likelihood and the behavior log calculated from the question classification model 2025 without making any changes to the question classification model 2025.

(Additional note 4)
In the step of selecting a category (step S203), the processor:
a step of calculating a second likelihood indicating the degree of association with the behavior log estimated from the behavior log for each of the plurality of categories stored in the storage unit based on the behavior log (step S2031);
The program according to any one of Supplementary Notes 1 to 3, which executes the step of selecting a category to which the question sentence falls based on the first likelihood and the second likelihood (step S2033).

(Appendix 5)
In the step of calculating the second likelihood (step S2031),
Behavior classification created by machine learning using learning data, where the behavior logs of multiple users are input data, and the correct answer output data is the category that is considered to be an appropriate answer to the question asked after each behavior log. The program according to appendix 4, which calculates the second likelihood for each of a plurality of categories by inputting an action log to the model 2026.

(Appendix 6)
to the processor,
Using a series of action logs related to input of questions from past users and evaluation results from users regarding the categories selected for the questions, the categories answered corresponding to the series of action logs are determined. , the program according to appendix 5, which executes the step of selecting data as learning data for the behavior classification model 2026 (step S214).

(Appendix 7)
In the step of selecting a category based on the first likelihood and the second likelihood (step S203),
The program according to any one of appendices 4 to 6, which selects a category based on an evaluation index obtained by multiplying the first likelihood by the second likelihood.

(Appendix 8)
In the step of selecting a category (step S203),
8. The program according to any one of Supplementary Notes 1 to 7, which selects a category based on a user's attributes in addition to the calculated first likelihood and the user's action log.

(Appendix 9)
A method for a computer comprising a processor and a memory to perform, the method comprising:
Based on the question text input by the user from the user terminal and the question pattern stored in the storage unit, estimation is made from the question text for each of the plurality of categories stored in the storage unit and associated with answer patterns. a step of calculating a first likelihood indicating the degree of association with the question text (step S202);
a step of selecting a category to which the question falls, based on the calculated first likelihood and the user's action log (step S203);
A method comprising: outputting an answer pattern linked to a selected category to a user terminal (step S205).

(Appendix 10)
A system realized by a computer comprising a processor and a memory, the system comprising:
Based on the question text input by the user from the user terminal and the question pattern stored in the storage unit, estimation is made from the question text for each of the plurality of categories stored in the storage unit and associated with answer patterns. means for calculating a first likelihood indicating the degree of association with the question text;
means for selecting a category to which the question falls, based on the calculated first likelihood and the user's action log;
A system comprising means for outputting an answer pattern linked to a selected category to a user terminal.

1 Automatic response system 10 User terminal 20, 20B Question and answer server 202 Storage unit 2021 User database 2022 Question and answer database 2023 Operation log database 2024 Learning data database 2025 Question classification model 2026 Behavior classification model 2027 Attribute classification model 203 Control unit 2031 Transmission/reception control Module 2032 Question text acquisition module 2033 First likelihood calculation module 2034 Second likelihood calculation module 2035 Category selection module 2036 Answer output module 2037 Questionnaire acquisition module 2038 Learning data selection module 2039 Third likelihood calculation module

Claims (10)

A program for causing a computer including a processor and a memory to execute, the program causing the processor to:
Based on the question text input by the user from the user terminal and the question pattern stored in the storage unit, estimation is made from the question text for each of the plurality of categories stored in the storage unit and associated with answer patterns. a step of calculating a first likelihood indicating the degree of association with the question text;
selecting the category to which the question text falls based on the calculated first likelihood and the user's action log;
A program that causes the program to execute the step of outputting the answer pattern linked to the selected category to the user terminal. In the step of calculating the first likelihood,
The question text is applied to a question classification model created by machine learning using learning data in which the plurality of question patterns are input data and the categories to which each of the plurality of question patterns corresponds are correct output data. The program according to claim 1, wherein the first likelihood is calculated for each of the plurality of categories by inputting the first likelihood. In the step of selecting the category,
The program according to claim 2, which selects the category based on the first likelihood calculated from the question classification model and the action log without making any changes to the question classification model. In the step of selecting the category, the processor:
Based on the action log, calculating a second likelihood indicating a degree of association with the action log estimated from the action log for each of the plurality of categories stored in the storage unit;
2. The program according to claim 1, wherein the program executes the step of selecting the category to which the question sentence applies based on the first likelihood and the second likelihood. In the step of calculating the second likelihood,
It was created by machine learning using learning data in which the behavior logs of multiple users are input data, and the categories that are considered appropriate answers to questions asked after each of the behavior logs are the correct answer output data. 5. The program according to claim 4, wherein the second likelihood is calculated for each of the plurality of categories by inputting the behavior log into a behavior classification model. the processor;
Answer corresponding to the series of action logs using a series of action logs related to the input of the question text from past users and evaluation results from the users regarding the categories selected for the question text. 6. The program according to claim 5, which causes the program to execute a step of selecting the categories as the learning data for the behavior classification model. In the step of selecting the category based on the first likelihood and the second likelihood,
The program according to any one of claims 4 to 6, which selects the category based on an evaluation index obtained by multiplying the first likelihood by the second likelihood. In the step of selecting the category,
The program according to claim 1, wherein the category is selected based on the user's attributes in addition to the calculated first likelihood and the user's action log. A method for execution by a computer comprising a processor and a memory, the method comprising:
Based on the question text input by the user from the user terminal and the question pattern stored in the storage unit, estimation is made from the question text for each of the plurality of categories stored in the storage unit and associated with answer patterns. a step of calculating a first likelihood indicating the degree of association with the question text;
selecting the category to which the question text falls based on the calculated first likelihood and the user's action log;
A method comprising: outputting the answer pattern linked to the selected category to the user terminal. A system realized by a computer comprising a processor and a memory, the system comprising:
Based on the question text input by the user from the user terminal and the question pattern stored in the storage unit, estimation is made from the question text for each of the plurality of categories stored in the storage unit and associated with answer patterns. means for calculating a first likelihood indicating the degree of association with the question text,
means for selecting the category to which the question text falls based on the calculated first likelihood and the user's action log;
A system comprising: means for outputting the answer pattern linked to the selected category to the user terminal.