WO2022201302A1 - Ｑａデータ評価装置 - Google Patents

Ｑａデータ評価装置 Download PDF

Info

Publication number
WO2022201302A1
WO2022201302A1 PCT/JP2021/011973 JP2021011973W WO2022201302A1 WO 2022201302 A1 WO2022201302 A1 WO 2022201302A1 JP 2021011973 W JP2021011973 W JP 2021011973W WO 2022201302 A1 WO2022201302 A1 WO 2022201302A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
chatbot
chat
user
question
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/011973
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
大地 蝶野
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to US18/282,113 priority Critical patent/US20240154921A1/en
Priority to PCT/JP2021/011973 priority patent/WO2022201302A1/ja
Priority to JP2023508213A priority patent/JP7670117B2/ja
Publication of WO2022201302A1 publication Critical patent/WO2022201302A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/02—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/21—Monitoring or handling of messages
    • H04L51/216—Handling conversation history, e.g. grouping of messages in sessions or threads

Definitions

  • the present invention relates to a QA data evaluation device, a QA data evaluation method, and a recording medium.
  • the chatbot system refers to a QA data DB (database) that stores QA data that associates expected question texts and response texts to the question texts, and responds according to the question texts sent from the chat user. Get the text and present it to the chat user. Therefore, it is no exaggeration to say that the reliability of a chatbot system is determined by the quality of QA data. Therefore, in order to improve the quality of QA data, the administrator of the chatbot system creates learning data representing the quality of QA data based on the results of actual operation, corrects QA data based on the learning data, Maintenance such as deletion and addition is being carried out.
  • QA data DB database
  • the quality of the QA data can be evaluated by the chat user inputting evaluation information indicating whether or not the response included in the QA data is appropriate for the question.
  • the evaluation information actively input by the chat user is hereinafter referred to as “active evaluation information”.
  • evaluation information that is not actively input by chat users is hereinafter referred to as “inactive evaluation information”.
  • Patent Literature 1 the inflection and pitch of a chat user's voice after presenting a response are acquired as inactive evaluation information, and based on the acquired information, learning data representing the quality of QA data is disclosed. It is disclosed to create a
  • Patent Document 2 describes text information obtained by converting the utterance made by the chat user to the response of the chatbot into text, voice data obtained by digitizing the voice of the above utterance, and the appearance of the chat user when listening to the response.
  • image data obtained by digitizing the captured image and the chat user's biological information pulse, heart rate, blood pressure, brain wave, respiratory rate, etc.
  • a main object of the present invention is to provide an information processing device that makes it possible to easily acquire non-active evaluation information.
  • a QA data evaluation device includes: Acquisition means for acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information related to the use of the chatbot by the user; extracting means for extracting a feature amount related to temporal behavior of the user's use of the chatbot from the log information; generating means for generating QA data evaluation information indicating whether the QA data is good or bad based on the feature quantity; is configured to include
  • a QA data evaluation method includes: Acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; extracting a feature amount related to temporal behavior of the user's use of the chatbot from the log information; QA data evaluation information indicating whether the QA data is good or bad is generated based on the feature quantity.
  • a computer-readable recording medium includes to the computer, A process of acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information related to the use of the chatbot by the user; A process of extracting a feature amount related to temporal behavior of the user's use of the chatbot from the log information; A process of generating QA data evaluation information indicating whether the QA data is good or bad based on the feature quantity; It is configured to record a program for causing the
  • the present invention can easily acquire non-active evaluation information.
  • FIG. 1 is a block diagram of an information processing device according to a first embodiment of the present invention
  • FIG. It is a figure which shows the structural example of QA data DB in the information processing apparatus which concerns on the 1st Embodiment of this invention.
  • 3 is a diagram showing a configuration example of a chat log DB in the information processing device according to the first embodiment of the present invention
  • FIG. It is a figure which shows the structural example of cluster DB in the information processing apparatus which concerns on the 1st Embodiment of this invention.
  • FIG. 5 is a diagram showing another example of rules in the information processing device according to the first embodiment of the present invention. It is a figure which shows the structural example of learning data DB in the information processing apparatus which concerns on the 1st Embodiment of this invention.
  • 4 is a flowchart showing an example of chatbot processing and chat log collection processing in the information processing apparatus according to the first embodiment of the present invention; 4 is a flow chart showing an example of learning data generation processing in the information processing apparatus according to the first embodiment of the present invention;
  • FIG. 3 is a diagram showing an example of chat log information in the information processing apparatus according to the first embodiment of the present invention;
  • FIG. 4 is a diagram showing an example of a document generated by collecting question texts and response texts in log information in the information processing apparatus according to the first embodiment of the present invention
  • FIG. 4 is a flow chart showing an example of processing executed in step S25 by a learning data generation unit in the information processing apparatus according to the first embodiment of the present invention
  • FIG. It is a figure which shows the example of the chatbot management screen in the information processing apparatus which concerns on the 1st Embodiment of this invention.
  • It is a block diagram of a QA data evaluation device according to a second embodiment of the present invention.
  • FIG. 1 is a block diagram of an information processing apparatus 100 according to the first embodiment of the invention.
  • the information processing apparatus 100 includes a chatbot function of outputting an appropriate response text to a terminal device operated by a chat user in response to a question text received from a terminal device operated by the chat user, and a chat function. and the ability to evaluate the QA data used by the bot.
  • the information processing apparatus 100 includes a communication I/F (interface) section 110, an operation input section 120, a screen display section 130, a storage section 140, and an arithmetic processing section 150 as main components.
  • the communication I/F unit 110 is composed of a data communication circuit, and is configured to perform data communication with one or more user terminals 160 wirelessly or by wire.
  • the user terminal 160 is an information processing device used by a user (chat user) who chats with a chatbot.
  • the user terminal 160 is, for example, a personal computer, a smart phone, a tablet terminal, etc., having a communication function. Any external device (not shown) other than the user terminal 160 may be connected to the communication I/F section 110 .
  • the operation input unit 120 is composed of devices such as a keyboard and a mouse, and is configured to detect an operator's operation and output it to the arithmetic processing unit 150 .
  • the screen display unit 130 is composed of a device such as an LCD (Liquid Crystal Display), and is configured to display various information on the screen according to instructions from the arithmetic processing unit 150 .
  • the storage unit 140 is composed of one or more storage devices such as hard disks and memories, and is configured to store processing information and programs 141 necessary for various processes in the arithmetic processing unit 150 .
  • the program 141 is a program that realizes various processing units by being read and executed by the arithmetic processing unit 150. From an external device (not shown) or a recording medium via a data input/output function such as the communication I/F unit 110, It is read in advance and stored in the storage unit 140 .
  • Main processing information stored in the storage unit 140 includes a QA data DB 142, a chat log DB 143, a cluster DB 144, a rule DB 145, and a learning data DB 146.
  • the QA data DB 142 is a database that stores QA data that associates question texts and response texts.
  • FIG. 2 shows a configuration example of the QA data DB 142.
  • the QA data DB 142 in this example consists of a plurality of entries each storing one QA data 1420 .
  • QA data 1420 stored in each entry consists of QA data ID 1421 , question text 1422 and response text 1423 .
  • An ID such as a number for uniquely identifying the QA data 1420 is set in the QA data ID 1421 item.
  • text information related to a question assumed to be asked by a chat user is set.
  • the item of response data 1424 is set with text information relating to the response to the inquiry by the question text 1422 .
  • the chat log DB 143 is a database that stores chat log information between chatbots and chat users.
  • FIG. 3 shows a configuration example of the chat log DB 143.
  • the chat log DB 143 in this example consists of a plurality of entries each storing log information 1430 of one chat. Chat log information 1430 stored in each entry is composed of a chat user ID 1431 , a chat ID 1432 , and a plurality of event data 1433 .
  • An ID for uniquely identifying a chat user is set in the chat user ID 1431 item.
  • the chat ID 1432 field contains an ID such as a number for uniquely identifying each chat with the chat user identified by the chat user ID 1431 .
  • the event data 1433 field contains data related to chat events.
  • the event data 1433 consists of a date and time 14331, a type 14332, a text 14333, and a QA data ID 14334.
  • the type of event data is set.
  • Session establishment means that a chat session has been established (connected) between the chatbot and the chat user.
  • Session release means that the session established between the chatbot and the chat user is released (disconnected).
  • Question means that the chatbot has received the question text from the chat user.
  • Response means that the chatbot has sent response text to the chat user.
  • the date and time when the event of the type occurred is set, for example, in the format of "year, month, day, hour, minute, second, comma second".
  • question text information is set when the type is question
  • response text information is set when the type is response.
  • a NULL value is set in the text 14333 item.
  • the ID of the QA data is set when the QA data including the question text matching the text 14333 related to the question exists, and when it does not exist, the corresponding question text is set. is not registered.
  • the type when the type is response, the same information as the information set in the item of QA data ID 14333 in the event data 1433 of the question that is the premise of the response is set.
  • a NULL value is set in the QA data ID 14333 item.
  • the cluster DB 144 is a database that stores information about one or more clusters generated by clustering semantically similar log information pieces of log information 1430 of a plurality of chats stored in the chat log DB 143 into the same cluster.
  • FIG. 4 shows a configuration example of the cluster DB 144.
  • the cluster DB 144 in this example consists of a plurality of entries each storing one cluster 1440 .
  • a cluster 1440 stored in each entry is composed of a cluster ID 1441 , a question label 1442 , a chat log number 1443 , and a chat log ID list 1434 .
  • An ID such as a number for uniquely identifying the cluster 1440 is set in the cluster ID 1441 item.
  • a question text commonly included in log information of chats belonging to the cluster 1440 is set as a question label.
  • a list of chat log IDs for identifying chat log information 1430 belonging to the cluster 1440 is set in the chat log ID list 1434 item.
  • the chat log ID may be composed of a combination of the chat user ID 1431 and the chat ID 1432 shown in FIG. 3, for example.
  • the rule DB 145 is a database that stores rules for creating learning data representing the quality of QA data from the log information in the clusters stored in the cluster DB 144.
  • FIG. 5A shows a configuration example of the rule DB 145.
  • the rule DB 145 in this example consists of multiple entries each storing one rule 1450 .
  • a rule 1450 stored in each entry is composed of a rule ID 1451 , a feature quantity type 1452 , learning target QA data 1453 , and evaluation value calculation criteria 1454 .
  • An ID such as a number for uniquely identifying the rule 1450 is set in the rule ID 1451 item.
  • the item of the feature type 1452 is set with the type of the feature amount of the chat user's behavior over time during the chat, which is calculated from the log information in the cluster 1440 saved in the cluster DB 144 .
  • Temporal behavior is the elapsed time from receiving a response to asking a question, the elapsed time from receiving a response to the end of the chat, the number of questions per unit time, and the elapsed time from the start to the end of the chat. and so on.
  • the learning target QA data 1453 field contains data specifying QA data for which learning data is to be created based on the feature quantity set in the feature quantity type 1452 field.
  • a standard for calculating an evaluation value representing the quality of the QA data set in the item of QA data to be learned 1453 is set.
  • FIG. 5B is a diagram showing an example of rules stored in the rule DB 145.
  • the item of the feature quantity type 1452 is set to "the time T1 from when the chat user receives the response text to the last question until the chat ends", and the QA data to be learned 1453 is set.
  • FIG. 5C is a diagram showing another example of rules stored in the rule DB 145.
  • FIG. 1 In the rule 1450-2 of this example, "Frequency N1 of asking the next question before a predetermined time has elapsed since the previous question" is set in the item of feature type 1452, and " QA data related to question content commonly included in the log information in the cluster" is set, and the item of the evaluation value calculation criteria 1454 is set as follows: "The higher the ratio of chats with a frequency N1 of a predetermined frequency TH2 or higher, the lower the evaluation value. ” is set.
  • This rule 1450-2 utilizes the chat user's tendency to rephrase the content of the question and sometimes repeat the question many times when an accurate response (answer) is not returned to the question. is doing.
  • the rules 1450 stored in the rule DB 145 are not limited to the rules 1450-1 and 1450-2 described above, and may be rules with other contents, or rules with three or more rules. good too.
  • a rule may be used in which the evaluation value calculation criterion 1454 of the rule 1450-1 is replaced with ⁇ the higher the percentage of chats in which the time T1 is equal to or longer than the predetermined time TH1, the higher the evaluation value''.
  • a rule may be used in which the item of evaluation value calculation criteria 1454 of rule 1450-2 is replaced with "the higher the rate of chats with frequency N1 less than predetermined frequency TH2, the higher the evaluation value”.
  • the learning data DB 146 is a database that stores learning data representing the quality of QA data.
  • FIG. 6 shows a configuration example of the learning data DB 146.
  • the learning data DB 146 in this example is composed of a plurality of entries each storing one piece of learning data.
  • Learning data 1460 stored in each entry consists of learning data ID 1461, question text 1462, response text 1463, QA data ID 1464, evaluation value 1465, cluster ID 1466, rule ID 1467, confirmation flag 1468, and administrator name 1469. be.
  • An ID such as a number for uniquely identifying learning data is set in the learning data ID 1461 item.
  • the question text 1462 and response text 1463 fields are set with QA data to be evaluated, that is, question texts and response texts exchanged between the chat user and the chatbot.
  • QA data ID 1464 field when there is QA data containing a question text that matches the question text set in the question text 1462, the ID of the existing QA data is set. Information is set to the effect that the question text to be asked was not registered.
  • the item of the evaluation value 1465 is set with a value representing the quality of the QA data to be evaluated.
  • the evaluation value 1465 may be, for example, a binary value representing that the QA data is good (eg 1) and a value representing that the QA data is bad (eg 0).
  • the evaluation value 1465 may be multivalued so that the degree of quality of the QA data can be set in three or more stages (for example, 10 stages). Alternatively, the evaluation value 1465 may further include a value (for example, NULL value) indicating that the evaluation value is not finalized.
  • the cluster ID 1441 of the cluster 1440 used to generate the learning data is set in the cluster ID 1466 item.
  • the rule ID 1467 field contains the rule ID 1451 of the rule 1450 used to generate the learning data.
  • the item of confirmation flag 1468 is set to a state indicating whether or not the learning data 1460 has been confirmed, for example, a value of 1 when confirmed and a value of 0 when unconfirmed.
  • the name of the administrator of the chatbot who confirmed the learning data 1460 for maintenance of the QA data, etc. is set.
  • the arithmetic processing unit 150 has a processor such as one or more MPUs and its peripheral circuits, and reads the program 141 from the storage unit 140 and executes it to cooperate with the hardware and the program 141 to perform various processes. It is configured to realize the part.
  • Main processing units realized by the arithmetic processing unit 150 are a chatbot 151 , a chat log collection unit 152 , a learning data generation unit 153 and a QA data management unit 154 .
  • the chat log collection unit 152, the learning data generation unit 153, and the QA data management unit 154 constitute a QA data evaluation device.
  • the chatbot 151 is configured to chat with chat users. Chatbot 151 establishes a chat session with a chat user according to a request from the chat user. Also, when a question text is sent from the chat user through the established session, the chatbot 151 receives the question text, and searches the QA data DB 142 for QA data including the question text that semantically matches the received question text. to obtain the response text included in the searched QA data. In addition, if the QA data DB 142 does not contain QA data containing a question text that semantically matches the received question text, the chatbot 151 uses a predetermined fixed phrase, such as "The question could not be recognized.
  • the chatbot 151 then transmits the acquired or generated response text to the user terminal 160 of the chat user who made the inquiry, and displays it on the terminal screen of the user terminal 160 . Also, the chatbot 151 releases the chat session established with the chat user according to a request from the chat user.
  • the chat log collection unit 152 is configured to collect log information of chats with chat users by the chat bot 151 and store it in the chat log DB 143 .
  • the chat log collection unit 152 secures a new entry in the chat log DB 143, and stores the chat user ID 1431, chat ID 1432, and Event data 1433 related to session establishment (session establishment date and time 1431, session establishment type 14332, NULL value text 14333, and QA data ID 14334) are set.
  • the chat log collection unit 152 stores the event data 1433 related to the question (the date and time 1431 when the question was received, Type 14332 to represent, text 14333 to represent question text information, and QA data ID 14334) are set.
  • the chatbot 151 transmits a response text to the chat user through the session the chat log collection unit 152 stores event data 1433 (response transmission date and time 1431, response transmission date and time 1431, Type 14332 representing response, text 14333 representing response text information, and QA data ID 14334) are set.
  • chat log collection unit 152 adds event data 1433 related to the session release to the secured entry in the chat log DB 143 (session release date and time 1431, session release type 14332, Set NULL value text 14333 and QA data ID 14334).
  • the learning data generation unit 153 uses the chat log information stored in the chat log DB 143 and the rules stored in the rule DB 145 to create learning data representing the quality of the QA data, and stores the learning data in the learning data DB 146.
  • the learning data generation unit 153 is generated when a certain amount of log information is accumulated in the chat log DB 143, when a certain amount of time has elapsed since the previous learning data was created, periodically, or when instructed by an operator. Start the process of creating learning data.
  • the learning data generating unit 153 clusters semantically similar pieces of log information of a plurality of chats stored in the chat log DB 143 into the same cluster, and stores the generated clusters in the cluster DB 144 .
  • the learning data generation unit 153 calculates the feature amount from the chat log information in the cluster by applying the rule stored in the rule DB 145 for each cluster stored in the cluster DB 144, and the calculated feature amount.
  • Statistical processing calculation of an evaluation value based on the results of the statistical processing, and the like are performed to generate learning data, and the generated learning data is stored in the learning data DB 146 .
  • Statistical processing includes creation of frequency distributions, histograms, mean values, median values, modes, and the like.
  • the QA data management unit 154 assists the chatbot manager in performing maintenance such as correcting, deleting, and adding QA data stored in the QA data DB 142 based on the learning data stored in the learning data DB 146.
  • the QA data management unit 154 displays a list of learning data stored in the learning data DB 146 on the screen display unit 130 so that the administrator can refer to the contents of the learning data.
  • the QA data management unit 154 also displays a list of QA data stored in the QA data DB 142 on the screen display unit 130 so that the administrator can interactively correct, delete, and add QA data.
  • the operation of the information processing device 100 is roughly divided into chatbot processing that is performed when an inquiry (question) from a chat user is received, and QA data evaluation processing. Further, the QA data evaluation process is roughly divided into a chat log collection process, a learning data generation process for generating learning data, and a maintenance process for maintaining the QA data.
  • chatbot processing and chat log collection processing will be described with reference to the flowchart of FIG. Chatbot processing and chat log collection processing are performed for each chat user and chat by chatbot 151 and chat log collection unit 152 .
  • chat bot 151 of the information processing device 100 receives an operation for starting a chat on the user terminal 160 from the chat user, the chat bot 151 performs chat start processing (step S1).
  • the chatbot 151 performs a process of establishing a session for chatting between the user terminal 160 used by the chat user and the chatbot 151 in the chat start process of step S1.
  • the chat bot 151 further displays a standard text at the time of chat start (for example, "Please enter your inquiry" on the screen of the user terminal 160 used by the chat user through the established session. Please.”) may be displayed.
  • chat log collection unit 152 When a chat session is established between the chat user and the chatbot 1510, the chat log collection unit 152 performs chat log collection processing (step S2). In the chat log collection process of step S2, the chat log collection unit 152 secures one new entry in the chat log DB 143, and stores the chat user ID 1431, chat ID 1432, And event data 1433 related to session establishment (session establishment date and time 1431, session establishment type 14332, NULL value text 14333, and QA data ID 14334) are set.
  • the chatbot 151 checks whether there are any new questions from the chat user (step S3).
  • a new question is a new chat input by a chat user.
  • the chatbot 151 proceeds to the process of step S9.
  • the chatbot 151 acquires the input chat content (question text) (step S4).
  • the chat log collection unit 152 stores the date and time 1431 when the question was received, the type 14332 representing the question, the text 14333 representing question text information, and the Then, the event data 1433 composed of the QA data ID 14334 (at this time, NULL value) is additionally set (step S5).
  • the chatbot 151 searches the QA data DB 142 for QA data containing question texts semantically matching the question text obtained from the chat user, and extracts the response text contained in the QA data obtained by the search. , as a response to the chat user (step S6).
  • the chatbot 151 sends a preset fixed phrase to the chat user. Generate as response.
  • the chat log collection unit 152 additionally sets the ID of the existing QA data in step S5. If the event data 1433 does not exist, it is set to the QA data ID 14333 to that effect.
  • the chatbot 151 transmits the generated response to the user terminal 160 used by the chat user, and displays it on the screen of the user terminal 160 (step S7).
  • the chat log collection unit 152 stores the response transmission date and time 1431, the response type 14332, and response text information in the noted entry of the chat log DB 143.
  • Event data 1433 consisting of text 14333 and QA data ID 14334 is additionally set (step S8). Then, the chatbot 151 proceeds to the process of step S9.
  • the chatbot 151 determines whether or not the end of the chat has been detected in step S9.
  • the chatbot 151 may determine that the end of the chat has been detected, for example, when it detects that the chat user has expressed his/her intention to end the chat on the user terminal 160 .
  • the chatbot 151 determines that the end of the chat has not been detected, it returns to the processing of step S3 and repeats the same processing as described above.
  • the chatbot 151 detects the end of the chat, the chatbot 151 performs chat termination processing (step S10).
  • the chatbot 151 performs a process of releasing (disconnecting) the session established with the chat user in the chat end process of step S10.
  • the chat bot 151 further displays a standard text at the end of the chat (for example, "Thank you for using .") may be displayed.
  • the chat log collection unit 152 adds event data 1433 related to the session release (date and time of session release 1431, type 14332 indicating session release, NULL Value text 14333 and QA data ID 14334) are set (step S11).
  • the learning data generation process is performed by the learning data generation unit 153 .
  • the learning data generating unit 153 of the information processing device 100 starts the learning data generating process, first, it reads the chat log information used for generating the learning data from the chat log DB 143 (step S21). For example, the learning data generation unit 153 may read all log information stored in the chat log DB 143 as log information used for generating learning data.
  • the learning data generation unit 153 refers to the date and time set in the date and time 14331, for example, all log information after a predetermined date and time specified by an administrator or the like, or all log information before a predetermined date and time, or , all log information after a predetermined start date and time and before a predetermined end date and time may be read from the chat log DB 143 as log information used for generating learning data.
  • the learning data generation unit 153 clusters log information that is semantically similar to the read log information into the same cluster (step S22).
  • Semantically similar means that the content of the exchanged question text and response text are similar overall and semantically between log information of mutual chats. For example, “I want to cancel my vacation application” and “I want to withdraw my vacation” are examples of chat log information that are semantically similar to each other. Also, “price is high” and “price is high”, and “looks great” and “looks great” are other examples of semantically similar chat log information. Any method may be used to cluster semantically similar chat log information into the same cluster. For example, a collection of question texts and response texts in log information of each chat can be regarded as one document, and a known document clustering method for classifying similar documents into the same cluster can be applied to these document groups. The above clustering may be performed by
  • Examples of known document clustering methods include, but are not limited to, the document clustering method described in Patent Document 4.
  • the document clustering method described in Patent Document 4 (hereinafter referred to as the document clustering method related to the present invention), first, out of the words appearing in two documents included in the document group, the words appearing in one document are and a word appearing in the other document, a concept tree structure representing the hierarchical relationship between the concepts of the two words is acquired.
  • the frequency of occurrence in the document group of common superordinate terms of the above two terms in the acquired concept tree structure or subordinate terms of the superordinate terms, and and the frequency of occurrence in each of the above-mentioned document groups is the maximum when the frequency of occurrence of the above-mentioned two terms is the same, and is the minimum when there is no common superordinate term of the two terms in the concept tree structure.
  • a conceptual similarity which is an index indicating the conceptual closeness of terms, is obtained.
  • the inter-document similarity which is the degree of semantic similarity between two documents included in the document group, is obtained.
  • the documents of the document group are clustered based on the inter-document similarity.
  • the learning data generation unit 153 clusters log information groups including two chat log information LU11 and LU21 shown in FIG. 9 using a document clustering method related to the present invention.
  • log information LU11 of the chat on the left indicates log information of the chat between the chat user U01 and the chatbot 151
  • log information LU21 on the right indicates the log information of the chat between the chat user U02 and the chatbot 151.
  • the two-way arrow indicates an event of establishment or release of a chat session
  • the balloon indicates a response comment sent from the chatbot 151 to the chat user or a question received by the chatbot 151 from the chat user. Indicates a comment event.
  • each event is given a reference numeral LU111 to LU117 and LU211 to LU217 for convenience.
  • the learning data generator 153 collects the question texts and response texts in the log information LU11 shown in FIG. 9 to generate one document LU11B as shown in FIG.
  • the chatbot 151 presents to the chat user at the start and end of the chat, such as "Please enter your inquiry” and "Thank you for using", which are common to all chats. Fixed phrases are excluded.
  • the learning data generation unit 153 also collects the question texts and response texts in the log information LU21 to generate one document LU21B as shown in FIG. Then, the learning data generation unit 153 clusters the document group including the documents LU11B and LU21B by applying the document clustering method related to the present invention. As a result, in the case of the two pieces of log information LU11 and LU21 shown in FIG. Even if "I want to cancel my vacation" exists in separate log information, the two pieces of log information LU11 and LU21 will be clustered in the same cluster.
  • step S22 the learning data generation unit 153 generates a cluster 1440 composed of a cluster ID 1441, a question label 1442, a chat log count 1443, and a chat log ID list 1434 for each of the clusters generated by the above clustering. , is stored in the cluster DB 144 .
  • the learning data generation unit 153 sets the question text "how to cancel vacation" that appears commonly in a plurality of chat log information to the question label 1442 of the cluster to which the two pieces of log information shown in FIG. 9 belong.
  • the learning data generator 153 focuses on one cluster 1440 among the one or more clusters stored in the cluster DB 144 (step S23).
  • the learning data generator 153 focuses on one rule 1450 among the one or more rules stored in the rule DB 145 (step S24).
  • the learning data generator 153 creates learning data 1460 based on the cluster 1440 of interest and the rule 1450 of interest, and stores it in the learning data DB 146 (step S25).
  • FIG. 11 is a flow chart showing an example of the process executed by the learning data generator 153 in step S25.
  • the learning data generation unit 153 first extracts the feature amount of the type set in the item of the feature amount type 1452 of the rule 1450 of interest from each of the chat log information 1430 of the cluster 1440 of interest. is calculated (step S31). For example, in the case of rule 1450-1, the learning data generation unit 153 calculates "the time T1 from when the chat user receives the response text to the last question until the chat ends" from each chat log information. do. For example, in the case of the log information LU11 shown in FIG.
  • the event LU116 is the response to the chat user's last question, so the time from the date and time of the event LU116 to the end of the chat of the event LU117 is calculated as time T1. Further, for example, in the case of rule 1450-2, the learning data generating unit 153 calculates "the frequency N1 of asking the next question before a predetermined time has elapsed since the previous question" from each chat log information. . For example, in the case of the log information LU11 shown in FIG. 9, the question is asked twice for the events LU113 and LU115. Therefore, if the elapsed time from the event LU113 to the event LU115 is less than the predetermined time, the frequency N1 is once. If it is longer than the predetermined time, the frequency N1 becomes 0 times. Incidentally, in the case of chat log information in which the total number of questions is M times, the maximum value of the frequency N1 is M-1.
  • the learning data generation unit 153 statistically processes the feature amount calculated from each piece of chat log information based on the evaluation value calculation criteria 1454 of the rule 1450 of interest (step S32). For example, in the case of the rule 1450-1, the learning data generation unit 153 first calculates the total number S1 of chat log information whose time T1 is less than the predetermined time TH1. Next, the learning data generating unit 153 calculates a ratio R1 of the total number S1 of chat logs in the cluster of interest to the total number S0 of chat logs. Further, in the case of rule 1450-2, the learning data generation unit 153 first calculates the total number S1 of chat log information whose frequency N1 is equal to or greater than the predetermined frequency TH2. Next, the learning data generating unit 153 calculates a ratio R1 of the total number S1 of chat logs in the cluster of interest to the total number S0 of chat logs.
  • the learning data generation unit 153 calculates an evaluation value from the results of statistical processing (step S33). For example, in the cases of rule 1450-1 and rule 1450-2, learning data generation unit 153 lowers the evaluation value as ratio R1 increases. For example, the learning data generation unit 153 sets the evaluation value to 0 if the ratio R1 is 80% or more, sets the evaluation value to 2 if the ratio R1 is 60% or more and less than 80%, and sets the evaluation value to 2 if the ratio R1 is 40% or more and less than 60%. The evaluation value is set to 5, the evaluation value is set to 8 when 20% or more and less than 40%, and the evaluation value is set to 10 when less than 20%. Here, the larger the evaluation value, the higher the evaluation.
  • the learning data generation unit 153 sets necessary information for each item of the learning data ID 1461, the question text 1466, the response text 1463, the QA data ID 1464, the evaluation value 1464, the cluster ID 1466, and the rule ID 1467, and sets the confirmation flag 1468. is set to a value indicating an unconfirmed state, and the administrator name 1469 is set to a NULL value, learning data 1460 is created and stored in the learning data DB 146 .
  • Learning data generation unit 153 sets cluster ID 1441 of cluster 1440 in focus and rule ID 1451 of rule in focus 1450 in the fields of cluster ID 1466 and rule ID 1467 .
  • the learning data generating unit 153 sets the evaluation value calculated in step S33 in the evaluation value 1465 item.
  • the learning data generation unit 153 stores the question text, response text, and set the QA data ID 1421 of the QA data containing them.
  • step S28 the learning data generating unit 153 focuses on one of the rules stored in the rule DB 145 that has not yet been applied to the cluster of interest.
  • step S28 returns to step S25 via step S27, and repeats the same processing as described above using another rule for the cluster of interest.
  • the learning data generation unit 153 finishes applying all the rules to the cluster of interest (YES in step S27)
  • the learning data generation unit 153 selects one of the clusters stored in the cluster DB 144 that has not yet been processed. Attention is shifted (step S28), the process returns to step S24 via step S29, and the same process as described above is repeated for another cluster.
  • the processing of FIG. 8 ends.
  • Data maintenance processing is performed by the QA data management unit 154 .
  • FIG. 12 shows an example of a chatbot management screen 170 displayed on the screen display unit 130 when the QA data management unit 154 is activated by the administrator of the information processing device 100.
  • the chatbot management screen 170 of this example has a learning data list display area 171 , a QA data editing area 172 , a cluster display area 173 , a rule display area 174 and a chat log display area 175 .
  • the learning data list display area 171 is an area for displaying a list of one or more learning data 1460 stored in the learning data DB 146.
  • the QA data management unit 154 may read all the learning data 1460 stored in the learning data DB 146 and display them in the learning data list display area 171 .
  • the QA data management unit 154 may selectively read some learning data 1460 from all the learning data stored in the learning data DB 146 and display it in the learning data list display area 171 .
  • the confirmation flag 1468 may be learning data indicating an unconfirmed state.
  • some learning data may have an evaluation value 1465 higher or lower than the evaluation value specified by the administrator.
  • the QA data management unit 154 sets one of the learning data displayed in the learning data list display area 171 as the current learning data.
  • the QA data management unit 154 clearly displays the current learning data to the administrator by highlighting it.
  • the QA data management unit 154 sets "confirmed" in the item of the confirmation flag 1468 of the current learning data, and sets the name of the administrator logged in to the management screen in the item of the administrator name 1469.
  • the QA data management unit 154 switches the current learning data to another instructed learning data when a change is instructed by an administrator's cursor operation.
  • the QA data editing area 172 is an area for editing such as updating, deleting, and adding QA data.
  • the QA data editing area 172 has a QA data ID column 1721 , a question text column 1722 , a response text column 1723 , an update button 1724 , a delete button 1725 and an add button 1726 .
  • the QA data management unit 154 displays the QA data ID 1464 , question text 1462 and response text 1463 of the current learning data in the QA data ID column 1721 , question text column 1722 and response text column 1723 . Also, the QA data management unit 154 edits the contents of the question text column 1722 and the response text column 1723 according to the administrator's editing operation of the operation input unit 120 .
  • the QA data management unit 154 updates the QA data ID column with the contents of the question text and response text set in the question text column 1722 and the response text column 1723 after editing.
  • the QA data in the QA data DB 142 identified by the QA data ID set in 1721 is updated (overwritten).
  • the QA data management unit 154 deletes the QA data in the QA data DB 142 identified by the QA data ID set in the QA data ID column 1721 .
  • the QA data management unit 154 has a new QA data ID, and adds the question text and response set in the question text field 1722 and response text field 1723 after editing. QA data having text content is created and added to the QA data DB 142 as new QA data.
  • the cluster display area 173 displays the contents of the cluster 1440, that is, the cluster ID 1441, the question label 1442, the number of chat logs 1443, and the chat log ID list 1434.
  • the QA data management unit 154 reads the contents of the cluster 1440 having the cluster ID 1441 matching the cluster ID 1466 of the current learning data from the cluster DB 144 and displays it in the cluster display area 173 .
  • the QA data management unit 154 sets one chat log ID in the chat log ID list 1434 displayed in the cluster display area 173 as the current chat log ID.
  • the QA data management unit 154 clearly indicates the current chat log ID to the administrator by highlighting or the like.
  • the QA data management unit 154 switches the current chat log ID to the specified chat log ID in the list 1434 of chat log IDs in response to a change instruction by the administrator's cursor operation.
  • the chat log display area 175 is an area for displaying chat log information.
  • the QA data management unit 154 reads chat log information having a chat log ID that matches the current chat log ID from the chat log DB 143 and displays it in the chat log display area 175 .
  • the rule display area 174 is an area that displays the contents of the rule 1450, that is, the rule ID 1451, the feature amount type 1452, the QA data to be learned 1453, and the evaluation value calculation criteria 1454.
  • the QA data management unit 154 reads the rule 1450 having the rule ID 1451 matching the rule ID 1467 of the current learning data from the rule DB 145 and displays it in the rule display area 174 .
  • the QA data management unit 154 Since the QA data management unit 154 performs the processing as described above using the chatbot management screen 170 shown in FIG. , the QA data subject to learning can be corrected, deleted, and added in an interactive manner. Further, the QA data management unit 154 displays the contents of the cluster 1440 used to create the learning data 1460 in the cluster display area 173, and displays the details of the chat log information forming the cluster 1440 in the chat log display area 175. Therefore, the administrator can correct, delete, or add QA data while confirming what kind of cluster 1440 and set of chat log information the learning data 1460 is generated from.
  • the QA data management unit 154 displays the contents of the rules 1450 used to create the learning data 1460 in the rule display area 174, the administrator can determine what rules 1450 were used to create the learning data 1460. Correction, deletion, and addition of QA data can be performed while confirming whether it is correct.
  • the information processing apparatus 100 collects chat log information, calculates feature amounts from the collected chat log information, and calculates evaluation values based on the calculated feature amounts. This is because it can be implemented, and it is not necessary to equip the chat user side with special equipment such as a microphone, a camera, or a biometric detection sensor.
  • a plurality of pieces of log information that are semantically similar are clustered into the same cluster, and each of the pieces of log information belonging to the same cluster are clustered into a predetermined feature amount. is extracted, and based on the result of statistically processing the plurality of extracted feature quantities, learning data representing the quality of QA data related to question texts commonly included in the log information in the cluster is created. Therefore, it is possible to reduce variations in evaluation due to behavior of specific chat users.
  • FIG. 13 is a block diagram of the QA data evaluation device 200 according to this embodiment.
  • the QA data evaluation device 200 includes acquisition means 201 , extraction means 202 and generation means 203 .
  • Acquisition means 201 is configured to acquire QA data including the contents of questions from users to chatbots and the contents of responses from chatbots to questions, and log information related to the use of chatbots by the users.
  • the extracting means 202 is configured to extract feature quantities relating to temporal behavior of the user's use of the chatbot from the log information.
  • the generation means 203 is configured to generate QA data evaluation information indicating the quality of the QA data based on the feature amount.
  • the QA data evaluation device 200 configured in this manner operates as follows. That is, first, the acquisition unit 201 acquires QA data including the contents of questions from the user to the chatbot and the contents of responses from the chatbot to the questions, and log information regarding the use of the chatbot by the user. Next, the extracting means 202 extracts a feature quantity relating to temporal behavior of the user's use of the chatbot from the log information. Next, the generating means 203 generates QA data evaluation information indicating whether the QA data is good or bad based on the feature amount.
  • non-active evaluation information can be easily acquired.
  • the QA data evaluation device 200 collects chat log information, calculates feature values from the collected chat log information, and calculates evaluation values based on the calculated feature values. This is because it can be implemented on the side of the chat user, and it is not always necessary to equip the chat user side with special equipment such as a microphone, a camera, and a biometric detection sensor.
  • the feature amount of the chat user's behavior over time during the chat is calculated from the chat log information, and the learning data representing the quality of the QA data is created based on the calculated feature amount.
  • other information may be considered to create the learning data.
  • active evaluation information, chat user's voice, image, biometric information (pulse, heart rate, blood pressure, brain wave, breathing rate, etc.), URL selection, date and time of use, user terminal information (PC, smartphone, etc.) are exemplified.
  • Active evaluation information is created based on information on reactions shown by chat users who received responses during operation of the chatbot.
  • Active evaluation information is information that a chat user actively and deliberately enters for the purpose of evaluating a presented response. Examples of active evaluation information include utterances, text, and pictograms such as “like”, “wonderful”, “clever”, etc. that indicate good evaluation, and "no", "no”, etc. that indicate bad evaluation. , stamps, etc.
  • active evaluation information is input by means of social buttons indicating "like” or "bad”, for example.
  • the active evaluation information necessary for generating learning data may be insufficient. It is said that active evaluation information is obtained for about 10% of all questions. Therefore, it is important to measure the chat user's degree of satisfaction and evaluation of the presented response using information other than active evaluation information, that is, non-active evaluation information, and create learning data. According to the present invention, such inactive evaluation information can be easily created.
  • the present invention can be applied to operational management of chatbots, and can be used, for example, to maintain QA data.
  • a QA data evaluation device comprising: [Appendix 2] Further comprising clustering means for clustering a plurality of the log information into a plurality of groups according to semantic similarity of the log information; The extracting means extracts the feature amount from each of a plurality of pieces of log information belonging to each of the plurality of clusters, The generating means generates the QA data evaluation information based on results of statistically processing a plurality of feature quantities extracted from
  • the QA data evaluation device according to appendix 1.
  • the feature amount is a feature amount related to the time from the output of the content of the response to the last question from the user to the end of the chat, The QA data evaluation device according to appendix 1 or 2.
  • the feature amount is a feature amount related to the frequency with which the content of another question is input before a predetermined time has passed since the content of the question was input to the chatbot by the user.
  • [Appendix 5] further comprising QA data management means for displaying the generated QA data evaluation information; 5.
  • the QA data evaluation device according to any one of Appendices 1 to 4.
  • the QA data management means updates, deletes, or adds the QA data in response to an operation input to the QA data by the manager of the chatbot;
  • the QA data management means displays the log information used to generate the QA data evaluation information.
  • the QA data management means displays a rule including the type of the feature amount used to create the QA data evaluation information from the log information and a calculation criterion for the evaluation value representing the quality of the QA data, The QA data evaluation device according to any one of Appendices 5 to 7.
  • [Appendix 9] Acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; extracting a feature amount related to temporal behavior of the user's use of the chatbot from the log information; generating QA data evaluation information indicating whether the QA data is good or bad based on the feature quantity; QA data evaluation method.
  • Information processing device 110 Communication I/F unit 120 Operation input unit 130 Screen display unit 140 Storage unit 141 Program 142 QA data DB 143 Chat Log DB 144 cluster database 145 Rule DB 146 learning data database 150 Arithmetic processing unit 151 Chatbot 152 Chat log collection unit 153 Learning data generation unit 154 QA data management unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Information Transfer Between Computers (AREA)
PCT/JP2021/011973 2021-03-23 2021-03-23 Ｑａデータ評価装置 Ceased WO2022201302A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US18/282,113 US20240154921A1 (en) 2021-03-23 2021-03-23 Qa data evaluation apparatus
PCT/JP2021/011973 WO2022201302A1 (ja) 2021-03-23 2021-03-23 Ｑａデータ評価装置
JP2023508213A JP7670117B2 (ja) 2021-03-23 2021-03-23 Ｑａデータ評価装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/011973 WO2022201302A1 (ja) 2021-03-23 2021-03-23 Ｑａデータ評価装置

Publications (1)

Publication Number Publication Date
WO2022201302A1 true WO2022201302A1 (ja) 2022-09-29

Family

ID=83396494

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/011973 Ceased WO2022201302A1 (ja) 2021-03-23 2021-03-23 Ｑａデータ評価装置

Country Status (3)

Country Link
US (1) US20240154921A1 (https=)
JP (1) JP7670117B2 (https=)
WO (1) WO2022201302A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7722745B1 (ja) 2024-08-26 2025-08-13 真由美 稲場 対話システム、及び対話方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240267391A1 (en) * 2023-02-06 2024-08-08 KnowBe4, Inc. Systems and methods for security event association rule refresh

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017117184A (ja) * 2015-12-24 2017-06-29 大日本印刷株式会社 ロボット、質問提示方法、及びプログラム
JP2019045978A (ja) * 2017-08-30 2019-03-22 国立大学法人 奈良先端科学技術大学院大学 対話制御装置、学習装置、対話制御方法、学習方法、制御プログラム、および、記録媒体
JP2019082964A (ja) * 2017-10-31 2019-05-30 エヌ・ティ・ティ・コムウェア株式会社 情報提供システム、管理装置、情報分析装置、及び管理方法
JP2020057117A (ja) * 2018-09-28 2020-04-09 株式会社リコー 情報処理装置、情報処理方法及びプログラム
JP2020091513A (ja) * 2018-12-03 2020-06-11 楽天株式会社 情報処理装置、情報処理方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6433614B1 (ja) 2018-04-16 2018-12-05 Ｊｅインターナショナル株式会社 チャットボット検索システムおよびプログラム

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017117184A (ja) * 2015-12-24 2017-06-29 大日本印刷株式会社 ロボット、質問提示方法、及びプログラム
JP2019045978A (ja) * 2017-08-30 2019-03-22 国立大学法人 奈良先端科学技術大学院大学 対話制御装置、学習装置、対話制御方法、学習方法、制御プログラム、および、記録媒体
JP2019082964A (ja) * 2017-10-31 2019-05-30 エヌ・ティ・ティ・コムウェア株式会社 情報提供システム、管理装置、情報分析装置、及び管理方法
JP2020057117A (ja) * 2018-09-28 2020-04-09 株式会社リコー 情報処理装置、情報処理方法及びプログラム
JP2020091513A (ja) * 2018-12-03 2020-06-11 楽天株式会社 情報処理装置、情報処理方法

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7722745B1 (ja) 2024-08-26 2025-08-13 真由美 稲場 対話システム、及び対話方法
JP7823961B1 (ja) 2024-08-26 2026-03-04 真由美 稲場 対話システム、及び対話方法
JP2026041055A (ja) * 2024-08-26 2026-03-10 真由美 稲場 対話システム、及び対話方法
JP2026047153A (ja) * 2024-08-26 2026-03-13 真由美 稲場 対話システム、及び対話方法

Also Published As

Publication number Publication date
US20240154921A1 (en) 2024-05-09
JP7670117B2 (ja) 2025-04-30
JPWO2022201302A1 (https=) 2022-09-29

Similar Documents

Publication Publication Date Title
US9514226B2 (en) Methods and systems useful for identifying the most influent social media users in query-based social data streams
Ampofo et al. Text mining and social media: When quantitative meets qualitative and software meets people
US11033216B2 (en) Augmenting questionnaires
CN112115299A (zh) 视频搜索方法、装置、推荐方法、电子设备及存储介质
US20190102374A1 (en) Predicting future trending topics
US20160026919A1 (en) System and method for social event detection
EP3996373A2 (en) Method and apparatus of generating bullet comment, device, and storage medium
CN113569037A (zh) 一种消息处理方法、装置以及可读存储介质
JP2013114648A (ja) サーバ装置、情報端末及びプログラム
US20130304819A1 (en) Methods and systems of detection of most relevant insights for large volume query-based social data stream
WO2022201302A1 (ja) Ｑａデータ評価装置
CN114818635A (zh) 数据报表生成方法、装置、电子设备及存储介质
US20190008466A1 (en) Life log utilization system, life log utilization method, and recording medium
CN106487642A (zh) 一种推送信息的方法和装置
US20150193529A1 (en) Opinion analyzing system and method
Steffan et al. Multinational and multimodal character framing of political candidates in online news: Do political and media system classifications matter?
KR101894060B1 (ko) 사용자 분석기반 챗봇을 이용한 지능형 광고 제공 서버
CN120562398A (zh) 一种基于用户画像的ai智能短信内容生成优化方法及系统
CN117473070B (zh) 智能机器人的多渠道应用方法、智能机器人和存储介质
CN119477328A (zh) 一种智能客服的回复方法、系统、设备及介质
CN118888156A (zh) 基于患者对话的临床试验项目匹配方法、装置及电子设备
JP7169031B1 (ja) プログラム、情報処理装置、情報処理システム、情報処理方法、情報処理端末
CN111680224A (zh) 跨平台课程推送方法、装置、电子设备、存储介质
CN118708808A (zh) 基于大模型的推荐方法、装置、设备以及存储介质
CN114765694B (zh) 基于直播的信息处理方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21932912

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023508213

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18282113

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21932912

Country of ref document: EP

Kind code of ref document: A1