WO2021135322A1 - Procédé, appareil et système de préparation automatique de questions - Google Patents

Procédé, appareil et système de préparation automatique de questions Download PDF

Info

Publication number
WO2021135322A1
WO2021135322A1 PCT/CN2020/111954 CN2020111954W WO2021135322A1 WO 2021135322 A1 WO2021135322 A1 WO 2021135322A1 CN 2020111954 W CN2020111954 W CN 2020111954W WO 2021135322 A1 WO2021135322 A1 WO 2021135322A1
Authority
WO
WIPO (PCT)
Prior art keywords
question
answer
matching degree
code
matching
Prior art date
Application number
PCT/CN2020/111954
Other languages
English (en)
Chinese (zh)
Inventor
陈宜琳
倪合强
张兵兵
徐垚
梁诗雯
Original Assignee
苏宁云计算有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 苏宁云计算有限公司 filed Critical 苏宁云计算有限公司
Priority to CA3166342A priority Critical patent/CA3166342A1/fr
Publication of WO2021135322A1 publication Critical patent/WO2021135322A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3329—Natural language query formulation or dialogue systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/334—Query execution
    • G06F16/3344—Query execution using natural language analysis

Definitions

  • This application relates to the field of question and answer questions, and in particular to a method, device and system for automatically generating questions.
  • the intelligent customer service of an e-commerce platform is a question and answer scenario with users.
  • a question bank is set up in advance, and the test questions are selected from the question bank in accordance with a certain rule, such as a sequence, to simulate the user asking questions to the intelligent customer service to obtain the answer of the intelligent customer service, so as to detect the accuracy of the intelligent customer service's answer.
  • a certain rule such as a sequence
  • This application provides an automatic question generation method, device and system to solve the problem that the real scene cannot be simulated when simulating question and answer in the prior art, so that there is no relevance between the previous questions and the previous questions.
  • the first aspect of the present application provides a method for automatically generating questions, and the method includes:
  • the candidate test question is used as a target test question to be asked.
  • the method further includes:
  • the next question in the question bank is used as the target test question to be asked.
  • the calculating the first degree of matching between the question and answer content of each historical test question and the question and answer content of the current test question includes:
  • the first degree of matching is calculated according to the degree of matching between the question and the answer.
  • the selecting the next question of the first historical question with the highest matching degree as the candidate question includes:
  • the next question of the first history question with the highest matching degree among the candidate history questions is selected as the candidate question.
  • the method further includes:
  • a second aspect of the present application provides an automatic question generation device, the device including:
  • the current examination question acquisition unit to obtain the content of the question and answer for the current examination question
  • the first matching degree calculation unit is used to calculate the first matching degree between the question and answer content of each historical test question and the question and answer content of the current test question;
  • the candidate test question unit is used to select the next question of the first historical test question with the highest matching degree as the candidate test question;
  • the second matching degree calculation unit is used to calculate the second matching degree between the candidate test question and the next question in the question bank
  • the question-making unit is configured to use the candidate test question as a target test question when the second matching degree satisfies a preset condition.
  • the question generating unit is further configured to use the next question in the question bank as a target test question when the second matching degree does not satisfy a preset condition.
  • the first matching degree calculation unit includes a first coding unit, a question matching degree calculation unit, an answer matching degree calculation unit, and a first matching degree calculation subunit;
  • the first coding unit is configured to code the question and answer content of the current test question according to a preset question and answer model to obtain the question code and answer code of the current test question;
  • the first coding unit is further configured to code the question and answer content of each historical test question according to a preset question and answer model to obtain the question code and answer code of each historical test question;
  • the question matching degree calculation unit is used to calculate the question matching degree between the question code of each historical test question and the question code of the current test question;
  • the answer matching degree calculation unit is used to calculate the answer matching degree between the answer code of each historical test question and the answer code of the current test question;
  • the first matching degree calculation subunit is configured to calculate the first matching degree according to the matching degree of the question and the degree of matching of the answer.
  • the device further includes:
  • a second coding unit a third matching degree calculation unit, a fourth matching degree calculation unit, and an answer score calculation unit;
  • the second coding unit is configured to code the question and answer content of the current test question according to a preset question and answer model to obtain the question code and answer code of the current test question;
  • the third matching degree calculation unit is used to calculate the third matching degree between the question code and the answer code of the current test question
  • the fourth matching degree calculation unit is configured to calculate the fourth matching degree between the answer of the current test question and the standard answer of the current test question according to a text similarity algorithm
  • the answer score calculation unit is configured to calculate the score of the answer to the current test question according to the third matching degree and the fourth matching degree.
  • the third aspect of this application provides a computer system, including:
  • One or more processors are One or more processors.
  • a memory associated with the one or more processors where the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, the method described above is executed.
  • the technical solution of this application selects the similar question and answer from the historical question and answer according to the current question and answer situation, and when the next sentence of the closest historical question and answer is highly similar to the next question in the question bank, select the next sentence of the historical question and answer As the next exam question. Realize that according to the user's answer to each question, automatically determine the relevant questions from the historical question and answer to continue to ask questions, build a scene-based simulation training, and improve the correlation between the test questions.
  • the degree of matching between the comprehensive standard answer and the test taker’s answer, and the degree of matching between the test question and the test taker’s answer are used to score the test taker’s answers, so that the scoring of the test taker’s answer meets the real scenario response and can be close to the standard answer specifications.
  • Figure 1 is the flow chart of the method of this application.
  • FIGS 2-4 are schematic diagrams of specific embodiments of the present application.
  • Fig. 5 is a structural diagram of a device according to an embodiment of the present application.
  • Figure 6 is a structural diagram of a computer system.
  • the way of preparing questions from the question bank in a preset order ignores the correlation between the question and answer of the previous question and the next question, so that the next question is separated from the question and answer of the previous question, resulting in correlation between the questions. Degree is low.
  • the purpose of this application is to provide a method for automatically generating questions, by selecting the closest historical question and answer from the historical question and answer according to the user’s question and answer to the current question, to determine the actual associated scene, and download the closest historical question and answer
  • One sentence is used as a candidate for the next question, and the final next question is determined based on the similarity between the candidate question and the next question in the question bank. Due to the combination of similar historical question and answer, the actual scene is determined, and the next question in the actual scene is used to ask questions, which will inevitably improve the correlation between the next question and the current question.
  • an automatic question generation method provided in Example 1 of this application includes:
  • the content of the question and answer for the current examination question is the question and answer of the current examination question.
  • the question and the answer can be associated and stored as a question to ⁇ Q, A>, where Q represents the question, and A represents the corresponding answer.
  • the Q&A content of historical test questions can be obtained by collecting data in actual scenarios.
  • the Q&A data of users and customer service on the e-commerce platform is collected as the Q&A content of historical test questions.
  • the question and answer content of each historical examination question can also be stored in association with one question to ⁇ Q,A>.
  • the actual scene associated with the current problem is determined. Selecting the next question in the actual scene as a candidate means that the question is selected based on the actual scene.
  • the candidate test question is used as a target test question for questioning.
  • the two steps S14 and S15 take into account that the actual scene must not deviate too much from the test questions in the question bank. Therefore, the candidate test questions selected by the calculation are matched with the next question in the question bank.
  • the matching degree meets the requirements, we think that the candidate test questions are also It satisfies the basic requirements for questioning, and is more relevant to the current exam questions than the next question in the question bank, so that the candidate exam questions are used as the target exam questions.
  • the method further includes:
  • step S12 When calculating the first matching degree in step S12, it can be performed in the following manner:
  • the first degree of matching is calculated according to the degree of matching between the question and the answer. Specifically, the two can be added together to calculate the first matching degree.
  • the above-mentioned encoding of the content of the question and answer can be performed by separately inputting the content into the question and answer model, and the vector sequence is obtained by encoding, which can be realized by using the existing question and answer model.
  • the relationship between the question matching degree of each historical examination question, the answer matching degree and their respective thresholds can be judged separately, and the question and answer content of all historical examination questions can be selected.
  • the historical test questions whose matching degree is greater than the first preset threshold and the answer matching degree is greater than the second preset threshold are used as candidate historical test questions;
  • the next question of the first history question with the highest matching degree among the candidate history questions is selected as the candidate question.
  • the part of the historical question and answer that deviates greatly from the current test question or answer can be eliminated, thereby further ensuring the overall relevance of the selected candidate test question and the current test question.
  • the user's answer can be scored, and the method further includes:
  • the fourth degree of matching between the answer of the current test question and the standard answer of the current test question is calculated.
  • the fourth matching degree can be calculated according to algorithms such as word segmentation comparison and keyword weight.
  • the final score combines the matching degree between the question and the answer and between the answer and the standard answer, which improves the accuracy of the score evaluation.
  • the above score can also be used as a feature value, combined with the difference in length between the user's answer and the standard answer, and the keyword matching degree of the standard answer, etc., to calculate the final score.
  • the corresponding feature score [9,5,8,0.5,2,1,0.38] is sent to the scoring model to get the final score -> 7.
  • Embodiment 2 of the present application provides an automatic question generation device. As shown in Fig. 5, the device includes:
  • the current examination question obtaining unit 51 obtains the question and answer content for the current examination question
  • the first matching degree calculation unit 52 is configured to calculate the first matching degree between the question and answer content of each historical test question and the question and answer content of the current test question;
  • the candidate test question unit 53 is used to select the next question of the first historical test question with the highest matching degree as the candidate test question;
  • the second matching degree calculation unit 54 is configured to calculate a second matching degree between the candidate test question and the next question in the question bank;
  • the question generating unit 55 is configured to use the candidate test question as a target test question when the second matching degree satisfies a preset condition.
  • the question generating unit 55 is further configured to use the next question in the question bank as a target test question when the second matching degree does not satisfy a preset condition.
  • the first matching degree calculation unit includes a first coding unit, a question matching degree calculation unit, an answer matching degree calculation unit, and a first matching degree calculation subunit;
  • the first coding unit is configured to code the question and answer content of the current test question according to a preset question and answer model to obtain the question code and answer code of the current test question;
  • the first coding unit is further configured to code the question and answer content of each historical test question according to a preset question and answer model to obtain the question code and answer code of each historical test question;
  • the question matching degree calculation unit is used to calculate the question matching degree between the question code of each historical test question and the question code of the current test question;
  • the answer matching degree calculation unit is used to calculate the answer matching degree between the answer code of each historical test question and the answer code of the current test question;
  • the first matching degree calculation subunit is configured to calculate the first matching degree according to the matching degree of the question and the degree of matching of the answer.
  • the candidate test question unit is specifically used to select all historical test questions in the question and answer content, the question matching degree is greater than the first preset threshold and the answer matching degree is greater than the second preset threshold of the historical test questions as the candidate historical test questions and select all The next question of the first history question with the highest matching degree among the candidate history questions is used as the candidate question.
  • the device further includes:
  • a second coding unit a third matching degree calculation unit, a fourth matching degree calculation unit, and an answer score calculation unit;
  • the second coding unit is configured to code the question and answer content of the current test question according to a preset question and answer model to obtain the question code and answer code of the current test question;
  • the third matching degree calculation unit is configured to calculate the third matching degree between the question code and the answer code of the current test question
  • the fourth matching degree calculation unit is configured to calculate the fourth matching degree between the answer of the current test question and the standard answer of the current test question according to a text similarity algorithm
  • the answer score calculation unit is configured to calculate the score of the answer to the current test question according to the third matching degree and the fourth matching degree.
  • Embodiment 3 of the present application provides a computer system, including:
  • One or more processors are One or more processors.
  • a memory associated with the one or more processors where the memory is used to store program instructions that, when read and executed by the one or more processors, execute the method described in Embodiment 1 .
  • FIG. 6 exemplarily shows the architecture of a computer system, which may specifically include a processor 1510, a video display adapter 1511, a disk drive 1512, an input/output interface 1513, a network interface 1514, and a memory 1520.
  • the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, and the memory 1520 may be communicatively connected through the communication bus 1530.
  • the processor 1510 may be implemented in a general-purpose CPU (Central Processing Unit, central processing unit), microprocessor, application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., for Perform relevant procedures to realize the technical solutions provided in this application.
  • a general-purpose CPU Central Processing Unit, central processing unit
  • microprocessor microprocessor
  • application specific integrated circuit Application Specific Integrated Circuit, ASIC
  • integrated circuits etc.
  • the memory 1520 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, etc.
  • the memory 1520 may store an operating system 1521 for controlling the operation of the computer system 1500, and a basic input output system (BIOS) for controlling the low-level operation of the computer system 1500.
  • BIOS basic input output system
  • a web browser 1523, a data storage management system 1524, and an icon font processing system 1525 can also be stored.
  • the foregoing icon font processing system 1525 may be an application program that specifically implements the foregoing steps in the embodiment of the present application.
  • the related program code is stored in the memory 1520 and is called and executed by the processor 1510.
  • the input/output interface 1513 is used to connect input/output modules to realize information input and output.
  • the input/output/module can be configured in the device as a component (not shown in the figure), or can be connected to the device to provide corresponding functions.
  • the input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and an output device may include a display, a speaker, a vibrator, an indicator light, and the like.
  • the network interface 1514 is used to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices.
  • the communication module can realize communication through wired means (such as USB, network cable, etc.), or through wireless means (such as mobile network, WIFI, Bluetooth, etc.).
  • the bus 1530 includes a path to transmit information between various components of the device (for example, the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, and the memory 1520).
  • various components of the device for example, the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, and the memory 1520.
  • the computer system 1500 can also obtain information about specific receiving conditions from the virtual resource object receiving condition information database 1541 for condition judgment, and so on.
  • the above device only shows the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, the memory 1520, the bus 1530, etc., in the specific implementation process, the The device may also include other components necessary for normal operation.
  • the above-mentioned device may also include only the components necessary to realize the solution of the present application, and not necessarily include all the components shown in the figure.

Abstract

L'invention concerne un procédé, un appareil et un système de préparation automatique de questions. Le procédé comprend les étapes consistant à : acquérir du contenu de question et de réponse d'une question d'examen courante (S11) ; calculer un premier degré de concordance entre le contenu de question et de réponse de chaque question d'examen historique et le contenu de question et de réponse de la question d'examen courante (S12) ; sélectionner la question suivante de la question d'examen historique ayant le plus haut premier degré de concordance comme question d'examen candidate (S13) ; calculer un second degré de concordance entre la question d'examen candidate et la question suivante dans une bibliothèque de questions (S14) ; et si le second degré de concordance satisfait à une condition prédéfinie, prendre la question d'examen candidate comme question d'examen cible pour la préparation de questions (S15). Selon le procédé, l'appareil et le système, un scénario réel peut être simulé afin de préparer automatiquement des questions.
PCT/CN2020/111954 2019-12-30 2020-08-28 Procédé, appareil et système de préparation automatique de questions WO2021135322A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CA3166342A CA3166342A1 (fr) 2019-12-30 2020-08-28 Procede, appareil et systeme de preparation automatique de questions

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911400187.3 2019-12-30
CN201911400187.3A CN111159379B (zh) 2019-12-30 2019-12-30 一种自动出题方法、装置及系统

Publications (1)

Publication Number Publication Date
WO2021135322A1 true WO2021135322A1 (fr) 2021-07-08

Family

ID=70559482

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/111954 WO2021135322A1 (fr) 2019-12-30 2020-08-28 Procédé, appareil et système de préparation automatique de questions

Country Status (3)

Country Link
CN (1) CN111159379B (fr)
CA (1) CA3166342A1 (fr)
WO (1) WO2021135322A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111159379B (zh) * 2019-12-30 2022-12-20 苏宁云计算有限公司 一种自动出题方法、装置及系统
CN113761832A (zh) * 2021-08-05 2021-12-07 联想(北京)有限公司 一种信息处理方法、设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090228264A1 (en) * 2003-02-11 2009-09-10 Microsoft Corporation Management of conversations
CN104321765A (zh) * 2012-06-29 2015-01-28 国际商业机器公司 对专家会话构建器的扩展
CN109255697A (zh) * 2018-08-15 2019-01-22 普信恒业科技发展（北京）有限公司 一种基于人工智能的自动信用评估方法和系统
CN109977208A (zh) * 2019-03-22 2019-07-05 北京中科汇联科技股份有限公司 一种融合faq和任务及主动引导的对话系统
CN110110066A (zh) * 2019-05-09 2019-08-09 腾讯科技（深圳）有限公司 一种交互数据处理方法、装置以及计算机可读存储介质
CN111159379A (zh) * 2019-12-30 2020-05-15 苏宁云计算有限公司 一种自动出题方法、装置及系统

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107220380A (zh) * 2017-06-27 2017-09-29 北京百度网讯科技有限公司 基于人工智能的问答推荐方法、装置和计算机设备
CN109697228A (zh) * 2018-12-13 2019-04-30 平安科技（深圳）有限公司 智能问答方法、装置、计算机设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090228264A1 (en) * 2003-02-11 2009-09-10 Microsoft Corporation Management of conversations
CN104321765A (zh) * 2012-06-29 2015-01-28 国际商业机器公司 对专家会话构建器的扩展
CN109255697A (zh) * 2018-08-15 2019-01-22 普信恒业科技发展（北京）有限公司 一种基于人工智能的自动信用评估方法和系统
CN109977208A (zh) * 2019-03-22 2019-07-05 北京中科汇联科技股份有限公司 一种融合faq和任务及主动引导的对话系统
CN110110066A (zh) * 2019-05-09 2019-08-09 腾讯科技（深圳）有限公司 一种交互数据处理方法、装置以及计算机可读存储介质
CN111159379A (zh) * 2019-12-30 2020-05-15 苏宁云计算有限公司 一种自动出题方法、装置及系统

Also Published As

Publication number Publication date
CN111159379A (zh) 2020-05-15
CA3166342A1 (fr) 2021-07-08
CN111159379B (zh) 2022-12-20

Similar Documents

Publication Publication Date Title
CN106202453B (zh) 一种多媒体资源推荐方法和装置
TWI718422B (zh) 對模型預測值進行融合的方法、裝置和設備
CN110992169A (zh) 一种风险评估方法、装置、服务器及存储介质
US9734239B2 (en) Prompting subject matter experts for additional detail based on historical answer ratings
CN108322317B (zh) 一种账号识别关联方法及服务器
CN106874253A (zh) 识别敏感信息的方法及装置
US20210311751A1 (en) Machine-learning models applied to interaction data for determining interaction goals and facilitating experience-based modifications to interface elements in online environments
CN109345417B (zh) 基于身份认证的业务人员的在线考核方法及终端设备
US20220138770A1 (en) Method and apparatus for analyzing sales conversation based on voice recognition
WO2021135322A1 (fr) Procédé, appareil et système de préparation automatique de questions
CN111898643A (zh) 一种语义匹配方法及装置
CN113392197B (zh) 问答推理方法、装置、存储介质及电子设备
US20230013796A1 (en) Method and apparatus for acquiring pre-trained model, electronic device and storage medium
CN117272011A (zh) 模型测评方法、装置、电子设备和存储介质
CN114356747A (zh) 显示内容的测试方法、装置、设备、存储介质及程序产品
CN106790171A (zh) 会话控制的方法、装置和计算机可读存储介质
CN108268602A (zh) 分析文本话题点的方法、装置、设备和计算机存储介质
CN111443973A (zh) 备注信息的填入方法、装置、设备及存储介质
CN110674632A (zh) 一种确定安全级别的方法及装置、存储介质和设备
WO2021098876A1 (fr) Procédé et appareil de questions-réponses basés sur un graphe de connaissances
CN113962766A (zh) 一种用于简单混合型隐私保护场景的商品推荐方法及系统
CN113822589A (zh) 智能面试方法、装置、设备及存储介质
JP2007128032A (ja) 電子学習監督方法とシステム
CN113837322B (zh) 课程的分类处理方法、装置、设备及介质
CN113190779B (zh) 网页评估方法和装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20908471

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 3166342

Country of ref document: CA

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20908471

Country of ref document: EP

Kind code of ref document: A1