WO2021135322A1

WO2021135322A1 - Automatic question setting method, apparatus and system

Info

Publication number: WO2021135322A1
Application number: PCT/CN2020/111954
Authority: WO
Inventors: 陈宜琳; 倪合强; 张兵兵; 徐垚; 梁诗雯
Original assignee: 苏宁云计算有限公司
Priority date: 2019-12-30
Filing date: 2020-08-28
Publication date: 2021-07-08
Also published as: CN111159379A; CA3166342A1; CN111159379B

Abstract

Provided are an automatic question setting method, apparatus and system. The method comprises: acquiring question and answer content of the current examination question (S11); calculating a first matching degree between the question and answer content of each historical examination question and the question and answer content of the current examination question (S12); selecting the next question of the historical examination question with the highest first matching degree to be a candidate examination question (S13); calculating a second matching degree between the candidate examination question and the next question in a question library (S14); and if the second matching degree satisfies a preset condition, taking the candidate examination question as a target examination question for question setting (S15). According to the method, apparatus and system, an actual scenario can be simulated in order to automatically set questions.

Description

Method, device and system for automatically generating questions

Technical field

This application relates to the field of question and answer questions, and in particular to a method, device and system for automatically generating questions.

Background technique

In many products and platforms, Q&A scenarios often appear. For example, the intelligent customer service of an e-commerce platform is a question and answer scenario with users. In order to improve the accuracy of the intelligent customer service response, it is necessary to simulate the user's question to evaluate the intelligent customer service's response, so as to facilitate subsequent targeted improvements.

In the prior art, a question bank is set up in advance, and the test questions are selected from the question bank in accordance with a certain rule, such as a sequence, to simulate the user asking questions to the intelligent customer service to obtain the answer of the intelligent customer service, so as to detect the accuracy of the intelligent customer service's answer.

However, in the prior art, the rule order for question bank is set, and no matter what the answer to the current question is, the next question will be selected from the question bank to continue questioning according to the established rules. This way of questioning is out of the actual Q&A scenario, and the questions before and after may not necessarily have relevance.

The same problem exists in other Q&A scenarios.

Summary of the invention

This application provides an automatic question generation method, device and system to solve the problem that the real scene cannot be simulated when simulating question and answer in the prior art, so that there is no relevance between the previous questions and the previous questions.

The first aspect of the present application provides a method for automatically generating questions, and the method includes:

Get the content of the question and answer for the current exam;

Calculate the first degree of match between the question and answer content of each historical test question and the question and answer content of the current test question;

Selecting the next question of the first historical question with the highest matching degree as the candidate question;

Calculating the second degree of matching between the candidate test question and the next question in the question bank;

If the second degree of matching satisfies a preset condition, the candidate test question is used as a target test question to be asked.

Preferably, the method further includes:

If the second matching degree does not meet the preset condition, the next question in the question bank is used as the target test question to be asked.

Preferably, the calculating the first degree of matching between the question and answer content of each historical test question and the question and answer content of the current test question includes:

Encode the question and answer content of the current test question according to the preset question and answer model to obtain the question code and answer code of the current test question;

Code the question and answer content of each historical test question according to the preset question and answer model, and obtain the question code and answer code of each historical test question;

Calculate the question matching degree between the question code of each historical test question and the question code of the current test question;

Calculate the degree of match between the answer code of each historical test question and the answer code of the current test question;

The first degree of matching is calculated according to the degree of matching between the question and the answer.

Preferably, the selecting the next question of the first historical question with the highest matching degree as the candidate question includes:

Select all historical test questions in which the question matching degree is greater than the first preset threshold and the answer matching degree is greater than the second preset threshold among the question and answer content of all historical test questions as candidate historical test questions;

The next question of the first history question with the highest matching degree among the candidate history questions is selected as the candidate question.

Preferably, the method further includes:

Calculate the third degree of matching between the question code of the current test question and the answer code;

According to the text similarity algorithm, calculate the fourth degree of matching between the answer of the current test question and the standard answer of the current test question;

Calculate the score of the answer to the current test question according to the third degree of matching and the fourth degree of matching.

A second aspect of the present application provides an automatic question generation device, the device including:

The current examination question acquisition unit, to obtain the content of the question and answer for the current examination question;

The first matching degree calculation unit is used to calculate the first matching degree between the question and answer content of each historical test question and the question and answer content of the current test question;

The candidate test question unit is used to select the next question of the first historical test question with the highest matching degree as the candidate test question;

The second matching degree calculation unit is used to calculate the second matching degree between the candidate test question and the next question in the question bank;

The question-making unit is configured to use the candidate test question as a target test question when the second matching degree satisfies a preset condition.

Preferably, the question generating unit is further configured to use the next question in the question bank as a target test question when the second matching degree does not satisfy a preset condition.

Preferably, the first matching degree calculation unit includes a first coding unit, a question matching degree calculation unit, an answer matching degree calculation unit, and a first matching degree calculation subunit;

The first coding unit is configured to code the question and answer content of the current test question according to a preset question and answer model to obtain the question code and answer code of the current test question;

The first coding unit is further configured to code the question and answer content of each historical test question according to a preset question and answer model to obtain the question code and answer code of each historical test question;

The question matching degree calculation unit is used to calculate the question matching degree between the question code of each historical test question and the question code of the current test question;

The answer matching degree calculation unit is used to calculate the answer matching degree between the answer code of each historical test question and the answer code of the current test question;

The first matching degree calculation subunit is configured to calculate the first matching degree according to the matching degree of the question and the degree of matching of the answer.

Preferably, the device further includes:

A second coding unit, a third matching degree calculation unit, a fourth matching degree calculation unit, and an answer score calculation unit;

The second coding unit is configured to code the question and answer content of the current test question according to a preset question and answer model to obtain the question code and answer code of the current test question;

The third matching degree calculation unit is used to calculate the third matching degree between the question code and the answer code of the current test question;

The fourth matching degree calculation unit is configured to calculate the fourth matching degree between the answer of the current test question and the standard answer of the current test question according to a text similarity algorithm;

The answer score calculation unit is configured to calculate the score of the answer to the current test question according to the third matching degree and the fourth matching degree.

The third aspect of this application provides a computer system, including:

One or more processors; and

A memory associated with the one or more processors, where the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, the method described above is executed.

According to the specific embodiments provided in this application, this application discloses the following technical effects:

The technical solution of this application selects the similar question and answer from the historical question and answer according to the current question and answer situation, and when the next sentence of the closest historical question and answer is highly similar to the next question in the question bank, select the next sentence of the historical question and answer As the next exam question. Realize that according to the user's answer to each question, automatically determine the relevant questions from the historical question and answer to continue to ask questions, build a scene-based simulation training, and improve the correlation between the test questions.

Further, in this application, the degree of matching between the comprehensive standard answer and the test taker’s answer, and the degree of matching between the test question and the test taker’s answer are used to score the test taker’s answers, so that the scoring of the test taker’s answer meets the real scenario response and can be close to the standard answer specifications.

Description of the drawings

In order to explain the embodiments of the present application or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings that need to be used in the embodiments. Obviously, the drawings in the following description are only some of the present application. Embodiments, for those of ordinary skill in the art, without creative work, other drawings can be obtained based on these drawings.

Figure 1 is the flow chart of the method of this application;

Figures 2-4 are schematic diagrams of specific embodiments of the present application;

Fig. 5 is a structural diagram of a device according to an embodiment of the present application;

Figure 6 is a structural diagram of a computer system.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art fall within the protection scope of this application.

As mentioned in the background art, the way of preparing questions from the question bank in a preset order ignores the correlation between the question and answer of the previous question and the next question, so that the next question is separated from the question and answer of the previous question, resulting in correlation between the questions. Degree is low.

The purpose of this application is to provide a method for automatically generating questions, by selecting the closest historical question and answer from the historical question and answer according to the user’s question and answer to the current question, to determine the actual associated scene, and download the closest historical question and answer One sentence is used as a candidate for the next question, and the final next question is determined based on the similarity between the candidate question and the next question in the question bank. Due to the combination of similar historical question and answer, the actual scene is determined, and the next question in the actual scene is used to ask questions, which will inevitably improve the correlation between the next question and the current question.

Example 1

As shown in Fig. 1, an automatic question generation method provided in Example 1 of this application includes:

S11. Obtain the question and answer content for the current examination question.

The content of the question and answer for the current examination question is the question and answer of the current examination question.

In order to determine the associated scenario, firstly, it is necessary to obtain the user's questions and answers for the current examination questions.

In a specific embodiment, the question and the answer can be associated and stored as a question to <Q, A>, where Q represents the question, and A represents the corresponding answer.

S12. Calculate the first matching degree between the question and answer content of each historical test question and the question and answer content of the current test question.

The Q&A content of historical test questions can be obtained by collecting data in actual scenarios. For example, the Q&A data of users and customer service on the e-commerce platform is collected as the Q&A content of historical test questions. The question and answer content of each historical examination question can also be stored in association with one question to <Q,A>.

S13. Select the next question of the first historical question with the highest matching degree as a candidate question.

Through this step, the actual scene associated with the current problem is determined. Selecting the next question in the actual scene as a candidate means that the question is selected based on the actual scene.

S14. Calculate the second degree of matching between the candidate test question and the next question in the question bank.

S15. If the second degree of matching satisfies a preset condition, the candidate test question is used as a target test question for questioning.

The two steps S14 and S15 take into account that the actual scene must not deviate too much from the test questions in the question bank. Therefore, the candidate test questions selected by the calculation are matched with the next question in the question bank. When the matching degree meets the requirements, we think that the candidate test questions are also It satisfies the basic requirements for questioning, and is more relevant to the current exam questions than the next question in the question bank, so that the candidate exam questions are used as the target exam questions.

As shown in Figure 1, the method further includes:

S16. If the second degree of matching does not meet the preset condition, the next question in the question bank is used as the target test question to be asked. This step ensures that the history test question that deviates from the question bank is taken as the next question.

When calculating the first matching degree in step S12, it can be performed in the following manner:

The first degree of matching is calculated according to the degree of matching between the question and the answer. Specifically, the two can be added together to calculate the first matching degree.

Wherein, the above-mentioned encoding of the content of the question and answer can be performed by separately inputting the content into the question and answer model, and the vector sequence is obtained by encoding, which can be realized by using the existing question and answer model.

In a preferred embodiment, when selecting the historical examination questions with the highest matching degree, the relationship between the question matching degree of each historical examination question, the answer matching degree and their respective thresholds can be judged separately, and the question and answer content of all historical examination questions can be selected. The historical test questions whose matching degree is greater than the first preset threshold and the answer matching degree is greater than the second preset threshold are used as candidate historical test questions;

Through the above method, the part of the historical question and answer that deviates greatly from the current test question or answer can be eliminated, thereby further ensuring the overall relevance of the selected candidate test question and the current test question.

In a preferred embodiment of the present application, the user's answer can be scored, and the method further includes:

According to the text similarity algorithm, the fourth degree of matching between the answer of the current test question and the standard answer of the current test question is calculated. Specifically, the fourth matching degree can be calculated according to algorithms such as word segmentation comparison and keyword weight.

Through the above method, the final score combines the matching degree between the question and the answer and between the answer and the standard answer, which improves the accuracy of the score evaluation.

In practice, the above score can also be used as a feature value, combined with the difference in length between the user's answer and the standard answer, and the keyword matching degree of the standard answer, etc., to calculate the final score.

The following is an example of a specific scenario of this application:

Exam question: I received a text message saying that my TV installation has been dispatched, and I don’t want to install it.

Standard answer: Hello, here you see that you have a Shenzhen Samsung TV installation. The scheduled installation time is 5.07. Has your family ever made an appointment?

Candidate’s answer: I’ll check it for you here. Did you buy this Samsung TV? QA82Q900RBJXXZ

According to the above scenarios, automatic scoring and automatic questioning are carried out:

Score the answers of the above candidates:

First, use the model shown in Figure 2 to score the test questions and standard answers for the first time, and get a score of 1->9

Then use the model shown in Figure 3 to score the standard answer and the test taker’s answer for the second time, and get a score of 2->5

Then use the model shown in Figure 4 to score the above two scores with other feature values.

Such as: [score 1, score 2, length difference, keyword matching rate, sentence count difference, short sentence sentence count difference, jaccard similarity]

For example, the corresponding feature score [9,5,8,0.5,2,1,0.38] is sent to the scoring model to get the final score -> 7.

Automatic question:

Similar scene retrieval:

Calculate the matching degree of the historical question and answer in the historical question and answer database, and select and determine the most matching historical question and answer as follows:

User: Why do you say that my TV installation has been dispatched? I don't want to install it.

Customer Service: I have checked it for you, is this a Samsung TV?

Then the next sentence in the scene:

User: Yes, can you help me see how to cancel

Compare with the next standard question set in the question bank. If the similarity is greater than the set threshold, the next sentence in the above scenario will be used to make the test questions more varied. If it is lower than the threshold, the next standard in the question bank will be used. Exam questions are set to prevent candidates from partial questions.

Example 2

In response to the above method, Embodiment 2 of the present application provides an automatic question generation device. As shown in Fig. 5, the device includes:

The current examination question obtaining unit 51 obtains the question and answer content for the current examination question;

The first matching degree calculation unit 52 is configured to calculate the first matching degree between the question and answer content of each historical test question and the question and answer content of the current test question;

The candidate test question unit 53 is used to select the next question of the first historical test question with the highest matching degree as the candidate test question;

The second matching degree calculation unit 54 is configured to calculate a second matching degree between the candidate test question and the next question in the question bank;

The question generating unit 55 is configured to use the candidate test question as a target test question when the second matching degree satisfies a preset condition.

Preferably, the question generating unit 55 is further configured to use the next question in the question bank as a target test question when the second matching degree does not satisfy a preset condition.

The candidate test question unit is specifically used to select all historical test questions in the question and answer content, the question matching degree is greater than the first preset threshold and the answer matching degree is greater than the second preset threshold of the historical test questions as the candidate historical test questions and select all The next question of the first history question with the highest matching degree among the candidate history questions is used as the candidate question.

Preferably, the device further includes:

The third matching degree calculation unit is configured to calculate the third matching degree between the question code and the answer code of the current test question;

Example 3

Corresponding to the foregoing method and device, Embodiment 3 of the present application provides a computer system, including:

One or more processors; and

A memory associated with the one or more processors, where the memory is used to store program instructions that, when read and executed by the one or more processors, execute the method described in Embodiment 1 .

Among them, FIG. 6 exemplarily shows the architecture of a computer system, which may specifically include a processor 1510, a video display adapter 1511, a disk drive 1512, an input/output interface 1513, a network interface 1514, and a memory 1520. The processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, and the memory 1520 may be communicatively connected through the communication bus 1530.

Among them, the processor 1510 may be implemented in a general-purpose CPU (Central Processing Unit, central processing unit), microprocessor, application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., for Perform relevant procedures to realize the technical solutions provided in this application.

The memory 1520 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, etc. The memory 1520 may store an operating system 1521 for controlling the operation of the computer system 1500, and a basic input output system (BIOS) for controlling the low-level operation of the computer system 1500. In addition, a web browser 1523, a data storage management system 1524, and an icon font processing system 1525 can also be stored. The foregoing icon font processing system 1525 may be an application program that specifically implements the foregoing steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented through software or firmware, the related program code is stored in the memory 1520 and is called and executed by the processor 1510.

The input/output interface 1513 is used to connect input/output modules to realize information input and output. The input/output/module can be configured in the device as a component (not shown in the figure), or can be connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and an output device may include a display, a speaker, a vibrator, an indicator light, and the like.

The network interface 1514 is used to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.), or through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

The bus 1530 includes a path to transmit information between various components of the device (for example, the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, and the memory 1520).

In addition, the computer system 1500 can also obtain information about specific receiving conditions from the virtual resource object receiving condition information database 1541 for condition judgment, and so on.

It should be noted that although the above device only shows the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, the memory 1520, the bus 1530, etc., in the specific implementation process, the The device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above-mentioned device may also include only the components necessary to realize the solution of the present application, and not necessarily include all the components shown in the figure.

From the description of the foregoing implementation manners, it can be known that those skilled in the art can clearly understand that this application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM/RAM, magnetic disk , CD-ROM, etc., including a number of instructions to enable a computer device (which may be a personal computer, a cloud server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments of the present application.

The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for the system or the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiment. The system and system embodiments described above are merely illustrative, where the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, It can be located in one place, or it can be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments. Those of ordinary skill in the art can understand and implement without creative work.

The method, device, and system for automatically generating questions provided by this application are described in detail above. Specific examples are used in this article to explain the principles and implementation of this application. The description of the above examples is only used to help understand this application. The method and its core idea; at the same time, for ordinary technicians in the field, according to the idea of this application, there will be changes in the specific implementation and the scope of application. In summary, the content of this specification should not be construed as a limitation on this application.

Claims

An automatic question generation method, characterized in that the method includes:

Get the content of the question and answer for the current exam;

Calculate the first degree of match between the question and answer content of each historical test question and the question and answer content of the current test question;

Selecting the next question of the first historical question with the highest matching degree as the candidate question;

Calculating the second degree of matching between the candidate test question and the next question in the question bank;

If the second degree of matching satisfies a preset condition, the candidate test question is used as a target test question to be asked.
The method of claim 1, wherein the method further comprises:

If the second matching degree does not meet the preset condition, the next question in the question bank is used as the target test question to be asked.
The method of claim 1, wherein the calculating the first degree of matching between the question and answer content of each historical question and the question and answer content of the current question comprises:

Encode the question and answer content of the current test question according to the preset question and answer model to obtain the question code and answer code of the current test question;

Code the question and answer content of each historical test question according to the preset question and answer model, and obtain the question code and answer code of each historical test question;

Calculate the question matching degree between the question code of each historical test question and the question code of the current test question;

Calculate the degree of match between the answer code of each historical test question and the answer code of the current test question;

The first degree of matching is calculated according to the degree of matching between the question and the answer.
The method according to claim 3, wherein the selecting the next question of the first historical question with the highest matching degree as a candidate question comprises:

Select all historical test questions in which the question matching degree is greater than the first preset threshold and the answer matching degree is greater than the second preset threshold among the question and answer content of all historical test questions as candidate historical test questions;

The next question of the first history question with the highest matching degree among the candidate history questions is selected as the candidate question.
The method according to any one of claims 1-4, wherein the method further comprises:

Encode the question and answer content of the current test question according to the preset question and answer model to obtain the question code and answer code of the current test question;

Calculate the third degree of matching between the question code of the current test question and the answer code;

According to the text similarity algorithm, calculate the fourth degree of matching between the answer of the current test question and the standard answer of the current test question;

Calculate the score of the answer to the current test question according to the third degree of matching and the fourth degree of matching.
An automatic question-making device, characterized in that the device comprises:

The current examination question acquisition unit, to obtain the content of the question and answer for the current examination question;

The first matching degree calculation unit is used to calculate the first matching degree between the question and answer content of each historical test question and the question and answer content of the current test question;

The candidate test question unit is used to select the next question of the first historical test question with the highest matching degree as the candidate test question;

The second matching degree calculation unit is used to calculate the second matching degree between the candidate test question and the next question in the question bank;

The question-making unit is configured to use the candidate test question as a target test question when the second matching degree satisfies a preset condition.
The device according to claim 6, wherein the question generating unit is further configured to use the next question in the question bank as a target test question when the second matching degree does not meet a preset condition .
7. The device according to claim 6, wherein the first matching degree calculation unit comprises a first coding unit, a question matching degree calculation unit, an answer matching degree calculation unit, and a first matching degree calculation subunit;

The first coding unit is configured to code the question and answer content of the current test question according to a preset question and answer model to obtain the question code and answer code of the current test question;

The first coding unit is further configured to code the question and answer content of each historical test question according to a preset question and answer model to obtain the question code and answer code of each historical test question;

The question matching degree calculation unit is used to calculate the question matching degree between the question code of each historical test question and the question code of the current test question;

The answer matching degree calculation unit is used to calculate the answer matching degree between the answer code of each historical test question and the answer code of the current test question;

The first matching degree calculation subunit is configured to calculate the first matching degree according to the matching degree of the question and the degree of matching of the answer.
The device according to claims 6-8, wherein the device further comprises:

A second coding unit, a third matching degree calculation unit, a fourth matching degree calculation unit, and an answer score calculation unit;

The second coding unit is configured to code the question and answer content of the current test question according to a preset question and answer model to obtain the question code and answer code of the current test question;

The third matching degree calculation unit is configured to calculate the third matching degree between the question code and the answer code of the current test question;

The fourth matching degree calculation unit is configured to calculate the fourth matching degree between the answer of the current test question and the standard answer of the current test question according to a text similarity algorithm;

The answer score calculation unit is configured to calculate the score of the answer to the current test question according to the third matching degree and the fourth matching degree.
A computer system, characterized in that it comprises:

One or more processors; and

A memory associated with the one or more processors, where the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, execute any one of claims 1-5 The operation described in