KR20230084635A - System based on natural language process for constructing questionnaire and verifying the reliabiliy of answer and method thereof - Google Patents

Abstract

Disclosed are a system for constructing questionnaires and verifying answer reliability based on natural language processing and a method thereof. The system comprises: a survey execution module for executing a survey online for a survey respondent terminal; an AI construction module for automatically collecting survey items, evaluating each of the survey items by analyzing a survey answer and the corresponding survey items in accordance with the survey executed by the survey execution module using the automatically collected survey items, and constructing a survey database on the basis of an evaluation result; a survey item user suggestion module for creating the survey items in accordance with an input of a user, comparing the created survey items with the survey items stored in the survey database, and selecting and suggesting the survey items; a final survey generation module for generating a final survey with respect to the survey items suggested by the survey item user suggestion module, and updating the survey database; and a survey answer reliability AI verification module for evaluating the reliability of the corresponding survey answer by comparing and verifying the survey answer of the same survey respondent terminal for the survey executed by the survey execution module. According to the present invention, survey items are analyzed and evaluated based on natural language processing to construct objective questionnaires, thereby preventing a survey respondent from being induced to a specific answer, or subjective thoughts of a survey creator from being implied.

Description

A system for constructing a collection of questions based on natural language processing and verifying the reliability of answers and its method

The present invention relates to an online survey, and more particularly, to a system and method for constructing a collection of survey items and verifying reliability of answers based on natural language processing.

Surveys are conducted face-to-face or non-face-to-face for statistical analysis.

Since the objectivity of a survey must be secured according to its purpose, the subjective thoughts of the creator of the survey question itself must not be inherent in the survey question itself, and a survey question question that induces an answer to the survey question must not be created. In addition, the use of phrases that cause confusion about the survey should be refrained from.

Subjective thoughts, induction of answers, use of phrases that cause confusion, etc. are incentives to lower the quality of the survey itself and reduce its reliability.

On the other hand, if a survey respondent is confused about a survey question with low reliability, he or she may worry about how reliable the survey answer is.

Therefore, it is necessary to create questionnaires for more objective surveys, and also to evaluate whether the answers of survey respondents also have reliability based on regularity and sincerity.

Registered Patent Publication 10-1997992 Publication No. 10-2016-0000487

An object of the present invention is to provide a system for constructing a collection of questions based on natural language processing and verifying reliability of answer answers.

Another object of the present invention is to provide a method for constructing a collection of questions based on natural language processing and verifying reliability of answers.

A system for constructing a collection of survey items and verifying answer reliability based on natural language processing according to the object of the present invention described above includes a survey execution module that executes a survey online for a survey respondent terminal; Survey questions are automatically collected, and each survey question is evaluated by analyzing the survey answer according to the survey executed by the survey execution module and the corresponding survey question using the automatically collected survey questions, and the survey database is based on the evaluation results. It can be configured to include a survey questionnaire AI building module that builds.

Here, a questionnaire item user suggestion module that creates a questionnaire item according to a user's input, compares the created questionnaire item and the questionnaire item stored in the questionnaire database, and selects and proposes the questionnaire item; It may be configured to further include a final survey generation module for generating a final survey based on the survey items suggested by the survey item user suggestion module and updating the survey database.

The survey execution module may be configured to execute the final survey generated by the final survey generation module online for the survey responder terminal.

And it may be configured to further include a survey answer reliability AI verification module that prepares and verifies the survey answers of the same survey responder terminal for the survey executed in the survey execution module and evaluates the reliability of the corresponding survey answer.

Meanwhile, the questionnaire database stores the subject, category, questionnaire item, item classification code, questionnaire answer answer, and attribute of the questionnaire, the subject is a criterion for classifying the questionnaire, and the category is the questionnaire item of the questionnaire. It is a criterion for classifying a collection, and the item classification code may be a code for classifying by associating at least two or more questionnaire items requiring the same questionnaire answer.

According to another object of the present invention described above, a method for constructing a collection of questions based on natural language processing and verifying reliability of answer answers includes: executing, by a survey execution module, a survey online for a survey respondent terminal; Automatically collecting survey questions by the survey question collection AI building module; Evaluating each questionnaire item by analyzing, by the questionnaire collection AI construction module, a questionnaire answer according to a questionnaire executed by the questionnaire execution module and the corresponding questionnaire item using the automatically collected questionnaire items; The questionnaire item collection AI building module may be configured to include a step of constructing a questionnaire database based on the evaluation result.

Here, a survey item user suggestion module prepares a survey item according to a user's input, compares the created questionnaire item with the questionnaire item stored in the questionnaire database, selects the questionnaire item, and proposes the questionnaire item; The final survey generation module may further include generating a final survey based on the survey items suggested by the survey item user suggestion module and updating the survey database.

The method may further include executing, by the survey execution module, the final survey generated by the final survey generation module online for the survey responder terminal.

In addition, the survey answer reliability AI verification module may be configured to further include evaluating the reliability of the corresponding survey answer by preparing and verifying the survey answer of the same survey responder terminal for the survey executed in the survey execution module.

According to the above-described natural language processing-based survey item collection and answer reliability verification system and method, it is configured to construct an objective survey item collection by analyzing and evaluating survey items based on natural language processing, thereby inducing a specific answer to a survey respondent or It has the effect of preventing the author's subjective thoughts from being implied.

In addition, the responses of the same survey respondent to similar survey questions are collected and analyzed to evaluate the reliability of the answers to each survey question, so that it can be determined whether the corresponding survey respondent answered falsely for each survey answer. It works.

1 is a block diagram of a system for constructing a questionnaire collection and verifying answer reliability based on natural language processing according to an embodiment of the present invention.
2 is a flowchart of a method for constructing a questionnaire collection and verifying the reliability of an answer based on natural language processing according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in specific contents for practicing the invention. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numbers have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a system for constructing a questionnaire collection and verifying answer reliability based on natural language processing according to an embodiment of the present invention.

Referring to FIG. 1, the system 100 for constructing a collection of questions based on natural language processing and verifying reliability of answers based on natural language processing according to an embodiment of the present invention includes a survey execution module 110, an AI survey collection building module 120, and user suggestions for survey questions. It may be configured to include a module 130 , a final questionnaire generation module 140 , and a questionnaire answer reliability AI evaluation module 150 .

Hereinafter, a detailed configuration will be described.

The survey execution module 110 may be configured to execute the survey online for the survey responder terminal 10 .

The AI questionnaire collection building module 120 may be configured to automatically collect questionnaire items.

In addition, the AI questionnaire collection building module 120 may be configured to evaluate each questionnaire item by analyzing the questionnaire answer according to the questionnaire executed by the questionnaire execution module 110 and the corresponding questionnaire item using the automatically collected questionnaire items. . And it can be configured to build a questionnaire database 123 based on the evaluation result.

In the questionnaire database 123, evaluation scores are assigned to the questionnaire items themselves, and high scores may be given to questionnaire items that are objective, do not induce a specific answer, and do not confuse the respondents.

The AI questionnaire collection building module 120 may be configured to include an automatic questionnaire collection unit 121, a first questionnaire item attribute extraction unit 122, a questionnaire database 123, and a questionnaire item analysis/evaluation unit 124. .

Hereinafter, a detailed configuration will be described.

The survey item collection unit 121 may be configured to collect surveys of existing surveys or newly create and collect surveys.

The first questionnaire item property extractor 122 may be configured to extract the properties of the questionnaire items among the items collected by the questionnaire collection unit 121 based on natural language processing. Attribute extraction of survey items may be performed by extracting keywords based on a dictionary while removing stopwords.

The questionnaire database 123 may be configured to store the questionnaire collected by the questionnaire collection unit 121 and the properties extracted from the first questionnaire item property extractor 122 .

At this time, the questionnaire database 123 may be configured to store the subject, category, questionnaire item, item classification code, questionnaire answer and attributes of the questionnaire. The attributes extracted based on natural language processing can be used to set the subject, category, etc. of the survey for each survey item.

Here, the subject is a criterion for classifying the questionnaire, and may be, for example, a customer satisfaction survey or an environmental pollution survey.

The category is a criterion for classifying a collection of questionnaire items in a survey, and may be, for example, a personal information input item, a cognitive level evaluation item, a future development possibility evaluation item, and the like.

Also, the item classification code is a code for classifying by associating at least two or more survey items requiring the same survey answer. Survey items with the same item classification code may be written in different formats even if they require the same survey answer. For example, questionnaire items asking whether you know the global warming situation, such as "Do you know the global warming situation?" and "You must know a little about global warming?" The same question classification code may be set for all of these survey questions.

The survey item analysis/evaluation unit 124 may be configured to evaluate each survey item by analyzing the survey answer of the survey responder terminal 10 and the corresponding survey item in the survey executed by the survey execution module 110 .

The questionnaire analysis/evaluation unit 124 collects questionnaire items having the same item classification code, analyzes the questionnaire answers, and evaluates whether each questionnaire item is good or bad according to the distribution of the questionnaire answers. can be configured.

For example, if the answers to questions A, B, C, D, and E with the same question classification code are 1, 1, 1, 1, and 2, respectively, most of the survey questions requiring the same answer are 1. While it came out as a questionnaire answer, Question E has a questionnaire answer of 2.

In this case, questionnaire items A, B, C, and D can be evaluated as good questionnaire items that are relatively objective and do not induce a specific answer, and questionnaire item E, which is not, induces a specific questionnaire answer or the subjective thoughts of the survey writer are involved. It can be evaluated as a question item that is likely to be unfavorable.

That is, survey questions A, B, C, and D may be evaluated as good questions (1 point), and survey question E may be evaluated as bad questions (0 points). Assuming that the distribution of all survey responses is the same, each survey question can be evaluated as a normal question (0.5 points).

The questionnaire item analysis/evaluation unit 124 may be configured to additionally store evaluation results in the questionnaire database 123 .

The survey item user suggestion module 130 may be configured to create a survey item according to a user's input, compare the created questionnaire item and the questionnaire item stored in the questionnaire database 123, select the questionnaire item, and suggest the questionnaire item.

The questionnaire item user suggestion module 130 may include a questionnaire item preparation unit 131 , a second questionnaire item attribute extraction unit 132 , and a questionnaire item proposal unit 133 .

Hereinafter, a detailed configuration will be described.

The questionnaire preparation unit 131 may be configured to prepare questionnaire items according to a user's input. That is, before conducting a survey (survey), a user may write desired survey items.

The second questionnaire item property extraction unit 132 may be configured to extract the properties of the questionnaire item prepared by the questionnaire item preparation unit 131 based on natural language processing. Attribute extraction of survey items may be performed by extracting keywords based on a dictionary while removing stopwords.

The questionnaire item suggesting unit 133 may be configured to search the questionnaire database 123 for a similar questionnaire item to the questionnaire item prepared by the questionnaire item preparing unit 131 by performing a similarity analysis based on the extracted attributes.

Here, the questionnaire item suggesting unit 133 may be configured to analyze similarity between sentences corresponding to the subject and category of the same questionnaire as the questionnaire item prepared by the questionnaire item preparation unit 131 . That is, questionnaire items of the questionnaire database 123 similar to those created by the user may be compared with each other.

At this time, the similarity analysis between the survey items prepared by the user and the survey items in the survey database 123 may be performed using the cosine similarity analysis method of Equation 1 below.

Here, A and B may be survey questions to be compared.

When the evaluation result of the searched similar questionnaire item is higher than a predetermined standard, the questionnaire item suggesting unit 133 proposes the questionnaire item prepared by the questionnaire item preparation unit 131, and the evaluation result of the searched similar questionnaire item is equal to or lower than the predetermined standard. If it is low, it may be configured to propose a survey item having the highest evaluation result among the questionnaire items of the corresponding item classification code.

That is, if similar survey items obtained as a comparison result have low evaluation scores, it is possible to select survey items having high evaluation scores from among questionnaire items having low evaluation scores and items having the same item classification code.

The final questionnaire generation module 140 may be configured to generate a final questionnaire based on the questionnaire items suggested by the questionnaire item user suggestion module 130 and update the questionnaire database 123 . The final survey is the final version of the survey the user intended to do in the past, and is a survey selected by referring to the surveys in the survey database 123 as described above. The user may conduct a survey using the final survey.

Here, the survey execution module 110 may be configured to execute the final survey generated by the final survey generation module 140 with respect to the survey responder terminal 10 online.

From a different point of view, while the objectivity or excellence of the survey questions themselves can be evaluated as above, the authenticity of the survey respondents or the reliability of the survey answers themselves, which will be described later, also needs to be evaluated.

The survey answer reliability AI evaluation module 150 may be configured to compare and verify survey answers of the same survey responder terminal 10 for the survey executed in the survey execution module 110 and evaluate the reliability of the corresponding survey answer.

The questionnaire answer reliability AI evaluation module 150 may include a questionnaire similarity analysis unit 151 , an AI verification questionnaire generator 152 , and a reliability evaluation unit 153 .

Hereinafter, a detailed configuration will be described.

The questionnaire similarity analyzer 151 may be configured to compare questionnaires of the same questionnaire respondent with each other and analyze the similarity of the questionnaires. Here, the similarity of the questionnaire may be determined by the similarity of the questionnaire in which the questionnaire respondent prepared the questionnaire answer.

The questionnaire similarity analysis unit 151 may be configured to extract the properties of the questionnaire and analyze the similarity using a term frequency-inverse document frequency (TF-IDF) algorithm. The degree of similarity can be obtained according to Equation 2 below.

와

는 각각 수학식 3과 수학식 4를 통해 구해질 수 있다.here,

and

Can be obtained through Equations 3 and 4, respectively.

In Equations 2, 3 and 4, t is a term, w is a total term, d is a document, and D is a total document.

The AI verification questionnaire generation unit 152 may be configured to prepare questionnaire answers of questionnaire items having the same item classification code in the questionnaire analyzed by the questionnaire similarity analysis unit 151 to have a similarity of a questionnaire higher than a predetermined standard.

The AI verification questionnaire generating unit 152 may be configured to generate verification items for the corresponding questionnaire items when the above comparison result questionnaire answers differ from each other.

The AI verification questionnaire generation unit 152 may be configured to select and generate a questionnaire item that is not used in the questionnaire and has the highest evaluation result as a verification item among the questionnaire items stored in the questionnaire database 123 .

The reliability evaluation unit 153 performs verification on the corresponding survey responder terminal 10 using the verification questions generated by the AI verification questionnaire generator 152, and the reliability of the existing corresponding questionnaire answer according to the execution result. It can be configured to evaluate.

The reliability evaluation unit 153 may evaluate reliability according to Equation 5 below.

For example, if there are 5 multiple-choice survey answers, if the previously selected survey answer was 3, but 1 was selected in the validation question, (1-(5-2/5))*100 = 40% confidence. do. That is, reliability can be evaluated by the distance between the existing survey answer and the answer reselected by the verification question.

If the correct answer is selected, the reliability can come out as 0% or 100%.

In the case of short-answer questionnaire answers, the evaluation is based on the similarity between texts, and the similarity can be calculated based on the cosine similarity.

2 is a flowchart of a method for constructing a questionnaire collection and verifying the reliability of an answer based on natural language processing according to an embodiment of the present invention.

Referring to FIG. 2 , the survey execution module 110 executes the survey online with respect to the survey responder terminal 10 (S101).

Next, the survey question collection AI construction module 120 automatically collects survey questions (S102).

Next, the survey question collection AI building module 120 analyzes the survey answers according to the survey executed by the survey execution module 110 using the automatically collected survey questions and evaluates each survey question (S103). ).

Next, the questionnaire item collection AI building module 120 builds the questionnaire database 123 based on the evaluation result (S104).

Next, the questionnaire user suggestion module 130 prepares a questionnaire according to the user's input, compares the created questionnaire with the questionnaire stored in the questionnaire database 123, selects and proposes the questionnaire (S105).

Next, the final survey generation module 140 creates a final survey based on the survey items suggested by the survey item user suggestion module 130 and updates the survey database 123 (S106).

Next, the survey execution module 110 executes the final survey generated by the final survey generation module 140 online with respect to the survey responder terminal 10 (S107).

Next, the survey answer reliability AI verification module 150 prepares and verifies the survey answer of the same survey responder terminal 10 for the survey executed in the survey execution module 110 and evaluates the reliability of the corresponding survey answer (S108). ).

Although described with reference to the above embodiments, those skilled in the art will understand that the present invention can be variously modified and changed without departing from the spirit and scope of the present invention described in the claims below. You will be able to.

110: Survey execution module
120: AI questionnaire collection building module
121: automatic survey collection unit
122: first questionnaire item attribute extraction unit
123: survey database
124: Questionnaire analysis/evaluation unit
130: Questionnaire user suggestion module
131: Questionnaire preparation unit
132: second questionnaire item attribute extraction unit
133: Questionnaire proposal unit
140: final survey generation module
150: Answer reliability AI verification module
151: Survey similarity analysis unit
152: AI verification questionnaire generation unit
153: reliability evaluation unit

Claims (9)

a survey execution module that executes a survey online for a survey responder terminal;
Survey questions are automatically collected, and each survey question is evaluated by analyzing the survey answer according to the survey executed by the survey execution module and the corresponding survey question using the automatically collected survey questions, and the survey database is based on the evaluation results. A survey question proposal and answer reliability verification system based on a survey question collection including a survey question collection AI building module that builds a.
According to claim 1,
a survey item user suggestion module that prepares a survey item according to a user's input, compares the prepared questionnaire item with the questionnaire item stored in the survey database, selects the questionnaire item, and proposes it;
Questionnaire proposal and answer reliability based on the questionnaire collection, characterized in that it is configured to further include a final questionnaire generation module for generating a final questionnaire based on the questionnaire items suggested by the questionnaire item user suggestion module and updating the questionnaire database. verification system.
The method of claim 1, wherein the survey execution module,
A system for proposing a questionnaire item and verifying answer reliability based on a questionnaire item collection, characterized in that it is configured to execute the final questionnaire generated by the final questionnaire generating module online for the questionnaire respondent terminal.
According to claim 3,
Based on the questionnaire collection, characterized in that it is configured to further include a survey answer reliability AI verification module that prepares and verifies the survey answer of the same survey responder terminal for the survey executed in the survey execution module and evaluates the reliability of the survey answer. Questionnaire proposal and answer reliability verification system.
According to claim 4,
The survey database,
The subject of the survey, category, survey question, question classification code, survey answer and attributes are stored;
The topic is a criterion for classifying the questionnaire,
The category is a criterion for classifying a collection of survey items in the survey,
The question classification code is a code for classifying at least two or more survey questions requiring the same survey answer, wherein the survey item proposal and answer reliability verification system based on the questionnaire item collection is characterized in that.
executing, by a survey execution module, a survey on a survey responder terminal online;
Automatically collecting survey questions by the survey question collection AI building module;
Evaluating each questionnaire item by analyzing, by the questionnaire collection AI construction module, a questionnaire answer according to a questionnaire executed by the questionnaire execution module and the corresponding questionnaire item using the automatically collected questionnaire items;
A method for proposing a questionnaire item based on a questionnaire item collection and verifying reliability of an answer, comprising the step of constructing a questionnaire database based on the evaluation result by the AI building module of the questionnaire item set.
According to claim 6,
preparing a survey item according to a user's input by a survey item user suggestion module, comparing the created survey item with the questionnaire item stored in the survey database, and selecting and suggesting the questionnaire item;
Suggesting a survey item based on a collection of questionnaire items, characterized in that the final questionnaire generation module is configured to further include generating a final questionnaire based on the questionnaire items suggested by the questionnaire item user suggestion module and updating the questionnaire database, and How to verify answer reliability.
According to claim 7,
The survey execution module is configured to further include the step of online execution of the final survey generated by the final survey generation module for the survey responder terminal. Method for proposing survey items and verifying answer reliability based on a survey item collection, characterized in that.
According to claim 8,
The survey answer reliability AI verification module is configured to further include the step of evaluating the reliability of the survey answer by preparing and verifying the survey answer of the same survey responder terminal for the survey executed in the survey execution module. A method for proposing survey questions and verifying the reliability of answers based on .

Images