JP2023037406A - Information processing method and information processing apparatus - Google Patents

Abstract

To efficiently perform accurate classification by applying a technology of machine learning to a response for a free answer style question of a questionnaire.SOLUTION: An information processing method comprising: a transmission step of transmitting a questionnaire including a free answer style question to respondents; a reception step of receiving answers for the questionnaire from a plurality of respondents; an acquisition step of acquiring vector information by vectorizing a sentence of a free answer in a plurality of answers; a first clustering step of clustering on the basis of vector information to perform classification of the plurality of answers into a plurality of classes and acquiring class information including label information for each class; a generation step of generating a learned model for re-clustering the sentence of the free answer using a teacher data in which label information is associated with the sentence of the free answer; and a second clustering step of performing re-clustering by applying the learned model to sentences of the plurality of free answers, is used.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing method and an information processing apparatus.

2. Description of the Related Art There is a questionnaire survey in which a questionnaire including questions is sent to respondents and the answers obtained are aggregated. By conducting questionnaires to a large number of respondents and statistically analyzing the aggregated responses, the researcher can be used for various purposes such as decision support in marketing and product development, the creation of statistical data, and academic research. You can get useful information for your activities.

In particular, in recent years, the use of online research, in which a researcher conducts questionnaires to a large number of respondents via the Internet, is increasing. Online research has advantages such as relatively light burden on respondents, easy tabulation of response results, and relatively low cost. When the questionnaire is entrusted to a research company, it is possible to use a membership organization formed by the research company that consists of a large number of respondents whose attribute information has already been grasped.

One way to classify questions in a questionnaire is whether they are closed-ended questions or open-ended questions. A multiple-choice question is one in which the respondent selects an answer from options prepared in advance. Among multiple-choice questions, there are alternative selection methods such as a "Yes/No" method and a 5-level evaluation method, and there are also cases in which a plurality of options can be selected. On the other hand, the free answer format is a format also called a free description format or a free answer (FA) format, and is a format in which respondents freely describe their answers to questions. An open-ended question may include numerical values, words, etc., or may require text to be entered.

One of the characteristics of open-ended questions is that, in contrast to multiple-choice questions, where only answers that are included in the existing options can be obtained, free-answer questions can provide answers based on free ideas that were not anticipated by the operator. is mentioned.

On the other hand, answers to closed-ended questions are limited to those included in the options, and are therefore suitable for automatic aggregation. Since it is a sentence consisting of natural language, automatic aggregation and quantitative analysis are difficult. In other words, in order to process free answers quantitatively, it is necessary to classify the contents of the answers manually or using a tool. Conventionally, an investigator visually confirms the responses one by one, and assigns keywords or codes called "tags" or "labels" according to the contents of the responses to classify them. As a result, the cost and working time are increased.

Therefore, in recent years, various studies have been made to efficiently classify free answers. Patent Document 1 (Japanese Laid-Open Patent Publication No. 2001-266060) discloses a questionnaire analysis system that automatically analyzes free answers using a text classification engine. In addition, Patent Document 2 (Japanese Patent Application Laid-Open No. 2011-022874) discloses a data processing device that facilitates quantitative analysis by replacing words and phrases in answer data of free-form answers using a conversion word database. disclosed.

In addition, from the viewpoint of classification of natural language, Patent Document 3 (Japanese Patent Application Laid-Open No. 2020-126631) discloses that natural language is vectorized, classified by clustering based on similarity, and labeled. Techniques are disclosed.

Furthermore, attempts have been made to improve the efficiency and accuracy of classification by applying machine learning to technical fields related to natural language. For example, in Patent Document 4 (Japanese Patent Application Laid-Open No. 2021-056591), in order to create teacher data used for machine learning, clustering results are presented to the user, and machine learning is performed using labels specified by the user. Techniques for doing so are disclosed. In addition, Patent Document 5 (Japanese Patent Application Laid-Open No. 2021-068225) discloses that as a general machine learning technique, a trained model is created using teacher data, and language processing is performed using the model. ing.

Japanese Patent Application Laid-Open No. 2001-266060 JP 2011-022874 A JP 2020-126631 A JP 2021-056591 A JP 2021-068225 A

However, in such prior art, the application of machine learning in the classification of answers to open-ended questionnaire questions has not been sufficiently studied, and further technical development is required.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to apply machine learning technology to answers to open-ended questionnaire questions, and to perform efficient and highly accurate classification.

In order to achieve the above objects, the present invention employs the following configuration.
i.e.
a sending step of sending a questionnaire containing open-ended questions to the respondent;
a receiving step of receiving responses to the questionnaire from a plurality of respondents;
an obtaining step of obtaining vector information by vectorizing open-ended sentences in the plurality of answers;
a first clustering step of performing clustering based on the vector information to classify the plurality of answers into a plurality of classes and obtaining class information including label information for each class;
a generation step of generating a trained model for re-clustering the free-answer sentences using teacher data in which the label information and the free-answer sentences are associated;
A second clustering step of applying the learned model to the plurality of free-answer sentences to perform the re-clustering;
An information processing method characterized by having

The present invention also employs the following configurations.
i.e.
a sending means for sending a questionnaire containing open-ended questions to a respondent;
receiving means for receiving answers to the questionnaire from the plurality of respondents;
Acquisition means for acquiring vector information by vectorizing free-response sentences in the plurality of answers;
a first clustering means for performing clustering based on the vector information to classify the plurality of answers into a plurality of classes and obtaining class information including label information for each class;
generating means for generating a trained model for re-clustering the free-answer sentences using teacher data in which the label information and the free-answer sentences are associated;
a second clustering means for performing the re-clustering by applying the learned model to the plurality of free-answer sentences;
An information processing apparatus characterized by having

According to the present invention, it is possible to apply machine learning technology to answers to open-ended questions in a questionnaire, and to perform efficient and highly accurate classification.

It is a figure explaining the structure of an information processing system. FIG. 5 is a diagram for explaining the processing flow of the first embodiment; It is a figure explaining the functional block of an information processing system. It is a figure explaining the question contained in a questionnaire. It is a figure explaining the answer and class information by a respondent.

Preferred embodiments of the present invention will be described below with reference to the drawings. However, the components and their relative arrangement described below should be changed as appropriate according to various conditions such as the system to which the invention is applied, and the scope of this invention is limited to the following description. It is not intended to

INDUSTRIAL APPLICABILITY The present invention can be preferably applied to questionnaire surveys in which respondents are asked questions and answers are obtained. The present invention can be regarded as an information processing system or an information processing method for performing such investigation. The present invention can also be regarded as an information processing apparatus that constitutes an information processing system. The present invention can also be regarded as a control method for an information processing system or an information processing apparatus. The present invention can also be regarded as a program that operates using computational resources of an information processing apparatus and executes each step of an information processing method. The present invention can also be regarded as a storage medium storing such a program. The storage medium may be a non-transitory computer-readable storage medium.

In the following examples, an investigator is an entity that conducts an investigation, conducts questionnaires to respondents according to arbitrary investigation purposes such as economic activities and research, and aggregates and analyzes the collected answers. The researcher may be a company or the like itself having a research purpose, or may be a research agency such as a research agency. A respondent is a natural person who responds to a questionnaire sent by an investigator, and is also called a monitor.

[Example 1]
<System configuration>
A system configuration of this embodiment will be described with reference to FIG. The information processing system of this embodiment is a survey system 1 in which a surveyor who is an Internet researcher conducts a questionnaire survey of respondents 3 . A survey system 1 includes at least a server 4 used by a surveyor and a terminal 2 used by a respondent 3 . The operator 8 belongs to a researcher, who is a researcher, and operates the server 4 to perform tasks relating to questionnaires, and is engaged in setting and conducting questionnaires, analyzing answers to questions, and the like. Server 4 and terminal 2 can communicate with each other via Web 6, a dedicated line, or the like.

The researcher requests the monitor to participate in the questionnaire by a contact method such as e-mail, and the respondent 3 accesses the server 4 using the browser of the terminal 2 or a dedicated application, and the questionnaire is started. Note that a preliminary questionnaire may be conducted before the actual questionnaire, a respondent group consisting of a plurality of respondents having attributes in line with the purpose of the survey may be determined, and the questionnaire allocation conditions may be satisfied. Also, if the researcher who is the investigator organizes the respondent candidates as members, the respondent may be selected from those candidates and requested to participate in the questionnaire.

The terminal 2 comprises at least image display means 21 for displaying questionnaire questions to the respondent 3, input means 22 for the respondent 3 to answer the questions, and control means 23 for controlling the operation of the device. is. The terminal 2 includes a processor such as a CPU as the control means 23, a storage means such as a memory, a communication means, etc., and operates according to program instructions and operation input from the respondent 3. PC, workstation, smartphone, tablet An information processing apparatus such as a device is suitable. When the terminal 2 is a PC or a workstation, a display device such as a liquid crystal display or an organic EL display can be used as the image display means 21, and an input interface such as a keyboard or mouse can be used as the input means 22. Further, when the terminal 2 is a touch panel type smartphone or tablet device, the touch panel may serve as both the image display means 21 and the input means 22 .

The server 4 is an information processing device for transmitting a questionnaire including questions to the terminal 2 via the Web 6 and collecting and analyzing answers from the respondent 3 . The server 4 includes at least an image display means 41 and an input means 42 which are interfaces for the operator 8 to set, execute and analyze a questionnaire, and a control means 43 for controlling the operation of the apparatus. The server 4 includes a processor such as a CPU as the control means 43, storage means such as a memory, communication means, etc., and operates according to program instructions and operation input from the operator 8. Information processing such as a PC or a work station. A device is preferred. Note that the server 4 does not necessarily have to consist of a single information processing device, and a plurality of information processing devices may be combined to realize the functions of the server. Also, the server 4 may use computing resources on the cloud. The investigation method as the information processing method in this embodiment is realized by the control means 43 executing the function of the program.

The server 4 is connected to the database 45 via connection means such as an adapter and a cable so as to be able to communicate with each other. The database 45 stores data used in the investigation method of the present embodiment, and uses a storage device having a storage medium such as a hard disk or flash memory. The physical configuration and arrangement of the database 45 are not particularly limited, and either an on-premise system or a cloud system may be adopted. The database 45 of this embodiment is a relational database, and includes at least a respondent table 451 storing respondent information about the respondent 3, a question table 452 storing question information about questions included in the questionnaire, and answers obtained from the respondent. includes an answer table 453 that holds answer information about.

<Processing flow>
Next, the flow of processing in this embodiment will be described. FIG. 2 is a processing flow diagram, and each step is basically executed by each functional block of the server 4 . FIG. 3 is a functional block diagram of the server 4, and each functional block denoted by ". be. Each functional block may be configured as a program module that is read by a CPU and performs information processing on input information, for example.

In step S1, respondent 3 is determined. The respondent determination unit 400 refers to the respondent table 451 of the database 45 and determines in advance the respondent 3 who meets the questionnaire conditions from the respondent candidates. For example, the respondent 3 may be determined based on the attributes of the respondent candidates held by the researcher, or the respondent 3 may be narrowed down by conducting a preliminary questionnaire. After the respondent group satisfying the questionnaire allocation conditions is determined, the process proceeds to the next step.

In step S2, a questionnaire is sent to the respondent 3. The questionnaire implementation unit 405 transmits the questionnaire held by the question table 452 to the respondent 3 by the communication means included in the respondent information. Any means of communication may be used, such as an email format, a web format, or a dedicated application format. Also, the questionnaire of this embodiment shall include at least one open-ended question.

In step S3, an answer is received from respondent 3. Questionnaire implementation unit 405 stores the received responses in response table 453 .

Here, as shown in FIG. 4, the questionnaire of this embodiment includes a question Q1 of a selection type from five options and a question Q2 of a free-answer type.

FIG. 5A is an example of answers stored in the answer table 453. FIG. The respondent ID is an ID that identifies the respondent 3, and various attribute information is stored in the respondent table 451 in association with the respondent ID. The questionnaire ID is an ID that identifies a questionnaire, and questions are stored in the question table 452 in association with the questionnaire ID. Furthermore, in the answer table 453, answers by each respondent 3 are stored in association with the questionnaire ID and the respondent ID. By combining the respondent ID and the questionnaire ID, it is possible to specify the sample ID of the response used in this embodiment. As shown in the figure, the answer A1 stores options, and the answer A2 stores the content of free answers.

Subsequently, in steps S4 to S6, a series of processes up to vectorization and clustering of the free answers are performed, which is also called first clustering in this embodiment.

In step S4, preprocessing is performed on the free answer. The pre-processing unit 410 pre-processes the free-response text so that subsequent processing can be properly performed. Examples of pre-processing include data cleansing to remove or modify irrelevant or inaccurate data such as noise, word extraction from answers, and the like. The extracted words can be used, for example, when automatically generating labels for each cluster.

In step S5, vectorization of free answers is performed. The vectorization unit 415 applies a model to the preprocessed open-ended answers and obtains vector information for each sentence. The vector information obtained by the processing of the vectorization unit 415 includes a vector that expresses the characteristics of the free-response sentence to be processed by a set of numerical values of a predetermined dimension. As a model used at that time, a machine learning model for natural language processing, for example, a model such as SentenceBERT for the purpose of high-quality clustering of documents can be used.

In step S6, clustering is performed based on the vector information for each sentence. The clustering unit 420 applies a clustering method to the data set of vector information, compares the similarity based on the feature vector, groups free answers obtained from a plurality of respondents, and classifies them into a plurality of classes. . The number of classes may be specified by the operator 8 as a specific numerical value, or may be selected from a predetermined number of classes. The class information obtained by the processing of this step includes a label attached to each class, information on classification of each data into classes, a sentence representing a certain class (hereinafter referred to as a representative sentence), and the like. Although any clustering method can be used, for example, a Gaussian mixture model (GMM), which is an unsupervised machine learning model that automatically classifies data, can be preferably used.

Here, an example of data obtained in step S6 will be described with reference to FIG. 5(b). The class name here is label information given to each class, and in this embodiment, the class name automatically determined based on the feature vector by the clustering algorithm is used. In addition, representative sentences are automatically determined.

As described above, in the first clustering performed from steps S6 to S4, clustering is performed after a plurality of answers given by a plurality of respondents 3 to a certain open-ended question are input to the server 4. Processing is performed until the representative sentence is acquired. Since a series of these processes can be executed by subjecting the answer data to predetermined information processing, it is preferable to automate the process using a tool that operates on the server.

Subsequently, in steps S7 and S8, an adjustment process is performed in which the operator 8 adjusts the output result of the first clustering so as to obtain a result in line with the research purpose. This processing is performed to further optimize the data obtained by the first clustering for the purpose of investigation before using it as training data for machine learning. That is, if the processing of steps S4 to S6 is automatically performed by a tool, the processing time and labor can be reduced. may not have been converted. Also, if the researcher is a researcher who conducts questionnaires at the request of customers, it is necessary to provide survey results in an easy-to-understand format according to the customer's background. Therefore, it is preferable to perform steps S7 and S8 if possible.

In step S7, the output section 425 outputs to the image display means 41 so that the operator 8 can confirm the class information. Therefore, the operator 8 checks the output information and adjusts the class information. The output unit 425 is a program module that controls display on the output interface, and causes the image display unit 41 to display class information, for example.

In step S8, the input reception unit 430 receives the adjustment result. The input reception unit 430 is a program module that receives and interprets the content input by the operator 8, and receives the representative sentence adjustment result input by the input unit 42, which is an input interface such as a keyboard. As described above, in the adjustment process performed from steps S7 to S8, the class information adjusted by the operator 8 is acquired for each class in which free answers are clustered.

Here, an example of data acquired in step S8 will be described with reference to FIG. 5(c). The objects to be adjusted in this embodiment are class names and representative sentences. As for the class name, the operator 8 assigns an appropriate name while referring to the contents of the free answers included in the class, and inputs the modified class name as the modified label name. Also, the class labeled "I want to drink coffee" and the class labeled "I like coffee" are integrated into the same class "I like coffee and want to drink". In addition to such class integration, classes may be split or some of the answers contained in one class may be moved to another class. In addition, adjustment processing that is determined to be appropriate based on the knowledge and experience of the operator may be performed. As for representative sentences, we create modified representative sentences so that they are natural sentences that are easy for customers to understand.

Subsequently, in steps S9 and S10, a model M is created based on the adjusted data, and second clustering, which is re-clustering using the model, is performed. In step S9, the model creation unit 435 performs machine learning using the teacher data in which the sentence of the free answer and the class information of the adjusted class to which the free answer belongs are associated as a pair, and generates a model M. do. Model M is a trained model suitable for classifying free responses to questionnaires into appropriate classes.

In step S10, the classification unit 440 applies the model M to the free answers in the answer table 453 to classify them. As a result, each free answer is classified based on the adjusted class information and given a class name as label information. As a result, the free answers are classified in a form suitable for quantitative analysis and optimized for the purpose of the survey and the background of the customer. The operator 8 may further fine-tune the class information and representative sentences obtained as a result of the second clustering.

In step S11, the tallying unit 445 uses the classified free responses to perform tallying processing according to desired research purposes such as marketing, creation of statistical data, and academic research, and outputs a report. For example, it is possible to cross-tabulate between respondent attributes and other open-ended questions. If the questionnaire includes a plurality of free-answer questions, the processing of this flow may be performed for each question.

As described above, according to the information processing method of this embodiment, various answers to open-ended questions are given appropriate, easy-to-understand labels according to the purpose of the survey based on the operator's knowledge and experience. can do. As a result, it becomes easier to quantitatively tabulate and analyze free answers, and it is possible to reduce costs and improve survey accuracy.

4: Server, 41: Image display means, 42: Input means, 43: Control means, 405: Questionnaire execution unit, 415: Vectorization unit, 420: Clustering unit, 425: Output unit, 430: Input reception unit, 435: model generator, 440: classifier, M: model

Claims (8)

a sending step of sending a questionnaire containing open-ended questions to the respondent;
a receiving step of receiving responses to the questionnaire from a plurality of respondents;
an obtaining step of obtaining vector information by vectorizing open-ended sentences in the plurality of answers;
a first clustering step of performing clustering based on the vector information to classify the plurality of answers into a plurality of classes and obtaining class information including label information for each class;
a generation step of generating a trained model for re-clustering the free-answer sentences using teacher data in which the label information and the free-answer sentences are associated;
A second clustering step of applying the learned model to the plurality of free-answer sentences to perform the re-clustering;
An information processing method characterized by having an output step of outputting the label information acquired in the first clustering step to an interface together with the free answer sentence;
an input receiving step of receiving an input of corrected label information corrected from the label information by an operator's operation using the interface;
further having
2. The information processing method according to claim 1, wherein said teacher data in which said correction label information and said free answer sentence are associated is used in said generating step. In the output step, for each class, a representative sentence is output to the interface together with the label information;
3. The information processing method according to claim 2, wherein said input receiving step receives an input of a corrected representative sentence corrected from said representative sentence by said operator's operation. 4. The information processing method according to any one of claims 1 to 3, further comprising a preprocessing step of preprocessing the text of the free answer received in the receiving step. 5. An information processing method according to claim 4, wherein said preprocessing includes at least one of data cleansing processing and word extraction processing. 6. The information according to any one of claims 1 to 5, further comprising an aggregation step of performing aggregation processing using the plurality of answers re-clustered into the plurality of classes in the second clustering step. Processing method. A program for causing an information processing apparatus to execute each step of the information processing method according to any one of claims 1 to 6. a sending means for sending a questionnaire containing open-ended questions to a respondent;
receiving means for receiving answers to the questionnaire from the plurality of respondents;
Acquisition means for acquiring vector information by vectorizing free-response sentences in the plurality of answers;
a first clustering means for performing clustering based on the vector information to classify the plurality of answers into a plurality of classes, and acquiring class information including label information for each class;
generating means for generating a trained model for re-clustering the free-answer sentences using teacher data in which the label information and the free-answer sentences are associated;
a second clustering means for performing the re-clustering by applying the learned model to the plurality of free-answer sentences;
An information processing device comprising:

Images