JP6997046B2 - Annotation support device - Google Patents

Description

The present invention relates to an annotation support device that streamlines annotations that give meaning to data, and more particularly to an annotation support device that streamlines annotations that add attribute information to spoken text in a dialogue scenario.

In order to realize a high-level dialogue in the dialogue agent system, a large-scale text data (dialogue scenario corpus) that describes the expected dialogue exchange (dialogue scenario) is required. In order for the system to have a more accurate dialogue, the dialogue behavior (greetings, questions, responses, etc.), context (time, place, surroundings, etc.), Kansei indicators (emotions, emotions, interest level, etc.) of the dialogue scenario, etc. ) And other attribute information is required.

Patent Document 1 discloses a method of selecting an appropriate response by estimating the attributes of user utterances (question type, dialogue action, topic category, etc.) for a dialogue system for chatting.

Patent Document 2 discloses a method of learning an utterance pattern using a time-series model and estimating a dialogue action in order to estimate the dialogue action with high accuracy.

Non-Patent Document 1 proposes a system that uses crowdsourcing to create a scenario by interactive input in a chat format by two workers. The system provides a function that can perform annotations such as dialogue actions at the same time as creating a scenario.

Japanese Unexamined Patent Publication No. 2017-027234 Japanese Unexamined Patent Publication No. 2018-025747

How to Build a Dialogue Corpus Utilizing Open Platform and Crowdsourcing, Hiroshi Tsukahara, Kei Utsumi, Association for Natural Language Processing Annual Conference (March 2015)

Patent Document 1 and Patent Document 2 propose a method of estimating attribute information such as dialogue actions and question types by analyzing text information of utterance contents. This makes it possible to estimate the attributes of user utterances and select a response according to the attributes from the dialogue scenario.

In estimating the user's utterance, a method of learning the characteristics of the utterance text to which the attribute information is added by machine learning or the like is used. By applying this method to the dialogue scenario, it is possible to automatically add attribute information to the dialogue scenario.

However, since the estimation accuracy by machine learning is not always high, when the estimation method is applied to both the dialogue scenario and the user's utterance, there is a problem that the accuracy is doubly lowered and is not at a practical level. Specifically, assuming that the estimation accuracy is 80%, the problem is that the response accuracy is significantly reduced, such as 80% of the attribute information of the dialogue scenario x 80% of the attribute information of the user's utterance = 64%. Become.

In order to avoid the above technical problems, there is a method of realizing a highly accurate response by manually annotating the attribute information in the dialogue scenario in advance.

Non-Patent Document 3 proposes a system that allows a large number of workers to create and annotate scenarios using crowdsourcing. However, for attribute information such as dialogue, context, and emotional index, it may be necessary to select the correct attribute value from dozens of categories (attribute values), which requires a lot of effort and relatively high expertise. Be done. Therefore, it is a problem that an enormous cost is required to annotate a large number of dialogue scenarios.

The first object of the present invention is to solve the above technical problems, estimate the attribute information of the dialogue scenario using machine learning, and limit the candidate attribute values in advance, thereby examining a large number of attribute values for the worker. It is an object of the present invention to provide an annotation support device for dialogue scenarios that can improve work efficiency without the need for the operation.

A second object of the present invention is a dialogue that makes it possible to reduce the work of a worker to the minimum necessary by controlling the required redundancy according to the reliability of estimation and the annotation result of the worker. The purpose is to provide a scenario annotation support device.

In order to achieve the above object, the present invention is characterized in that it has the following configuration in an annotation support device that supports annotation to a dialogue scenario.

(1) A means for estimating the attribute for each spoken text of the dialogue scenario, a means for determining the task condition of the annotation task to select at least one attribute from a plurality of estimated attributes, and creation based on the above task condition. It is provided with a means for providing the annotation task to the worker and a means for aggregating the annotation results from each worker who executed the annotation task.

(2) The estimation means are a means for extracting features for each spoken text of the dialogue scenario and a means for extracting features.
It is characterized by including an estimation model that estimates attributes from the extracted features and outputs an estimated likelihood.

(3) The estimation means further includes means for updating the estimation model by machine learning the relationship between the result of the annotation and the extracted feature.

(4) The estimation model is characterized in that attribute value candidates are estimated for each attribute type from the extracted features and the estimated likelihood is output.

(5) The means for determining the task condition affects the determination of the attribute value of the utterance text for which the annotation is requested by determining the utterance display number L when the utterance text for which the annotation is requested is displayed in chronological order together with other utterance texts. It is characterized in that it is determined based on the number of recent utterances.

(6) The means for determining the task condition is characterized in that the display number M of the attribute value candidates to be displayed in descending order of the estimated likelihood is determined based on the estimated likelihood.

(7) The means for determining the task condition is characterized in that the number of workers N requesting the annotation task for each utterance text is determined based on the estimated likelihood.

According to the present invention, the following effects are achieved.

(1) Since the attribute information of the dialogue scenario can be estimated using machine learning and the candidate attribute values can be limited in advance, the worker does not have to consider a large number of attribute values and the work efficiency is improved. Is possible.

(2) Redundancy such as the number of attribute value candidates to be presented, the number of utterances to be presented, and the number of workers to be requested is controlled according to the reliability of estimation and the annotation result of the worker, so the minimum required worker is the minimum required. Annotation is possible just by checking the limited information, and it is possible to improve work efficiency.

(3) Since the machine learning model is relearned based on the annotationd result, the annotation estimation is made highly accurate, and the number of attribute values and the number of workers to be annotated can be limited.

It is a block diagram which showed the structure of the annotation support system of the dialogue scenario which concerns on one Embodiment of this invention. It is a figure which showed the example of the dialogue scenario. It is a figure which showed the example of the field for storing the attribute information. It is a figure which showed an example of the annotation task request screen. It is a figure which showed the attribute value candidate of the attribute type "dialogue act" for each utterance text shown in FIG. 3 in descending order of the estimated likelihood. It is a figure which showed the example of the attribute value candidate estimated for each utterance text, and the estimated likelihood thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an annotation support system for a dialogue scenario according to an embodiment of the present invention.

The annotation support system is a scenario DB2 that stores a large amount of dialogue scenarios collected by means such as cloud sourcing, and annotation support that annotates this dialogue scenario with the support of a cloud worker (worker) W. The main configuration is the device 1.

The annotation support device 1 mainly includes an annotation estimation unit 11, a task condition determination unit 12, an annotation task generation unit 13, and an annotation update unit 14. Such an annotation support device 1 can be configured by implementing an application (program) that realizes each function on a general-purpose computer or server. Alternatively, it can be configured as a dedicated machine or a single-purpose machine in which a part of the application is made into hardware or ROM.

As shown in FIG. 2, the dialogue scenario includes text information (spoken text) of the expected user utterance and system response, as well as link information to the next utterance, and may be structured to include a branch. ..

Further, as shown in FIG. 3, in the scenario DB2, in addition to the utterance text, the attribute value is set for each attribute type such as "dialogue action", "topic category", "sensitivity index", and "context". It has a field for storing the registered attribute information.

For example, in the utterance "I play tennis in a circle.", The attribute value of the attribute type "dialogue" is "response, flat", and the attribute value of the attribute type "topic category" is "sports / tennis". "Acceptance" is registered as an attribute value of the type "sensitivity index".

In the annotation support device 1, the annotation estimation unit 11 includes a feature extraction unit 111, a feature learning unit 112, and an estimation probability calculation unit 113, and each feature and a feature of each dialogue scenario utterance text stored in the scenario DB2. Annotation candidates are estimated based on an estimation model that has been machine-learned in advance. In this embodiment, a plurality of attribute value candidates are estimated for each attribute type related to each utterance text.

The feature extraction unit 111 extracts features effective for annotation estimation from the dialogue scenario. Specifically, there is a method of performing morphological analysis or the like for each utterance text and using words as features. In addition, there is also a method of using a distributed expression that is numerically converted so that highly related words have close numerical values based on the appearance tendency of words.

The feature learning unit 112 learns the appearance pattern of the feature by machine learning from the scenario in which the attribute value of the estimation target is annotated. For machine learning, use SVM etc. when learning by a single utterance. Use HMM, RNN, etc. when learning utterances in chronological order and estimating including the context of utterances before and after. The input to machine learning can be the frequency of occurrence or distributed expression of words that are characteristic of the utterance, and the attribute value of the utterance.

The estimation likelihood calculation unit 113 estimates each attribute value candidate for the dialogue scenario in which the annotation is not performed, using the estimation model learned in the feature learning unit 112. In many machine learning tools, the estimated likelihood is calculated internally, and the estimated likelihood for each attribute value can be output.

The task condition determination unit 12 includes the utterance display number determination unit 121, the attribute value candidate number determination unit 122, and the task request number determination unit 123, and is described in detail below as conditions (task conditions) for requesting annotation from the worker W. Determine the number of utterance displays L, the number of attribute value candidates M, and the number of task requests N to be described.

FIG. 4 is a diagram showing an example of the annotation task request screen provided to each worker W when the annotation support device 1 requests the annotation task to the worker W. Here, the system response utterance “You are good at it, isn't it? An example of requesting an annotation task for "." Is shown.

On the request screen, the latest L past utterances (5 in this case) including the annotation target utterance "You're good at it." And each utterance text of the system response are alternately stepped to reflect the time-series relationship. It is displayed.

At this time, if the number of utterance displays L is large, highly accurate annotation is possible, but the time required for the worker W to read the utterance content becomes long. On the other hand, if the number of utterances displayed is small, the time is shortened, but the accuracy of annotation tends to decrease.

In addition, in order to reduce the burden on the annotation task of Worker W, M attribute value candidates are listed for each attribute type. Worker W can perform annotation by checking one of the attribute value candidate radio buttons for each attribute type.

In this embodiment, three (M = 3) attribute value candidates "communication, non-past, creation", "question, Yes No", and "request" are prepared for the attribute type "dialogue act". For the attribute type "topic category", four (M = 4) attribute value candidates "education / university", "sports / tennis", "art / painting" and "art / music" are prepared. Worker W refers to the past utterance history and selects the attribute value corresponding to the utterance text "You are good" to be annotated with the radio button.

At this time, if the number of attribute value candidates M to be displayed is large, highly accurate annotation can be expected, but the time required for the worker W to judge becomes long. On the other hand, if the number of attribute value candidates M is small, the time is shortened, but the accuracy of annotation tends to decrease.

Furthermore, since the correct attribute value is not always selected by requesting a single worker W for annotation, request N worker W for each utterance text and make the attribute more accurate by a method such as majority voting. It is desirable to determine the value.

At this time, if the number of annotation requests N for each utterance text is large, highly accurate annotation can be performed, but the cost burden increases. On the other hand, if the number of annotation requests N is reduced, the cost burden can be reduced, but the accuracy of annotation tends to decrease.

In the present embodiment, the task condition determination unit 12 determines the utterance display number L, the attribute value candidate number M, and the task request number N to the optimum values in consideration of these trade-offs.

The utterance display number determination unit 121 determines the utterance display number L by analyzing how many previous utterances affect the attribute value determination of the utterance text for which annotation is requested.

Specifically, by inputting the speech sequence (length L = LA) to the machine learning device used for annotation estimation such as HMM (Hidden Markov Model) and RNN (Recurrent Neural Network), each attribute value is estimated. The likelihood is calculated. Here, the estimated likelihood of each attribute value when the length L of the input utterance series is gradually shortened to L = LA-1, LA-2, LA-3 ... Is also output, and the estimated likelihood of each attribute value is also output. Determine the minimum L for which the difference in estimated likelihood is less than the threshold.

FIG. 5 is a diagram showing the attribute value candidates of the attribute type “dialogue act” in descending order of their estimated likelihoods for the utterance “You are good at it.” And the four utterance texts immediately before it shown in FIG. ..

In the example shown in the figure, for the utterance "You're good at it.", The estimated likelihood of the attribute value candidate "communication, non-past, creation" is the highest at 0.35, and the estimated likelihood of the attribute value candidate "question, Yes / No". The degree is 0.30, which is the second highest. Similarly, for the utterance "It's been nearly 10 years since I was from university", the estimated likelihood of the attribute value candidate "response, declarative" is the highest at 0.60, and the estimated likelihood of the attribute value candidate "transmission, non-past, creation". 0.35 is the second highest.

Here, consider a case where the utterance "You are good at it." Is targeted for annotation, and the attribute value candidate to be presented to the worker W is selected from a plurality of attribute value candidates.

Referring to FIG. 5, the difference in estimated likelihood at L = 5, 4, 3 is sufficiently small, so if only the last three utterances L = 1 to 3 are presented, the worker W will be annotated with the dialogue action. It is expected that it will be possible. When requesting annotations for multiple attributes at the same time, apply the maximum value of L calculated for each attribute.

As described above, in the present embodiment, the utterance display number L when displaying the utterance text for which annotation is requested together with other utterance texts in chronological order is the attribute of the utterance text for which annotation is requested and at least one utterance text thereof. It is determined based on the difference in the estimated likelihood of each attribute value candidate for each type.

The attribute value candidate number determination unit 122 sets the display number M so that the sum of the estimated likelihoods of the upper M attribute value candidates exceeds the threshold value θ _M. With reference to FIG. 6, it is also assumed that the attribute value candidates are registered in descending order of the estimated likelihood for each utterance text.

When θ _M = 0.9 is set, the estimated likelihood 0.90 of the attribute value candidate “question, what” is already set to the threshold value θ _M for the attribute type “dialogue” of the utterance “what are you doing on holidays?”. Have reached. Therefore, it is decided not to annotate the attribute value candidate "question, what" as the correct answer.

For the utterance "I play tennis in a circle", the estimated likelihood of the attribute value candidate "response, debate" of the attribute type "dialogue" is 0.75 and the estimated likelihood of the attribute value candidate "transmission, non-past, reality" is 0.15. Since the sum of and reaches 0.90 and reaches the threshold θ _M , the top two attribute value candidates are presented to the worker W for annotation.

Regarding the utterance "It's been nearly 10 years since I'm from university.", The sum of the estimated likelihoods of the top two attribute value candidates is 0.85, which is less than the threshold θ _M , so the top three or more attribute values. Present the candidate to the worker.

As the threshold value θ _M is set lower, the number of candidates presented decreases, so that the time required for annotation can be shortened and the cost can be reduced, but the possibility that the correct attribute value is not included increases. Annotation quality is degraded. The threshold value θ _M is determined according to the desired annotation quality.

As described above, in the present embodiment, the display number M is determined based on the added value obtained by sequentially adding the estimated likelihoods of the attribute value candidates of each attribute type for each utterance text for which annotation is requested.

The task request number determination unit 123 determines the number of workers N to request annotation based on the estimated likelihood of the attribute value and the attribute value candidate number M for each utterance text to be annotated.

For example, when the estimated likelihood of the attribute value candidate having the highest estimated likelihood exceeds the threshold value θ _N , the annotation result of the attribute can be determined to be the attribute value having the highest estimated likelihood (without annotation). N = 0).

When the estimated likelihood of the attribute value candidate with the highest estimated likelihood is below the threshold θ _M , Pi is the i-th (i = 1 to M) estimated likelihood for each of the M attribute values presented to the worker W. When the estimated likelihood of the high attribute value of is set and the number of times the worker W selects the i-th attribute value is Li, the necessity of requesting another worker is determined based on the following equation (1). ..

Maximum value of Pi * Li-Second largest value of Pi * Li <threshold value θ _N … (1)

In the present embodiment, if the above equation (1) is satisfied, one worker W is requested to perform additional annotation, and the value of Li is updated according to the annotation result of the worker W. If the above equation (1) does not hold, no additional annotation is requested.

In the example of Fig. 6, when requesting annotation to the utterance "How many years have you continued?", The attribute value display number M = 2, P1 = 0.85 <θ _N (= 0.90), P2 = 0.10. .. Since the maximum value and the second largest value of Pi * Li are both 0 (because L1 = L2 = 0), request annotation from one worker W.

Here, when the worker W selects the attribute value of i = 1 (here, the estimated likelihood 1), the estimated likelihood is 0.85, so the above-mentioned "maximum value of Pi * Li" is 0.85. Further, since the above-mentioned "second largest value of Pi * Li" is still 0, the difference is 0.85, which satisfies the above equation (1). Therefore, one more worker W is added.

Furthermore, when the next worker W also selects the attribute value of i = 1, the "maximum value of Pi * Li" is 1.70 and the "second largest value of Pi * Li" is still 0, so the difference is. The difference is 1.70, which does not satisfy the above equation (1). Therefore, the annotation regarding the utterance "How long have you been doing this?" Ends.

As described above, in the present embodiment, the product of the annotated attribute value and the number of annotations is obtained for each attribute value based on the result of the annotation for each utterance, and the number of workers N is calculated based on the difference between the products. decide.

By requesting annotation by the above method, the result of an attribute value with high estimated likelihood can be confirmed at the stage when a small number of workers annotate the attribute value, and efficiency can be improved. In addition, since the attribute value with low estimated likelihood is annotated by many workers and the result is confirmed, reliability can be ensured.

The above is an example of the mounting method, and the threshold value θ _N is adjusted according to the quality of the desired annotation. Further, since the characteristics of the initial value Pi differ depending on the machine learning device to be used, a function for adjusting may be used.

The annotation task generation unit 13 generates a task for annotation described with reference to FIG. 4 based on the values of the task conditions L and M, and is a necessary worker based on the task request necessity determination. Make a request for a few minutes.

The annotation update unit 14 receives the annotation result of the worker W, and registers the attribute value having Pi * Li as the maximum value in the scenario DB2 as the annotation result. Further, when a certain amount of annotation results are added to the scenario DB2, the estimation model machine-learned by the feature learning unit 112 of the annotation estimation unit 11 is updated.

1 ... Annotation support device, 2 ... Scenario DB, 11 ... Annotation estimation unit, 12 ... Task condition determination unit, 13 ... Annotation task generation unit, 14 ... Annotation update unit, 111 ... Feature extraction unit, 112 ... Feature learning unit, 113 ... Estimated likelihood calculation unit, 121 ... Speech display number determination unit, 122 ... Attribute value candidate number determination unit, 123 ... Task request number determination unit

Claims (8)

In the annotation support device that supports annotation to dialogue scenarios
A means of estimating the attributes of each spoken text in a dialogue scenario,
A means of determining the task conditions of an annotation task that causes at least one attribute to be selected from multiple estimated attributes.
A means for providing an annotation task created based on the above task conditions to a worker,
It is provided with a means for aggregating the annotation results from each worker who has executed the annotation task.
The estimation means is
A means of extracting features for each spoken text in a dialogue scenario,
Including an estimation model that estimates attributes from the extracted features and outputs estimated likelihood.
The estimation model estimates attribute value candidates for each attribute type from the extracted features and outputs an estimated likelihood.
The means for determining the task condition is to determine the number of utterance displays L when displaying the utterance text for which annotation is requested together with other utterance texts in chronological order, which affects the determination of the attribute value of the utterance text for which the annotation is requested. Annotation support device characterized in that it is determined based on the number of utterances of . In the annotation support device that supports annotation to dialogue scenarios
A means of estimating the attributes of each spoken text in a dialogue scenario,
A means of determining the task conditions of an annotation task that causes at least one attribute to be selected from multiple estimated attributes.
A means for providing an annotation task created based on the above task conditions to a worker,
It is provided with a means for aggregating the annotation results from each worker who has executed the annotation task.
The estimation means is
A means of extracting features for each spoken text in a dialogue scenario,
Including an estimation model that estimates attributes from the extracted features and outputs estimated likelihood.
The estimation model estimates attribute value candidates for each attribute type from the extracted features and outputs an estimated likelihood.
The means for determining the task condition is an annotation support device characterized in that the display number M of attribute value candidates to be displayed in descending order of estimated likelihood is determined based on the estimated likelihood . In the annotation support device that supports annotation to dialogue scenarios
A means of estimating the attributes of each spoken text in a dialogue scenario,
A means of determining the task conditions of an annotation task that causes at least one attribute to be selected from multiple estimated attributes.
A means for providing an annotation task created based on the above task conditions to a worker,
It is provided with a means for aggregating the annotation results from each worker who has executed the annotation task.
The estimation means is
A means of extracting features for each spoken text in a dialogue scenario,
Including an estimation model that estimates attributes from the extracted features and outputs estimated likelihood.
The estimation model estimates attribute value candidates for each attribute type from the extracted features and outputs an estimated likelihood.
The means for determining the task condition is an annotation support device characterized in that the number N of workers requesting an annotation task for each utterance text is determined based on the estimated likelihood . The estimation means is
The annotation support device according to claim 1 , further comprising means for updating the estimation model by machine learning the relationship between the annotation result and the extracted feature. The means for determining the task condition is to determine the number of utterance displays L when displaying the utterance text for which annotation is requested together with other utterance texts in chronological order, which affects the determination of the attribute value of the utterance text for which the annotation is requested. The annotation support device according to claim 2 or 3 , wherein the annotation is determined based on the number of utterances. The means for determining the task condition is to determine the utterance display number L based on the difference in the estimated likelihood of each attribute value candidate for each attribute type of the utterance text for which annotation is requested and at least one utterance text immediately thereof. The annotation support device according to claim 1 , wherein the annotation support device is characterized. The means for determining the task condition is to set the display number M based on the added value obtained by sequentially adding the estimated likelihoods of the attribute value candidates of each attribute type for each utterance text for which annotation is requested. The annotation support device according to claim 2 , which is characterized. The means for determining the task condition is to obtain the product of the annotated attribute value and the number of annotations for each attribute value based on the result of the annotation for each utterance text, and the number of workers based on the difference between the products. The annotation support device according to claim 3 , wherein N is determined.

Images