KR20210049518A - Method and apparatus for interpreting intention of query - Google Patents

Abstract

The method in which a computing device operated by at least one processor interprets a query comprises: a step of receiving queries in which intentions interpreted by an arbitrary device are labeled; a step of classifying the input queries into a reference query and comparison queries expected to be erroneously classified, and analyzing a vocabulary pattern of the reference query; a step of generating training data with the reference query, the comparison queries, and the vocabulary pattern; and a step of using the training data to train a query interpretation model. The comparison queries are queries indicating a frequency of a predetermined value or lower among the input queries.

Description

Query interpretation method and apparatus {METHOD AND APPARATUS FOR INTERPRETING INTENTION OF QUERY}

The present invention relates to a technique for interpreting the intention of a query.

In a dialogue system or a document analysis system, a query intention analysis system that accurately classifies the intention of a user query is required, and most of these query intention analysis systems are learned through machine learning methods. In this case, in order to interpret using a machine learning method, a large amount of learning sentences to which the user's intention is attached is required.

Since a large amount of sentence logs are generated every day through a commercial system, learning data can be generated using this, but in the process of automatically responding to sentences and intentions, there are many difficulties in terms of cost and time when using experts' manpower. have.

In order to solve these shortcomings, a method of using logs collected from an actual commercial service system has also appeared. The sentence and intention are matched in advance, the expert confirms and corrects the result of the actual commercial service, and the modified content is used again for learning to improve the performance of the system.

However, it is difficult to check all of the large number of commercial service logs manually and it takes a lot of time, so most of them are not used.

The task to be solved is to provide a method and apparatus for selecting a few queries that are expected to be misclassified among queries whose intention is determined in a commercial service, and generating a query analysis model using the selected misclassified candidate queries. will be.

A method of interpreting a query by a computing device operated by at least one processor according to an embodiment, the step of receiving queries labeled with an intention interpreted by an arbitrary device, wherein the input queries are misclassified with a reference query. Classifying into expected collation queries, analyzing the vocabulary pattern of the reference query, generating learning data from the reference query, the collating query, and the vocabulary pattern, and analyzing the query using the learning data And training a model, wherein the matching queries are queries that appear with a frequency less than or equal to a predetermined value among the input queries.

The generating of the training data comprises: generating a filtering rule using the semantic similarity of the reference query and the matching queries, selecting misclassification candidates from the matching queries according to the filtering rule, and querying the reference query Comparing the vocabulary pattern with the vocabulary patterns of the misclassification candidates, extracting core vocabularies that exist in the reference query and do not exist in the misclassification candidates.

The selecting of the misclassification candidates includes linguistically analyzing the reference query and the reference query to generate each embedding vector, and determining a semantic similarity between the embedding vector of the reference query and each embedding vector of the reference query. It may include calculating, and determining, as candidates for misclassification, contrast queries having a calculated semantic similarity value equal to or greater than a predetermined value.

In the filtering rule, a control query determined to be the same intention as the reference query may be excluded from misclassification candidates.

After the step of determining the candidate for misclassification, receiving a determination result from an expert whether the intention labeled on the candidate for misclassification matches the intention of the actual user, and referring to the result, the labeled intention When a query such as the user's intention is included in the misclassification candidates, the step of selecting the misclassification candidates may be included again.

In the step of extracting the core vocabulary, a vocabulary that is not recognized as an arbitrary entity name from among vocabularies constituting the reference query may be extracted.

After the learning step, inputting a new query into the learned query interpretation model, determining the intention of the new query from the learned query interpretation model, and the intention labeled in the new query and the determined intention If is different, the step of changing the intention of the new query to the determined intention may be further included.

A computing device according to an embodiment, comprising a memory and at least one processor that executes instructions of a program loaded in the memory, wherein the program is a query labeled with an intention interpreted by an arbitrary device. Receiving inputs, classifying the input queries into control queries expected to be misclassified from a reference query, generating an embedding vector of each of the reference query and the reference query, and the embedding vector of the reference query and the Calculating the semantic similarity between each embedding vector of the comparison queries, selecting the comparison queries with the calculated semantic similarity value equal to or greater than a predetermined value as misclassification candidates, analyzing the vocabulary pattern of the reference query and the misclassification candidates Extracting core vocabulary that exists in the reference query and does not exist in the misclassification candidates, generating training data from the reference query, the misclassification candidates, and the core vocabulary, and using the learning data A computing device comprising instructions described to perform the step of training a query interpretation model.

In the classifying step, a query having an intention label that performs the same function as the intention label of the reference query among the input queries may be classified as correctly classified.

In the step of extracting the core vocabulary, a semantic pattern may be generated in which a vocabulary constituting the reference query is replaced with a vocabulary having a significant relationship with the vocabulary.

According to the present invention, since a process of efficiently extracting a small number of candidates expected to be misclassified among queries determined in advance in a commercial service, speed and accuracy can be improved compared to a query analysis device targeting all misclassified queries. have.

In addition, according to the present invention, by using a filtering rule and a deep learning-based algorithm together, it is possible to analyze various types of queries obtained from a large number of commercial service logs.

1 is a block diagram of an apparatus for analyzing a query and its surrounding environment according to an embodiment.
2 is a flowchart of a method of selecting a candidate expected to be misclassified among input data according to an embodiment.
3 is a flowchart of a method of operating a language analyzer according to an embodiment.
4 is an exemplary diagram of a method of correctly interpreting the intention of a misclassified query according to an embodiment.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

The data used in this specification is a form in which a user's query and an intention of the query analyzed by a commercial system are attached with a label as shown in 410 to 430 of FIG. 4. In other words, it means that a query requested by a user to a commercial system such as a conversation system or a document analysis system and the intention of the query interpreted by the commercial system itself are combined. In this case, the commercial system is a program or device that performs a requested task by exchanging a conversation with a user, and may be, for example, an artificial intelligence speaker or the query analysis apparatus 1000 of the present invention.

In addition, when the commercial system interprets the query intention differently from the actual user's intention, it is called'misclassification', and the interpretation that suits the user's intention is called'correct classification'.

Hereinafter, we will look at how the query analysis device correctly grasps the intention of a misclassified query.

1 is a block diagram of an apparatus for analyzing a query and its surrounding environment according to an embodiment.

Referring to FIG. 1, the query analysis device 1000 interprets the input query using a training data generator 100, a learner 200 for training a query analysis model 210, and a learned query analysis model 210. It includes a query interpreter (300).

For the sake of explanation, the learning data generator 100, the learner 200, and the query interpreter 300 are called, but these are computing devices operated by at least one processor. Here, the learning data generator 100, the learner 200, and the query interpreter 300 may be implemented in one computing device or distributedly implemented in a separate computing device. When distributed in a separate computing device, the learning data generator 100, the learner 200, and the query interpreter 300 may communicate with each other through a communication interface. The computing device may be a device capable of executing a software program written to perform the present invention, and may be, for example, a server, a laptop computer, or the like.

Each of the training data generator 100, the learner 200, and the query interpreter 300 may be an artificial intelligence model or may be implemented as a plurality of artificial intelligence models. In addition, the query analysis model 210 may also be an artificial intelligence model, or may be implemented as a plurality of artificial intelligence models. The query analysis apparatus 1000 may be a single artificial intelligence model, or may be implemented as a plurality of artificial intelligence models. Accordingly, one or a plurality of artificial intelligence models corresponding to the above-described configurations may be implemented by one or a plurality of computing devices.

The training data generator 100 generates some of the data input to the query analysis device 1000 as training data, and includes a misclassification candidate extractor 110 and a language analyzer 120.

The misclassification candidate extractor 110 extracts a small amount of misclassification candidate queries that are expected to misinterpret the user's intention from the input large amount of data and uses them as training data of the query learning model 210.

The misclassification candidate extractor 110 generates a rule for filtering a large amount of data using the correctly classified query, and the final misclassification candidate query after an expert judges whether the correct classification query is included among the filtered false classification candidate queries. Select. The operation method of the misclassification candidate extractor 110 will be described in detail with reference to FIG. 2.

The language analyzer 120 linguistically analyzes the correctly classified query and the misclassified query, respectively, and extracts a core vocabulary necessary for the misclassified query to be correctly interpreted. Details of the detailed analysis method and core vocabulary will be described with reference to FIG. 3.

The training data generator 100 obtains a correct classification query with a correct intention label and a small number of misclassification candidates with an incorrect intention label from the misclassification candidate extractor 110, and learns by obtaining key vocabularies from the language analyzer 120. It is created with data.

The learner 200 learns the query analysis model 210 using the generated training data. The deep learning algorithm used at this time may be ELMo (Embedding from Language Models)-LSTM using a pretrained language model, and is not limited to any one.

The query interpreter 300 determines whether the intention label attached to the input query is correct using the learned query analysis model 210. That is, it is determined whether the intention label attached to the input user's query corresponds to the actual user's intention, and if the intention label is incorrect, the intention label obtained from the query interpreter 300 is changed.

Meanwhile, FIG. 1 assumes that a query in the form of an intention label attached is inputted through a query analysis process through a commercial system, but a query without an intention label may be input to the query interpreter 300 without going through a commercial system. have. In this case, the query interpreter 300 may perform an operation of newly attaching a correct intention label to the query.

2 is a flowchart of a method of selecting a candidate expected to be misclassified among input data according to an embodiment.

Referring to FIG. 2, the misclassification candidate extractor 110 classifies a query that frequently appears among a large amount of input data as a reference query and a query with a low appearance frequency as a control query (S101). For example, out of 1000 input data, if the first query occurs 980 times, the second query occurs 10 times, the third query occurs 5 times, and the fourth query occurs 5 times, the misclassification candidate extractor 110 1 Query is the reference query, and the second, third, and fourth queries are classified as control queries.

In this specification, a reference query means a query that is expected to be correctly interpreted in a commercial system and that an intent label appropriate for the user's intention is attached. That is, a reference query is a pattern that is frequently uttered by a user in probability, and is a query that can be patterned with the standard of the query.

The misclassification candidate extractor 110 extracts vocabulary and semantic information of the reference query and the contrast query to generate an embedding vector for each query (S102).

For example, the misclassification candidate extractor 110 is used to determine the vocabulary, morpheme/syntax analysis information, entity name, and the situation in which the query occurred in the reference query, the first query, the contrast query, the second query, the third query, and the fourth query. Information, history information, etc. are extracted to generate vocabulary/semantic information, and based on this, an embedding vector of each query is generated.

Specifically, the misclassification candidate extractor 110 may classify a noun, verb, etc. of a query through morpheme analysis, and grasp the structure of a query through syntax analysis.

In general, natural language processing is performed in the order of morpheme analysis, entity name recognition, and semantic domain determination. Morphological analysis refers to attaching parts of speech by analyzing words constituting a sentence into morphemes. Named entity refers to a word or phrase that has a specific meaning in a document, and entity name recognition refers to the process of classifying the categories of proper nouns extracted through morpheme analysis. Semantic Role refers to the role of a noun that appears in the relationship with a predicate in a sentence, and Semantic Role Determination is to recognize and classify the semantic relationship between the predicate contained in the sentence and the argument subject to the modification of the predicate. Even if each component of a sentence is arranged in a different structure, it can have the same semantic domain, or phrases arranged in the same structure can have different semantic domains.

The methods used for morpheme analysis, entity name recognition, and semantic domain determination are well known, and are known as Head-tail Classification, Tabular Parsing, Hidden Markov Model (HMM), Support Vector Machine (SVM), Conditional Random Field (CRF), and Maximum. Algorithms such as Entropy (ME) and Structural SVM can be used, and are not limited to any one.

For example, the misclassification candidate extractor 110 analyzes the morpheme for the example sentence “Play Captain Marvel.” to identify “Captain Marble/Chon-Common Noun, Play/Pronunciation-Verb”, and perform syntax analysis. Through this, it is possible to grasp the example sentence as a combination of “both words-common nouns + verbs-verbs.”

In addition, the meaning of each word or phrase can be grasped through the analysis of the entity name. For the example sentence "Play Captain Marble" above, you can label the meaning of each phrase as "Captain Marble/Movie, Play/Play".

In addition, the context information refers to information related to the operation of the commercial system at the time the query is uttered. For example, when the user utters a query to play a specific movie while watching TV, the context information may be extracted as “watching TV”. The history information may include information on a user's past utterances or a specific service to which the user has subscribed.

Thereafter, the misclassification candidate extractor 110 generates an embedding vector of each query using information such as the extracted morpheme, entity name, and semantic domain. An embedding vector refers to the transformation of categorical data into a continuous vector form.

As an example of generating a sentence embedding vector, the sentence2vec model developed by Google can be used. The sentence2vec model is a model that extends the CBOW (Continuous Bag Of Words) model of word2vec in units of sentences. Table 1 is an example of generating two random queries as N-dimensional vectors using the sentence2vec model.

Question 1 (0.00011 0.09193 -1.93188… 0.335567) Question 2 (0.00011 0.11171 -1.31652… 1.89287)

The misclassification candidate extractor 110 calculates a semantic similarity between the embedding vector of the reference query and the embedding vector of the contrast query (S103).

The method of calculating the semantic similarity between embedding vectors is not limited to any one. As an example, cosine similarity may be calculated for the transformed embedding vector. The cosine similarity is the cosine value of the angles of two vectors in the dot product space, and the degree of similarity between the two vectors is expressed as a value between 0 and 1, and the closer the cosine similarity is to 1, the more similar the two vectors are.

For example, the misclassification candidate extractor 110 calculates semantic similarity by comparing the embedding vector of the first query and the embedding vector of the second query, and the semantic similarity between the embedding vector of the first query and the embedding vector of the third query, and The semantic similarity between the embedding vector of the first query and the embedding vector of the fourth query is calculated.

The misclassification candidate extractor 110 generates a filtering rule using the calculated semantic similarity or the intention label of the reference query (S104). The filtering rule is used to select only a small number of queries among the matching queries, and a plurality of filtering rules may be used. For example, a rule for extracting a collation query having a semantic similarity greater than or equal to a certain value as a candidate for misclassification, or a rule for extracting a query with a specific intent label as a candidate for misclassification may be used.

As an example of a filtering rule, the semantic similarity between the embedding vector of the first query and the embedding vector of the second query is 0.83, the semantic similarity between the embedding vector of the first query and the embedding vector of the third query is 0.81, and the embedding of the first query. It is assumed that the semantic similarity between the vector and the embedding vector of the fourth query is 0.77. In this case, when the user or the administrator of the query interpretation apparatus 1000 sets the reference semantic similarity value to 0.8, only the second query and the third query having a value equal to or greater than 0.8 are selected as candidates for misclassification.

At this time, queries with a similarity of meaning less than or equal to the standard value are considered to have different meanings. Therefore, since the fourth query is a query having a different intention than the first query, it cannot be used as training data of the query analysis model 210 for attaching the intention label of the first query.

However, the fourth query may be used as training data when the semantic similarity with other random reference queries is 0.8 or higher.

As another example of a filtering rule, a user or an administrator of the query analysis apparatus 1000 may register a case in which different intention labels have the same intention for interpretation as a rule and register it in the query analysis apparatus 1000.

For example, if the intention label attached to the first query is A, and the intention label attached to the second query is B, although formally different, A and B are considered to have the same intention, the rule that A and B are the same. You can register. At this time, the second query with the intention label B attached is interpreted as an intention that fits the actual user's intention, and is excluded from misclassification candidates.

The erroneous classification candidate extractor 110 extracts a erroneous classification candidate from among a plurality of collation queries according to the generated filtering rule (S105). As the example described above, among a plurality of collation queries, the second query, the third query, and the fourth query, the fourth query with a semantic similarity of less than 0.8 is filtered, and the intent label of the first query and the second query is the same. The second query is filtered according to the rule. Therefore, only the third query is extracted as a candidate for misclassification from the four control queries.

After that, the expert checks the candidates for misclassification (S106). Verification of the final training data goes through the expert's judgment, and verification time can be reduced because the expert verifies only the candidates for misclassification that have been filtered once instead of verifying all of the input data.

An expert checks whether the misclassification candidates actually attach an intention label different from the user's intention (S107), and if all are misclassified, the misclassification candidates are selected as final misclassification candidates (S108), and correct classification to the misclassification candidates If the query is included, the filtering rule generated in S104 is modified by reflecting the expert's judgment result (S109).

The misclassification candidate extractor 110 transmits the reference query and the final misclassification candidates generated through the above process to the language analyzer 120. Hereinafter, a method of analyzing the difference between the linguistic characteristics of the reference query and the final misclassification candidates in order for the language analyzer 120 to generate data used for training the query interpretation model 210 will be described.

3 is a flowchart of a method of operating a language analyzer according to an embodiment.

Referring to FIG. 3, the language analyzer 120 analyzes a vocabulary semantic pattern of a reference query (S201). A vocabulary semantic pattern is a sequence created by dividing a sentence into vocabulary, parts of speech, and synonyms and adding semantic information. That is, sentences with one meaning can be expressed in various ways, which is a grammar that defines the types of syntax for these methods.

Therefore, if one vocabulary semantic pattern is constructed from the reference query, sentences of other expressions having the corresponding pattern can be interpreted as having the same meaning as the reference query.

For example, if the reference query is "Play Captain Marble," the vocabulary semantic pattern of the reference query is changed through morpheme analysis, syntax analysis, and entity name analysis as described with reference to FIG. It can be analyzed as'verb-regeneration'.

In addition, by using the'drama' that has a significant relationship with the'movie', the meaning pattern of the significance can be generated as'body language-common noun-drama + proverb-verb-reproduction'.

The language analyzer 120 performs morpheme analysis and syntax analysis of final misclassification candidate queries (S202). Morphological analysis refers to separating a sentence into a minimum semantic unit called morpheme, and parsing refers to analyzing the roles of each morpheme within a sentence, as described in FIG. 2.

The language analyzer 120 extracts a candidate of a core vocabulary that exists in the reference query but does not exist in the final misclassification candidates based on the morpheme and syntax analysis results of the final misclassification candidates (S203). A method of extracting the core vocabulary will be described in detail with reference to FIG. 4.

Thereafter, the language analyzer 120 calculates the reliability of each of the extracted core vocabulary candidates (S204). To calculate the reliability, BiLSTM-CRF, which combines bi-directional Long Short-Term Memory (LSTM) and (Conditional Random Field, CRF) method, can be used, and other models that have already been learned can be used.

BiLSTM is a model that uses two LSTMs, which are the deep learning models most commonly used in sequential data utilization, to consider not only the left (forward) but also the right (backward) data for each data. BiLSTM is an algorithm that shows high performance in natural language processing that must consider both front and rear contexts.

Meanwhile, based on the result of analyzing the vocabulary meaning pattern of the reference query, the reliability can be calculated by limiting the type of the entity name.

Thereafter, the language analyzer 120 extracts a core vocabulary having the highest reliability among candidates for the core vocabulary (S205).

The language analyzer 120 generates the extracted core vocabulary, a reference query, and a final misclassification candidate as learning data, and transmits the generated data to the learner 200. The learner 200 learns the query analysis model 210. The query interpretation model 210 may be implemented with the aforementioned ELMo-LSTM, and is not limited to any one.

Meanwhile, the language analyzer 120 may transmit the vocabulary semantic pattern of the reference query to the misclassification candidate extractor 110 and include it in the filtering rule generated by the misclassification candidate extractor 110 (S104).

4 is an exemplary diagram of a method of correctly interpreting the intention of a misclassified query according to an embodiment.

Referring to FIG. 4, the query “Play Captain Marvel” 410 occurs 1000 times, “Can you find Captain Marvel” (420), and “I wish you turn on the Marvel Captain” 430 each 5 times. Occurred, and the commercial system labels the intention of “Play Captain Marvel” (410) as SearchContent, the intention of “Can you find Captain Marvel” (420), FindContent, and the intention of “I want you to play Marvel Captain” (430). It will be described assuming the situation where is labeled as ILikeContent.

Meanwhile, although FIG. 4 shows that data used for learning is input to the query interpreter 300 to calculate a result, it is natural that the learning data and the data to be analyzed may be different.

The query analysis device 1000 classifies “Play Captain Marble” 410, which occurred 1000 times, as a reference query, and “Can you find the Captain Marble” that occurred 5 times, 420 and “I wish I turned on the Marvel Captain” (430 ) Is classified as a control query. In other words, it is assumed that the commercial system “Play Captain Marble” 410 is classified correctly, “Can you find Captain Marble” (420) and “Would you like to play Marvel Captain” (430).

The query analysis apparatus 1000 extracts vocabulary and semantic information of the reference query and the contrast query, and generates an embedding vector for the three queries using sentence2vec.

The query analysis apparatus 1000 calculates a semantic similarity between embedding vectors. As a result of calculating the cosine similarity, the meaning of "Play Captain Marble" (410) and "Can you find Captain Marble" (420) is calculated as 0.81, and if you turn on "Play Captain Marble" (410) and "Marble Captain," The meaning similarity of "I wish you" 430 can be calculated as 0.83.

On the other hand, SearchContent and FindContent are labels that provide a service called'search for TV content' and have the same meaning.

Accordingly, the query analysis apparatus 1000 may generate a filtering rule under a condition that a semantic similarity with a reference query is 0.8 or more and a condition that SearchContent and FindContent are the same intention, and extract a candidate for misclassification from the matching query.

"Can you find Captain Marvel" (420) and "I wish you play Marvel Captain" (430) have a similarity of meaning to the standard query of 0.8 or higher, and the intention label of "Play Captain Marble" (410) and "Captain Since the intention label of "Can you find Marvel" (420) is considered to be the same, "I want to play the Marvel Captain" (430) is extracted as a candidate for misclassification. On the other hand, the intention label of "Can you find Captain Marble" 420, which is excluded from the misclassification candidate, is maintained as it is.

After that, the expert finally judges that the intention label ILikeContent of “I want to play Marvel Captain” 430 is different from the actual user's intention.

The query analysis device 1000 performs a linguistic analysis process to extract the core vocabulary for changing the intention label of "I wish to play Marvel Captain" 430, which is the final misclassification candidate, from ILikeContent to SearchContent, which is the intention label of the reference query. do.

First, the vocabulary semantic pattern of the reference query "Play Captain Marble" 410 is analyzed. The query “Play Captain Marble” 410 may have a vocabulary semantic pattern as shown in Table 2.

+ Noun-content + Post + Play + Noun-content + Noun-none + Play + Noun-content + Post + Play

In Table 2, Noun-content is a combination of a noun and content name, Post is an abbreviation of Postposition, meaning investigation, Play is a verb meaning playback, and Noun-none is a separate category information as a result of analyzing the entity name. It means no nouns.

The query analysis apparatus 1000 may analyze the morpheme and syntax of the final misclassification candidate and generate a semantic pattern of “I wish to play the Marvel Captain” 430 as “Noun-none + Post + Play + Like”. Thereafter, the query analysis apparatus 1000 extracts a core vocabulary that exists in the reference query "Play Captain Marble" 410 but does not exist in the final misclassification candidate "Play Marvel Captain" 430. That is, candidates that can be Noun-none are extracted as an essential element in order for an arbitrary query to be correctly classified. In this case, as shown in Table 3, three key vocabulary candidates can be extracted.

Captain Marvel/Noun-content Captain/Noun-content + Marvel/Noun-none Captain/Noun-content

In Table 3, the query interpretation apparatus 1000 selects'Captain/Noun-content' as the core vocabulary by calculating the reliability of the three core vocabulary candidates with the language learning embedding model. The query analysis apparatus 1000 generates a query analysis model 210 by transmitting the selected core vocabulary to the learner 200.

Meanwhile, the query analysis apparatus 1000 may reflect a new rule that a'captain/noun-content' must be included in the filtering rule used in the process of extracting a candidate for misclassification, in order for a misclassified query to be correctly classified.

Through the above process, the query analysis model 210 is trained using the reference query, the final misclassification candidate, and the core vocabulary as learning data.

Using the query analysis model 210, the query interpreter 300 changes the intent label of the query “I wish I would play Marvel Captain” to SearchContent as shown in 430a, and consequently “Play Captain Marvel” and “Marble Captain”. It is interpreted as having the same intention of “I wish to play it on”.

Subsequently, the query analysis apparatus 1000 including the learned query analysis model 210 may be used as a part of a conversation system or a document analysis commercial service, and may output a higher-performance conversation analysis result to a commercial log.

In addition, the output value of the query analysis apparatus 1000 may be used again as an input value, and may be used for learning of the query analysis model 210.

According to the present invention, since a process of efficiently extracting a small number of candidates expected to be misclassified among queries determined in advance in a commercial service, speed and accuracy can be improved compared to a query analysis device targeting all misclassified queries. have.

In addition, according to the present invention, by using a filtering rule and a deep learning-based algorithm together, it is possible to analyze various types of queries obtained from a large number of commercial service logs.

The embodiments of the present invention described above are not implemented only through an apparatus and a method, but may be implemented through a program that realizes a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (10)

A method for interpreting a query by a computing device operated by at least one processor,
Receiving an input of queries labeled with an intention interpreted by an arbitrary device,
Classifying the input queries into control queries that are expected to be misclassified from a reference query, and analyzing a vocabulary pattern of the reference query,
Generating training data from the reference query, the collation query, and the vocabulary pattern, and
Including the step of training a query analysis model using the training data,
The matching queries are queries that appear with a frequency less than or equal to a predetermined value among the input queries. In claim 1,
Generating the learning data,
Generating a filtering rule using the semantic similarity of the reference query and the matching queries, and selecting misclassification candidates from the matching queries according to the filtering rule, and
And comparing the vocabulary pattern of the reference query with the vocabulary patterns of the misclassification candidates, and extracting core vocabularies that exist in the reference query and do not exist in the misclassification candidates. In paragraph 2,
The step of selecting the misclassification candidates,
Generating each embedding vector by linguistically analyzing the reference query and the control query,
Calculating a semantic similarity between the embedding vector of the reference query and each embedding vector of the reference query, and
And determining, as misclassification candidates, control queries having a calculated semantic similarity value equal to or greater than a predetermined value. In paragraph 3,
The filtering rule,
A method of interpreting a query, wherein a control query judged to have the same intention as the reference query is excluded from misclassification candidates. In paragraph 3,
After the step of determining the candidate for misclassification,
Receiving a determination result from an expert whether the intention labeled in the misclassification candidate matches the intention of an actual user, and
Referring to the result, when a query having the labeled intention as the actual user's intention is included in the misclassification candidate, further comprising the step of selecting the misclassification candidates again. In paragraph 2,
The step of extracting the core words,
A query interpretation method for extracting a vocabulary that is not recognized as an arbitrary entity name from among the vocabulary constituting the reference query. In claim 1,
After the learning step,
Inputting a new query into the learned query analysis model,
Determining the intention of the new query from the learned query interpretation model, and
When the intention labeled in the new query and the determined intention are different, changing the intention of the new query to the determined intention. As a computing device,
Memory, and
Including at least one processor to execute instructions (instructions) of the program loaded in the memory,
The above program,
Receiving an input of queries labeled with an intention interpreted by an arbitrary device,
Classifying the input queries into control queries that are expected to be misclassified from a reference query,
Generating an embedding vector of each of the reference query and the reference query, and calculating a semantic similarity between the embedding vector of the reference query and each embedding vector of the reference query,
Selecting as candidates for misclassification of collation queries having the calculated semantic similarity value equal to or greater than a predetermined value,
Analyzing the vocabulary pattern of the reference query and the misclassification candidates to extract core vocabularies that exist in the reference query and do not exist in the misclassification candidates,
Generating training data from the reference query, the misclassification candidates, and the core vocabulary, and
Learning a query analysis model using the training data
A computing device comprising instructions described to execute a. In clause 8,
The classifying step,
A computing device for classifying a query having an intention label that performs the same function as an intention label of the reference query among the input queries as correctly classified. In clause 8,
The step of extracting the core words,
A computing device for generating a semantic pattern in which a vocabulary constituting the reference query is replaced with a vocabulary having a significant relationship with the vocabulary.

Images