KR102477782B1 - Data processing method and apparatus for training neural networks classifing natural language intentions - Google Patents

Abstract

A data processing method performed by a computing device is disclosed.
According to an embodiment, a data processing method for a natural language used for training a neural network for classifying an intention of a natural language includes receiving first data that is a natural language used for learning a neural network for classifying an intention of a natural language, and a list of synonyms. and performing pre-processing on the first data based on at least one of a first database in which .

Description

Data processing method and apparatus for training neural network that classifies natural language intent

The embodiments below relate to a data processing method and apparatus for training a neural network that classifies natural language intent.

Recently, deep learning technology has been applied to various fields, and it has become a key technology leading the rapid development of voice recognition and natural language processing.

In order to train a neural network that classifies natural language intentions, natural language data in which each intention is labeled may be essentially required. Since the amount of natural language data to be labeled is enormous, a method to reasonably reduce it is required.

According to an embodiment, a method of processing natural language data used to train a neural network for classifying a natural language intent includes receiving first data that is a natural language used for learning a neural network for classifying a natural language intent; and performing pre-processing on the first data based on at least one of a first database storing a list of synonyms and a second database storing a list of idioms unrelated to natural language intent classification.

The performing of the preprocessing may include determining whether third data corresponding to a predefined synonym in the first database exists in the first data; and if the third data exists in the first data, replacing the third data with fourth data predetermined as a term representing the third data.

In the step of replacing the third data, when the 3-1 data and the 3-2 data corresponding to the synonym continuously exist, the 3-1 data and the 3-2 data are replaced with the first data. and adding the fourth data.

The performing of the preprocessing may include determining whether or not fifth data corresponding to a predefined term in the second database exists in the first data; and removing the fifth data from the first data when the fifth data exists in the first data.

The first database may be constructed based on a synonym determined by a similarity between vectorized individual elements and a central keyword whose number of generating words by combining with the synonym is equal to or greater than a predetermined threshold.

According to another embodiment, a data processing method for generating a feature word used for training a neural network for classifying the intent of a natural language includes a first feature word list including a first feature word that is a standard for labeling training data. receiving; extracting a first individual element constituting the first characteristic word based on part-of-speech information for each of the first characteristic words; extracting a second individual element constituting a word by being combined with the first individual element from the learning data; and generating a second feature word list using a second feature word generated based on a combination of the second individual element and the first individual element.

In the step of extracting the second individual element, the 1-1 individual element corresponding to the synonym among the first individual elements is a representative word previously determined to represent the 1-1 individual element. substituting individual elements; extracting a 2-1 individual element constituting a word by being combined with the representative word from the learning data; and extracting, from the learning data, a 2-2 individual element constituting a word by being combined with a 1-2 individual element that does not correspond to a synonym among the first individual elements, wherein the second feature word list The generating may include generating the second feature word list based on at least one of the first individual element, the 2-1 individual element, and the 2-2 individual element.

The data processing method according to another embodiment further includes extracting a third individual element included only in training data corresponding to a specific label from among individual elements included in the training data, and generating the second feature word list. In the step of generating, the second feature word list may be generated based on at least one of the first to third individual elements.

According to another embodiment, a data processing method for updating a feature word used for training a neural network for classifying the intention of a natural language includes extracting an existing feature word composed of a plurality of individual elements; determining one individual element among the plurality of individual elements as a current candidate feature; comparing a first tentative labeling result based on the current candidate feature word with an existing labeling result based on the existing feature word; and updating the existing feature word based on a comparison result.

The updating may include, when the number of training data whose corresponding labels have been changed in the first temporary labeling result exceeds a predetermined threshold value, separate elements continuously existing with the current candidate feature word from the existing feature word. updating the current candidate feature by adding to the current candidate feature; and updating the existing feature word based on a comparison result of a second temporary labeling result based on the updated current candidate feature word and the existing labeling result.

In the determining of the current candidate feature word, an individual element corresponding to a label having the same label may be determined as a current candidate feature among the plurality of individual elements.

According to an additional embodiment, a data processing method for processing feature words used for training a neural network for classifying intent of a natural language includes extracting complex intent feature words corresponding to a plurality of labels; and excluding any one of the plurality of labels from the label corresponding to the complex intent feature based on a predetermined condition.

A computing device performing a data processing method according to an embodiment includes a communication unit; and a processor, wherein the processor receives first data corresponding to a natural language used for training a neural network that classifies the intent of the natural language, and receives a first database in which a synonym list is stored or an idiomatic expression unrelated to the natural language intent classification. Second data may be generated by performing pre-processing on the first data based on at least one of the second databases in which the list is stored.

1A is a diagram schematically illustrating an operation of a natural language intent inference device.
1B is a diagram for explaining a learning method of a natural language intent inference device.
2 is a diagram for schematically explaining a data pre-processing method.
3 is a flowchart for explaining a data pre-processing process.
4 is a diagram for explaining a preprocessing operation in more detail.
5 is a flowchart for explaining a method of constructing a first database for storing the above-described synonym list.
6 is a diagram illustrating an example in which a computing device calculates a degree of correlation.
7 is a diagram illustrating an example in which a data processing method is implemented.
8 is a flowchart for explaining a data processing method for generating a feature word.
9 is a diagram illustrating an example in which a data processing method for generating a feature word is implemented.
10 is a diagram for explaining a data processing method of processing a feature word.
11A is a diagram for illustratively explaining a process of processing complex intent learning data.
11B is a diagram for illustratively explaining a process of processing complex intent learning data.
12 is a diagram for explaining a data processing method of updating a feature word.
13 is a diagram illustrating an example in which a data processing method of updating a feature word is implemented.
14 is a diagram illustrating an example in which a data processing method is implemented.
15 is a diagram showing the configuration of a computing device according to the present invention.

Specific structural or functional descriptions disclosed in this specification are merely illustrated for the purpose of describing embodiments according to technical concepts, and the embodiments may be implemented in various other forms and are limited to the embodiments described herein. It doesn't work.

Terms such as first or second may be used to describe various components, but these terms should only be understood for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may also be termed a first element.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle. Expressions describing the relationship between elements, such as "between" and "directly between" or "adjacent to" and "directly adjacent to" should be interpreted similarly.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

The embodiments may be implemented in various types of products such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices. Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements.

And throughout the detailed description and claims of the present invention, 'learning' or 'learning' is a term that refers to performing machine learning through procedural computing, which is a mental function such as human educational activity. It will be appreciated by those skilled in the art that it is not intended to refer to.

1A is a diagram schematically illustrating an operation of a natural language intent inference device.

The natural language intent inference device 110 may infer the intent of the input natural language 111 and output information about the inferred intent. The intention may refer to information about whether or not the natural language has a purpose. For example, the natural language intention inference device 110 may use an intention (eg, "problem-solving ability judgment") of an input (eg, "Describe an experience in which you achieved results despite unexpected changes") for individual questions of the self-introduction letter. ") can be printed.

The natural language intent inference device 110 according to an embodiment may be implemented to output an intent corresponding to an input natural language using deep learning technology. Deep learning is a machine learning technique for solving problems such as natural language recognition or speech recognition from big data sets. Deep learning can be understood as an optimization problem solving process that finds a point where energy is minimized while training a neural network using prepared learning data. Through supervised or unsupervised learning of deep learning, the structure of a neural network or weights corresponding to a model can be obtained, and through these weights, input data and output data can be mapped to each other. can

The natural language intent inference apparatus 110 according to an embodiment may include a neural network 112 trained by deep learning technology to infer the intent of a natural language. The trained neural network 112 may be generated through repeated training (learning) of a given initial neural network. Creating the trained neural network 112 may mean determining neural network parameters. Here, the parameters may include, for example, various types of data input/output to the neural network, such as input/output activations, weights, and biases of the neural network. As the iterative training of the neural network progresses, the parameters of the neural network can be tuned to compute a more accurate output for a given input.

The natural language intention inference device 110 corresponds to a computing device having various processing functions such as a function of inferring a natural language intention based on the learned neural network 112 . For example, the natural language intent inference apparatus 110 may be implemented in various types of devices such as a personal computer (PC), a server device, and a mobile device.

1B is a diagram for explaining a learning method of a natural language intent inference device.

The learning step shown in FIG. 1B may be performed by a neural network learning device (not shown), which trains (or learns) a neural network or retrains (learns) a neural network. It corresponds to a computing device having various processing functions such as functions to retrain. For example, a neural network learning apparatus may be implemented in various types of devices such as a personal computer (PC), a server device, and a mobile device.

As shown in the learning step, the trained neural network 112 loaded in the natural language intent inference device 110 generates an initial neural network 111 based on the training data 121 and the label 122 corresponding to the correct answer. can be created by learning. More specifically, the initial neural network 111 may be trained to output an intention corresponding to a predetermined label 122 corresponding to the training data 121 from the training data 121 corresponding to natural language. Although the learning data 121 is exemplified as an item included in a self-introduction letter, those skilled in the art will understand that the learning data is not limited thereto and may be extended to any natural language.

In the learning step, the neural network training apparatus may perform a preprocessing process (eg, noise removal and stopword removal, etc.) 130 on the training data 121 and then perform a feature word extraction process 140 . The feature word may refer to a criterion for classifying training data, that is, a term that is a criterion for labeling. The neural network training apparatus may expand a term of the training data with a synonym included in the thesaurus in the training data expansion process 150 . Synonyms may refer to terms having similar meanings. For example, if the training data 121 includes “Describe the experience of demonstrating teamwork”, the neural network training apparatus performs learning corresponding to “Describe the experience of exhibiting cooperation” through the process 150. data can be added. Based on the expanded training data, the initial neural network 111 may be trained to infer an intention corresponding to the training data 121 .

In the inference step, the natural language intention inference device 110 may infer an intention corresponding to the natural language input 171 through the learned neural network 112 . For example, the natural language intention inference device 110 may infer a question intention corresponding to an input of "Write about an experience in which a result was obtained despite an unexpected change." If the natural language intention inference device 110 infers the intention for the above input as "problem-solving ability", the inference may be determined to be successful, but if the intention is inferred as "community living ability", the inference may be determined to be unsuccessful. there is. If inference fails, the natural language input 171 may be added as new training data 172, and additional learning may be performed based on the added training data 172, thereby improving inference accuracy of the neural network.

As shown in the implementation example, the natural language intention inference device 110 may be applied to any area requiring natural language intention detection, such as a self-introduction letter writing mentoring engine.

In order to supervise a neural network, training data labeled with correct answers may be required. In addition, the amount of training data required for training of the neural network may be enormous. For example, in order to train a neural network that infers the intention of a natural language, training data in which the intention of individual natural language data is labeled may be required, and since labeling proceeds with human involvement, a very large amount of manpower may be required.

Therefore, in the process of generating training data, a process of reducing resources required for labeling by reasonably reducing labeling target data may be required.

2 is a diagram for schematically explaining a data pre-processing method.

The computing device 220 according to an embodiment may reduce labeling target data through preprocessing of original data to be used as training data, thereby providing a means of reducing resources required for labeling. More specifically, the computing device 220 links learning data with a first database 231 storing synonym information, a second database 232 storing idiom information, and a third database 233 storing stopword information. It is possible to perform preprocessing to generate.

The first database 231 may store information about synonyms including allophone synonyms and representative words representing each synonym. For example, “moment, motive, reason, reason” may be matched as an allophone synonym and stored in the first database 231, and “reason” as a representative word representing an allophone synonym may be previously determined and the first database ( 231) can be stored.

The second database 232 may store information on idioms (eg, description, description, composition) irrelevant to the determination of intention.

The third database 233 may store information on stopwords that do not have special meanings, such as special characters, repeated phrases, and postpositions.

The computing device 220 may reduce the labeling target data by deleting or replacing individual elements included in the original data using at least one of the first database 231 to the third database 233 . A more detailed operation of the computing device 220 may be described in more detail through the accompanying drawings below.

3 is a flowchart for explaining a data processing process.

Referring to FIG. 3 , the computing device may receive original data for generating learning data in step 311 . The original data may be a natural language used for training of a neural network that classifies the intention of the natural language. According to embodiments, original data may include self-introduction item data, but is not limited thereto, and those skilled in the art will understand that it may include arbitrary natural language data.

In step 312, the computing device may remove stop words from the original data. The computing device may remove the stopwords stored in the third database described above if they are included in the original data.

In step 313, the computing device may unify proper nouns included in the original data into the same form. For example, if the original data contains "Co., Ltd. A", "Co., Ltd. A", "Co., Ltd. B" and "Co., Ltd. B", the computing device changes "Co., Ltd." to "Co., Ltd." By modifying the elements "A" and "B" to "C", which is a predetermined element corresponding to the company name, the form of the company name, which is a unique meditation, can be expressed as "Co., Ltd. C". A person skilled in the art will understand that the process of correcting proper nouns in the same form is not limited to company names, and extends to correcting all proper nouns into one predetermined form for each type of proper noun.

In step 314, the computing device may extract an individual element corresponding to a part of speech that may be a characteristic word serving as a labeling criterion. For example, parts of speech that may be feature words may include nouns, verbs, adjectives, and adverbs, and extracted individual elements may be words corresponding to nouns, verbs, adjectives, and adverbs.

In operation 315, the computing device may perform primary filtering by removing redundant original data based on the extracted individual elements. For example, the computing device may determine whether each original data that has gone through the previous steps is duplicated based on the extracted individual element, and if there is duplicated original data based on the determination result, the duplicated original data may be removed. there is.

In operation 316, the computing device may perform preprocessing on the original data subjected to the first filtering based on the first database storing the list of synonyms or the second database storing the list of idioms. A more detailed manner in which step 316 is performed is described in more detail with reference to FIG. 4 attached below.

The computing device may remove spaces from each of the original data preprocessed in step 317 and perform secondary filtering through step 318. The secondary filtering may be an operation of deleting redundant data from the preprocessing result from which spaces are removed.

In step 319, the computing device may provide the original data subjected to the first filtering and the second filtering as labeling target data.

4 is a diagram for explaining a preprocessing operation in more detail.

According to an embodiment, the computing device may generate more simplified second data through pre-processing of first data that is a natural language used for learning of a neural network that classifies the intention of a natural language. For example, the first data may be original data subjected to the above-mentioned primary filtering, and the second data may refer to original data subjected to preprocessing described below.

Referring to FIG. 4 , the computing device may determine whether third data corresponding to a synonym exists in the first data in operation 411 . More specifically, the computing device may determine whether third data that is a synonym stored in the first database exists in the first data. When it is determined in step 411 that the third data exists in the first data, the computing device may determine whether the synonym is a continuous synonym through step 412 . A continuous synonym may refer to a term in which a plurality of synonyms having similar meanings are consecutively combined. More specifically, when 3-1 data and 3-2 data having similar meanings continuously exist in the first data in step 412, the computing device may determine that consecutive synonyms exist. For example, if the first data is "Describe your vision from a long-term or short-term perspective", "long-term" (data 3-1) and "short-term" (data 3-2), which are synonyms for the future, are Since they are continuously present in one data, the computing device may determine that the consecutive allophone synonyms are present in the first data.

When it is determined in step 412 that there is no continuous synonym, the computing device may replace third data corresponding to the synonym with fourth data previously determined as a representative word corresponding to the third data through step 413. there is. For example, in a situation where the first data is "describe the incident motive", the computing device may perform step 412 when the term "motivation" in the "entry motive" corresponds to the second data defined in the first database. ), the third data may be substituted with “reason” which is fourth data predetermined as a representative word for “motive”.

If it is determined in step 412 that the continuous synonym exists, the computing device removes the 3-1 data and the 3-2 data corresponding to the continuous synonym, and the 3-1 data and the 3-2 data It is possible to add predetermined fourth data as a representative of the first data. In the preceding example, the computing device replaces "long-term" and "short-term" with "future" predetermined to represent "long-term" (Data 3-1) and "short-term" (Data 3-2), Data corresponding to "future perspective" can be created.

In operation 415, the computing device may determine whether fifth data corresponding to a term defined in a second database storing an idiom list exists in the first data. If it is determined that the fifth data exists, the computing device can complete the pre-processing process by removing the fifth data (step 416). For example, the computing device may complete the preprocessing process by removing an element corresponding to “describe” corresponding to the fifth data from “describe a vision from a future perspective”.

The idiom list stored in the second database may be determined based on a predetermined individual element corresponding to the idiom among individual elements included in the original data. In addition, in the case of an individual element determined as a characteristic word by being combined with other individual elements, it may be excluded from the idiom list.

The data preprocessed through step 316 may be in the form of a combination of individual elements as the previous steps 312 to 315 progress, or may be in the form of original data. For example, if the original data is "Describe your vision from a long-term or short-term perspective", the data passed through step 416 may have the form of "future perspective vision" or "future perspective vision", and detailed implementation The method can be easily selected by a person skilled in the art.

In the data for which preprocessing has been completed through step 316, spaces may be removed through step 317, and redundant data may be removed through step 318. Data provided through step 318 may be provided as labeling target data through step 319 . Through the previous operation, the amount of original data is reduced, so that the data to be labeled can be significantly reduced, and the training data generated through the above method may not have a significant effect on the neural network learning performance.

5 is a flowchart for explaining a method of constructing a first database for storing the above-described synonym list.

Referring to FIG. 5 , in step 510, the computing device may determine synonym candidate data for constructing the first database. The computing device may determine synonym candidate data based on original data obtained through step 311 of FIG. 3 and original data subjected to first filtering through step 315 described above. More specifically, when individual elements (words) equal to or greater than a predetermined threshold number are included in the original data, original data including the individual elements may be determined as synonym candidate data. In addition, when the original data subjected to the first filtering includes individual elements equal to or greater than a predetermined threshold number, original data including the individual elements may be determined as synonym candidate data. In the process of determining the synonym candidate data, the computing device may check all individual elements or only individual elements corresponding to the characteristic word.

In step 520, the computing device may extract individual elements of the synonym candidate data. More specifically, the computing device may extract individual elements corresponding to nouns, verbs, adjectives, and adverbs from synonym candidate data. For example, if the synonym candidate data is determined to be "write the field of hope and reason for application" and "write the field of hope and reason for application", the computing device will separate each individual element (hope, field, support, reason) and (Hope, Sector, Support, Momentum).

In operation 530, the computing device may measure similarities between individual elements. For example, the computing device may measure the similarity between individual elements by vectorizing each individual element through the Word2Vec method and measuring the similarity between the vectors. For example, in the above example, “field” and “division” may be measured as words having high similarity, and “reason” and “moment” may be measured as words having high similarity. The Word2Vec method is presented as an example for explanation, and those skilled in the art will understand that similarity between individual elements can be measured by vectorizing individual elements through an arbitrary embedding scheme.

In step 540, the computing device may generate a first synonym list based on the previously measured similarity. The computing device may determine individual elements whose similarity exceeds a predetermined threshold as synonyms, and generate a first synonym list based on the determination result. For example, the computing device may match “field” and “department” as synonyms, match “reason” and “moment” as synonyms, and determine the first synonym list based on the matching result.

In step 550, the computing device may extract a central keyword based on the first synonym list. The central keyword may refer to an individual element whose number of times of constituting a word by combining with individual elements having a mutually synonymous relationship is equal to or greater than a predetermined threshold value. For example, "hope" whose number of times of constituting a word by combining with "field" and "section" determined as synonyms is equal to or greater than a predetermined threshold value may be determined as the central keyword. In the same way, "support" whose number of times of constituting a word by combining "moment" and "reason" corresponding to each other synonymously with each other is equal to or greater than a predetermined threshold value may be determined as a central keyword. The computing device may extract the central keyword based on the collocations included in the synonym candidate data and the first synonym list. Salmon may refer to a form in which individual elements are combined in order, and may be determined as salmon N1, salmon N2, etc. based on the number of combined individual elements. For example, if the original data is i) "Please write the desired field and reason for application" and ii) "Please write the desired field and application reason", the individual elements are i) "desire, field, support, reason" and ii) Can be extracted as "hope, sector, support, opportunity". In this case, salmon N1 can be extracted as "hope", "field", "division", "support", "reason", and "moment", and salmon N2 is "hope-field", "hope-division", It can be extracted as “support-reason” and “support-moment”. In the above situation, the computing device combines the synonyms "field" and "support" to determine "hope" as a central keyword in which the number of constituting words satisfies a predetermined condition, and the synonyms "reason" and "moment" “Support”, in which the number of times a word is formed by combining words, satisfies a predetermined condition, may be determined as a central keyword.

In step 560, the computing device may generate a second synonym list based on the central keyword. More specifically, the computing device may measure the degree of correlation between each central keyword and other individual elements included in the synonym candidate data, and determine an individual element whose correlation with the central keyword is equal to or greater than a predetermined threshold value as a new synonym. Correlation is an index indicating the association between two individual factors, and is determined by at least one of Mutual Information (MI), Pointwise Mutual Information (PMI), and Normalized Pointwise Mutual Information (NPMI). can An exemplary method for calculating the amount of mutual information is described in more detail with reference to FIG. 6 attached below.

In the previous example, “support” corresponding to the central keyword may be calculated to have a correlation of a predetermined threshold or higher with “motivation” and “reason” in addition to “reason” and “moment” determined as synonyms. The computing device may generate a second synonym list by determining “motivation” and “reason” for which a correlation between “support” and a threshold value or higher is calculated as synonyms for “reason” and “moment”.

In step 570, the computing device may determine a representative word for each synonym included in the second synonym list. For example, any one individual element among “reason,” “moment,” “motivation,” and “reason” defined as a synonym on the second synonym list may be determined as a representative word for the synonym. Depending on embodiments, an individual element appearing most frequently among synonyms may be determined as a representative word, but a method of selecting a representative word is not limited thereto.

In operation 580, the computing device may build a first database based on the second synonym list and the representative word determined for each synonym.

6 is a diagram illustrating an example in which a computing device calculates a degree of correlation.

Referring to FIG. 6 , as described in Equation 610, the degree of correlation may be calculated based on any one of the amount of mutual information, the amount of mutual information for each point, and the amount of mutual information for each normalized point.

As shown in the calculation method 620, the probability is calculated based on the number of items including each individual element (hope, occupation, etc.) on the synonym candidate data, and the amount of mutual information can be calculated based on the calculation result. there is.

The computing device may additionally derive a synonym based on the calculated amount of mutual information, and through this, a synonym list may be determined.

7 is a diagram illustrating an example in which a data processing method is implemented.

Referring to FIG. 7 , in the original data of the cover letter item to be used as learning data, stopwords are removed based on the stopword dictionary corresponding to the third database described above (step 312), and proper nouns are unified (step 313). )), a process of extracting parts of speech (step 314), and a first duplicate elimination process (step 315) corresponding to the first filtering may proceed. The original data for which the primary redundancy removal is completed may go through synonym substitution (step 413), successive synonym substitution (step 414), and idiom removal (step 416) processes (step 316). Additionally, secondary filtering (step 318) may be performed through a process of removing blanks (step 317), and provided as training data that is target data for labeling (step 319). In the secondary filtering, additional parts of speech extraction, synonym substitution, and idiom removal processes may be performed.

The first database corresponding to the thesaurus includes a process of determining vectorization target data corresponding to thesaurus candidate data based on repetitively appearing words (step 510), parts of speech extraction (step 520), similarity measurement and new It can be built through the process of constructing a synonym list based on the synonym (steps 530 to 580). As shown in the figure, the thesaurus may have a form in which "organization" corresponding to a representative word and "team", "organization", "organization", and "group" corresponding to synonyms are mapped.

Also, the idiom dictionary may correspond to the second database described above.

The first database corresponding to the thesaurus may be used in the previous pre-processing process (step 316) and the subsequent model training process.

8 is a flowchart for explaining a data processing method for generating a feature word.

Labeling may be performed based on feature words present in the original data. For example, in the original data "Describe your experience of solving community problems", "solving community problems" may be a feature word related to the label "problem solving ability" corresponding to the intention of the original data. Accordingly, a label corresponding to "problem solving ability" may be mapped to the original data.

A computing device according to an embodiment may provide a means for increasing inference accuracy of a neural network in the future by improving labeling accuracy by providing a method of increasing the number of feature words.

In operation 810, the computing device may receive a first feature word list including a first feature word serving as a labeling criterion. The first feature word list may include first feature words previously determined by a system designer. For example, "solving a community problem" corresponding to the first feature word and "problem solving ability" corresponding to the label. may be in this mapped form. The first feature word list may correspond to candidate feature word data for generating a final feature word list.

In operation 820, the computing device may extract a first individual element constituting the first characteristic word based on part-of-speech information (noun, adjective, adverb, verb) for each of the first characteristic words. For example, the computing device may extract "community", "task", and "solution" as first individual elements included in "solving community problem" corresponding to the first feature word.

In operation 830, the computing device may extract a second individual element that is combined with the first individual element in the learning data. For example, when the first individual element is “task”, the computing device may extract all individual elements combined with “task” in the learning data as second individual elements.

More specifically, the computing device may replace the 1-1 individual element corresponding to a synonym among the first individual elements with a representative word previously determined to represent the 1-1 individual element. there is. For example, when the 1-1 individual element is “task”, the computing device may replace “task” with a representative word “problem” based on the first database described above.

The computing device may extract the 2-1st individual element combined with the representative word from the learning data. For example, when the representative word is "problem", the computing device may extract all individual elements combined with "problem" in the learning data as second individual elements.

The computing device may extract, from the learning data, a second-second individual element combined with a first-second individual element that does not correspond to a synonym among the first individual elements.

In operation 840, the computing device may generate a second feature word list through a second feature word generated based on a combination of the first individual element and the second individual element. The computing device may generate the second feature based on any combination of the first discrete element and each second discrete element. For example, when the computing device derives "team", "community", "organization", "current problem", "face", and "solution" as second individual elements in response to the first individual element "problem". , the second feature word may be generated as "problem solving", "problem facing", "problem facing", "team problem", "community problem", and the like. In the previous example, it was suggested that a second individual element is combined with a first individual element to generate a second feature, but the embodiment is not limited thereto, and the number of second individual elements combined with the first individual element and The order is not limited, and it will be appreciated by those skilled in the art that the second feature may be generated based on any combination of the first individual element and the second individual element.

Additionally, the computing device may extract a third individual element included only in the training data corresponding to a specific label from among individual elements included in the training data, and generate a second feature word list based on the third individual element. For example, if an individual element called “task” included in the training data is included only in training data labeled with a label corresponding to “problem-solving ability,” the computing device may extract “task” as a third individual element. and a second feature word may be generated based on the third individual element. For example, “task” itself, which is a third individual element, may be determined as a second characteristic word. It will be appreciated by those skilled in the art that the third individual element is not limited to one word and may correspond to a combination of consecutive words. The computing device may derive a third individual element based on an individual element whose weight information given through the TF-IDF exceeds a predetermined threshold.

The second feature word list may be labeled based on a reviewer's review, and the neural network may be learned in a more accurate manner based on the second feature word list in which the number of feature words is extended compared to the first feature word list. there is.

9 is a diagram illustrating an example in which a data processing method for generating a feature word is implemented.

Referring to FIG. 9 , the computing device receives feature word candidate data corresponding to a first feature word list set by a system designer (step 810), extracts parts of speech corresponding to individual elements (step 820), and A synonym replacement process can be performed on the extracted individual elements. A person skilled in the art can easily implement the synonym replacement process based on the method using the synonym list stored in the first database described above.

A computing device may provide a second distinct element (eg, "team", "community", "organization", "at hand", "face") that is combined with a first distinct element (eg, "problem") in the learning data. , “solution”) may be extracted (step 830), and a synonym substitution process may be performed for the extracted individual elements.

The computing device may generate the second feature word list by generating a second feature word list based on a combination of individual elements (step 840) or by adding an individual element included only in a specific label as a new feature word.

A final feature word of the generated second feature word list may be determined by a reviewer's review and duplicate elimination.

10 is a diagram for explaining a data processing method of processing a feature word.

In step 1010, the computing device may extract a complex intent training data list corresponding to a plurality of labels from among the labeled training data. The compound intention training data may be generated in a situation in which two or more feature words corresponding to different labels are included in the training data, or feature words correspond to a plurality of labels.

In operation 1020, the computing device may modify the label of the compound intent training data by determining only one label among a plurality of labels corresponding to the compound intent training data as a label corresponding to the compound intent training data.

More specifically, the computing device may extract compound intention feature data from compound intent learning data. For example, the computing device may extract a feature word included in both the first label and the second label as a compound intent feature word.

In a situation where there are compound intent features corresponding to both the first label and the second label, and the second label is a label to which no other feature word is mapped other than the composite intent feature, the computing device converts the second label to a composite intent feature. It can be excluded from the label corresponding to the word. Through this, it may be determined that the compound intent feature corresponds only to the first label, and the compound intent learning data corresponds only to the first label.

In step 1030 , the computing device may provide a compound intent learning data list or a label modification result according to step 1020 to an external entity. A reviewer may modify a label of the compound intent training data or a label corresponding to each compound intent training data by reviewing the compound intent training data.

11A is a diagram for illustratively explaining a process of processing complex intent learning data.

11A (a) and (b) may include regions 1130 and 1160 corresponding to complex intent features belonging to both different labels. Complex-intent learning data including complex-intent features belonging to the regions 1130 and 1160 may correspond to both of the two different labels.

In the case of (a), since the second label 1120 includes an additional feature word in addition to the compound intent feature word, it may not be possible to modify the label for the composite intent feature word.

However, in the case of (b), since the fourth label 1150 does not include additional feature words, the computing device modifies the compound intent feature word included in the area 1160 to correspond only to the third label 1140. And, the corresponding complex intention learning data may be modified to correspond only to the third label 1140 .

According to an embodiment, the computing device may determine a corresponding label by applying a classification algorithm (eg, BR-SVM) to features that are not included in all of the labels 1110, 1120, 1140, and 1150. there is.

11B is a diagram for illustratively explaining a process of processing complex intent learning data.

Referring to FIG. 11B, as shown in (a), the complex intention learning data list may be continuously updated through comparison and modification by a reviewer (human). Also, as described above, it may be automatically modified by the computing device.

More specifically, as shown in (b), the ambiguous feature word (complex intent feature word) included in the compound intent learning data list is modified by a reviewer, so that the feature word data can be updated (step 1171). . For example, a composite intent feature may be modified to correspond to a single label by a reviewer. Also, as described above with reference to FIGS. 10 and 11A , the label of the complex intent feature may be automatically modified by the computing device.

Based on the updated feature word, reclassification of the training data may proceed (step 1172). As described above with reference to FIGS. 10 and 11A , the label may be automatically modified by the computing device or the label may be modified by a reviewer.

Based on the classification result, a compound intent learning data list may be regenerated. More specifically, the computing device may regenerate a compound intent training data list by extracting training data corresponding to different labels in duplicate. A modified compound intention learning data list may be generated by excluding the existing compound intention learning data from the regenerated compound intention learning data list (step 1173). By repeating steps 1171 to 1173, the complex intent learning data list and the composite intent feature word can be continuously updated.

12 is a diagram for explaining a data processing method of updating a feature word.

Referring to FIG. 12 , in step 1210, a third feature word composed of a plurality of individual elements may be extracted. For example, the third characteristic word may be a characteristic word composed of a plurality of individual elements (words), such as a characteristic word corresponding to “within a monetary remuneration schedule” corresponding to the label “social experience/school life”.

In operation 1220, the computing device may determine one individual element among a plurality of individual elements as a current candidate feature. More specifically, among a plurality of individual elements, an individual element corresponding to a label in which all feature words including individual elements are the same may be determined as a current candidate feature word. For example, in the case of "reward" among the four individual elements included in "within the monetary reward schedule", if all feature words including "reward" correspond to the same label, the computing device selects "reward" as the current candidate feature. can be decided by

In operation 1230, the computing device may compare a temporary labeling result based on the current candidate feature word with an existing labeling result based on the third feature word. For example, the result of labeling the training data through the current candidate feature corresponding to "remuneration" and the labeling of the training data through the third feature corresponding to "within the monetary reward schedule" results can be compared. The process of comparing results may be performed based on whether there is a variation in labeling for each training data.

In step 1240, the computing device may determine whether the comparison result satisfies a threshold condition. For example, the threshold condition may be whether the number of training data whose labeling is not changed exceeds a predetermined threshold.

When the number of training data whose labeling is not changed is less than a predetermined threshold, in step 1250, the computing device adds an individual element that continuously exists with the current candidate feature word in the third feature word to the current candidate feature word. Candidate features can be updated. For example, the computing device may add "schedule" or "money" to "reward" corresponding to the current candidate feature to determine "reward schedule" or "monetary reward" as a new current candidate feature in step 1250. there is. When the computing device further performs steps 1230 and 1240 and enters step 1250 again, "remuneration schedule" or "monetary reward" corresponding to the current candidate feature is set as "monetary reward schedule". , "within the maintenance schedule", etc. As such, the computing device may repeat the operation of step 1250 of adding individual elements one by one until a predetermined critical condition is satisfied in step 1240 and step 1260 is performed.

When the number of training data whose labeling is not changed exceeds a predetermined threshold, the computing device may update the third feature word based on the current candidate feature word through step 1260 . For example, "within a monetary reward schedule", which is the existing third characteristic word, may be modified to "reward".

13 is a diagram illustrating an example in which a data processing method of updating a feature word is implemented.

Referring to FIG. 13 , in step 1310, the computing device extracts a third feature word composed of a plurality of individual elements that may be referred to as feature collocations (step 1210), and individual elements included in the third feature word. (words) can be extracted. 1 gram on the drawing may mean that the current candidate feature word is composed of one individual element, and 2 gram and 3 gram may mean that there are two and three individual elements, respectively.

In operation 1320, the computing device may determine, among the extracted individual elements, an individual element corresponding to a label in which feature words including the individual elements are all the same as the current candidate feature (operation 1220). For example, the feature words related to "remuneration" referred to as 1 gram words "monetary pay schedule", "within monetary pay schedule", "within pay schedule", "monetary pay schedule", "monetary pay schedule", "remuneration" Since both "received schedule" and "rewarded within schedule" correspond to the same label, the computing device may determine "reward" as the current candidate feature.

In operation 1330, labeling based on the current candidate feature word is performed on the training data, so that a temporary labeling result may be generated. Labeling may be performed on deduplication data (data for which the above-described secondary filtering has been completed) corresponding to training data from which duplicates have been removed. In this case, in order to improve the accuracy of the result, labeling may be performed on data in which synonyms for individual elements included in the deduplication data are substituted. More specifically, the computing device replaces the synonym of "compensation" with respect to data including the synonym of "compensation", such as "compensation payment schedule" and "compensation payment schedule" among deduplication data ("compensation" is replaced by "remuneration"). Substituting with ") and labeling the synonym replacement result, a temporary labeling result may be generated. An existing labeling result compared with the temporary labeling result may also be determined based on the same synonym replacement process.

In step 1340, the computing device may compare the temporary labeling result with the existing labeling result (step 1230). In step 1350, the computing device may replace the third feature word with the current candidate feature word when the ratio of training data whose labeling is not changed is equal to or greater than a predetermined criterion (step 1260). That is, when the current candidate feature word is not included in the training data of labels other than the corresponding label, the current candidate feature word can replace the third feature word. Otherwise, the current candidate feature word is the third feature word. cannot be substituted, and a larger number of individual elements can be updated to a combined term and the comparison procedure repeated. In step 1350, the computing device may update the current candidate feature word when the ratio of training data whose labeling is not changed is less than a predetermined criterion (step 1250). It may be the same way as described step 1250. For example, the current candidate feature "remuneration" may be updated with "remuneration schedule".

A series of processes of performing temporary labeling by updating current candidate feature words may be repeated until the temporary labeling result satisfies the predetermined condition described above.

In step 1340, processing of the unlabeled learning data may be additionally performed. More specifically, if an individual element included in the unlabeled training data is an individual element not included in an existing feature word, it may be extracted as a new feature word, or if there is a classification error for the individual element, it may be corrected.

In the foregoing description, it has been described as a method of performing temporary labeling by increasing individual elements for one collocation and updating a current candidate feature, but the embodiment of the present invention is not limited thereto, and may be performed in units of a plurality of data sets. It will be appreciated by those skilled in the art. For example, in step 1310, a plurality of parallel candidate feature words (“reward”, “a little bit”) composed of one individual element may be derived, and temporary labeling based on the plurality of candidate feature words may be performed; A feature update procedure may be performed based on comparison between the temporary labeling result and the existing labeling result. In this case, in the case of the temporary labeling result related to the candidate feature word corresponding to "reward", even if there is no difference from the existing labeling result, when the change in the temporary labeling result for the candidate feature word related to "slightly" is large, each candidate feature The process of updating with candidate feature words corresponding to 2 grams ("monetary compensation", "remuneration schedule", ??, "slight improvement"??) and comparing the temporary labeling result with the existing labeling result can be additionally performed. there is.

14 is a diagram illustrating an example in which a data processing method is implemented.

Referring to FIG. 14 , among training data that has been labeled based on feature words, complex intention learning data is a data processing process in which ambiguous features are modified through a complex intention processing process (the process described above with reference to FIGS. 10 to 11B ). may proceed. In addition, in the case of learning data without complex intention, a new feature word may be extracted through the new feature word extraction process described above with reference to FIG. 9 .

The unlabeled training data may be further classified based on a classification algorithm (eg, BR-SVM).

The feature words determined through the previous process may be updated through the data processing method described above with reference to FIGS. 12 and 13 referred to as Ngram optimization.

15 is a diagram showing the configuration of a computing device according to the present invention.

Referring to FIG. 15 , a computing device 1500 according to an embodiment includes a processor 1520 . The computing device 1500 may further include a memory 1510 and a communication interface 1530 . The processor 1520, memory 1510, and communication interface 1530 may communicate with each other through a communication bus (not shown).

According to an embodiment, the processor 1520 may perform data processing on natural language used for learning of a neural network that classifies the intent of the natural language.

The processor 1520 receives first data that is a natural language used for learning a neural network that classifies the intent of the natural language, and receives at least one of a first database storing a list of synonyms and a second database storing a list of idioms unrelated to the classification of natural language intent. Based on one, preprocessing may be performed on the first data.

According to another embodiment, the processor 1520 may generate a feature word used for training of a neural network that classifies the intention of a natural language. The processor 1520 receives a first feature word list including a first feature word that is a reference for labeling of training data, and configures the first feature word for each first feature word based on part-of-speech information. Extracts a first individual element that is combined with the first individual element from the training data to extract a second individual element constituting a word, and a second feature generated based on the combination of the second individual element and the first individual element. The second feature word list may be generated through the word.

According to another embodiment, the processor 1520 may perform data processing to update a feature word used for learning of a neural network that classifies the intent of a natural language.

The processor 1520 extracts an existing feature word composed of a plurality of individual elements, determines one individual element among the plurality of individual elements as a current candidate feature word, and determines a first temporary labeling result based on the current candidate feature word and An existing labeling result performed based on the existing feature word may be compared, and the existing feature word may be updated based on the comparison result.

According to an additional embodiment, the processor 1520 may perform a data processing method of processing a feature word used for learning a neural network that classifies the intention of a natural language.

The processor 1520 may extract compound intent features corresponding to a plurality of labels and exclude any one label among the plurality of labels from labels corresponding to the composite intent features based on a predetermined condition.

Memory 1510 may be volatile memory or non-volatile memory.

In addition, the processor 1520 may execute a program and control the computing device 1500 . Program codes executed by the processor 1520 may be stored in the memory 1510 . The computing device 1500 may be connected to an external device (eg, a personal computer or network) through an input/output device (not shown) and exchange data. The computing device 1500 may be mounted on a server.

The computing device 1500 may perform the data processing method described above with reference to FIGS. 1 to 14 through the processor 1520 .

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (4)

A data processing method for updating a feature word that is a criterion for labeling in training for a neural network that classifies the intention of a natural language, performed by a computing device, comprising:
extracting existing feature words composed of a plurality of words;
determining one of the plurality of words as a current candidate feature word;
comparing a first tentative labeling result performed based on the current candidate feature word with an existing labeling result performed based on the existing feature word; and
Updating the existing feature word based on the comparison result
Including, data processing method.
According to claim 1,
The updating step is
In the first tentative labeling result, when the ratio of training data whose corresponding label has been changed exceeds a predetermined threshold, a word continuously existing with the current candidate feature word in the existing feature word is added to the current candidate feature word. updating the current candidate feature word by doing so;
Updating the existing feature word based on a comparison result of a second temporary labeling result based on the updated current candidate feature word and the existing labeling result
Including, data processing method.
According to claim 1,
The step of determining the current candidate feature word,
and determining, among the plurality of words, a word corresponding to a label in which all feature words including the word are identical, as a current candidate feature word.
A computer readable storage medium storing one or more programs including instructions for performing the method of claim 1 .

Images