KR102456513B1 - Data augmentation processing system using the generative model and methods therefor - Google Patents

Abstract

A data augmentation processing system using a generation model and a method thereof are disclosed. The present invention can reduce the cost invested in the verification process which occurs when a text data augmentation technique is used in a natural language processing model. The data augmentation processing system includes: a collection unit; a pre-verification unit; a data generation unit; a verification unit; and a restoration unit.

Description

Data augmentation processing system using the generative model and methods therefor

The present invention relates to a data augmentation processing system using a generative model and a method therefor, and more particularly, to a data augmentation processing system using a generative model that can lower the cost of an inspection process that occurs when a text data augmentation technique is used. and methods thereof.

In general, a natural language processing model is a model in which a computer implements syntactic and semantic representations that can be understood by humans.

This natural language processing model uses a word embedding technology that expresses lexical meaning by arranging similar words close to each other on a vector space using a language model based on a neural network.

The language model is a model that calculates the probability that the next word will appear based on previous words in a given sentence. can

The natural language processing model as described above performs prior learning using a large amount of data collected using a crawling technique (a technique of collecting texts existing on a web page in large quantities).

And, after the prior learning of the natural language processing model, the text specified to fit the purpose of the natural language processing model is learned, and fine-tuning is performed so that the model meets the purpose.

At this time, the collection of texts specified to meet the above purpose is directly collected by a separate manpower, and since the collection of texts is made directly by manpower, additional costs are required, and furthermore, such manpower is accompanied by experts in the field. The disadvantage is that the cost of manpower is set high.

In order to solve this disadvantage, a text data argmentation technique using a generative model is used.

The generative model is a model that performs natural additional learning data generation by learning a character pattern that is difficult for a human to describe piecemeal. There is a model.

The GPT model and the BERT model are language models capable of pre-training and transfer learning, the GPT model is capable of unidirectional learning, and the BERT model is a natural language processing task in both directions. It is a pre-trained model.

1 is a diagram for explaining a text data augmentation technique using a conventional generative model.

Referring to the drawing, in order to operate a generative model (2) such as a pre-trained GPT model or a BERT model using a large amount of text data (1) to meet a specific purpose, the collected data ( 3) is taught to the generative model 2, and fine-tuning is performed according to the purpose.

Then, in the future, the fine-tuned natural language processing model can generate the result data 3 - 1 meeting a specific purpose from the collected data 3 .

This text data augmentation technique has good productivity in natural language processing because it does not require a separate manpower to collect text suitable for the purpose, but since the generative model 2 itself does not have a human-level knowledge base, The detail (detail) of 3-1) is often insufficient or incomplete.

For example, in the case where the detail of the result data 4 is insufficient or incomplete, the result data 3-1 is irrational or rules Deviations from this occur frequently.

That is, in the case of address data, unconventional result data of “Gwangjin-gu, Incheon, Gyeonggi-do” is generated, or in case of phone number data, result data that deviates from the rules of “Contact: 110-9920-222” is generated.

Therefore, when the result data 3-1 is generated using the generation model 2, the result data with poor detail is removed from the result data 3-1 by the inspector V of a separate manpower, and the final result The verification process for deriving the data (3-2) is necessarily accompanied.

Therefore, even in the method of using the text data augmentation technique, additional costs are incurred because the inspection process of the result data has to be performed by an inspector of a separate manpower. There is a need to develop a method to lower

On the other hand, as prior art related to the above-described natural language processing model, Korean Patent Application Laid-Open No. 10-2020-40652, a description of a natural language processing system and a word expression method in natural language processing, and Korean Patent Application Laid-Open No. 10-2020-110400 The technique of learning data augmentation policy and the technique of a deep learning-based emotional speech synthesis apparatus and method using the generation model-based data augmentation technique of Korean Patent Application Laid-Open No. 10-2020-44337 are known.

1. Korean Patent Laid-Open Publication No. 10-2020-40652 (Title of the invention: natural language processing system and word expression method in natural language processing) 2. Korean Patent Application Laid-Open No. 10-2020-110400 (Title of Invention: Learning Data Augmentation Policy) 3. Republic of Korea Patent Publication No. 10-2020-44337 (Title of the invention: Deep learning-based emotional speech synthesis apparatus and method using generative model-based data augmentation technique)

In order to solve this problem, an object of the present invention is to provide a data augmentation processing system and method using a generative model capable of lowering the cost of an inspection process that occurs when a text data augmentation technique is used in a natural language processing model. do it with

An embodiment of the present invention for achieving the above technical problem is a data augmentation processing system using a generative model, the collection unit for storing the collected data collected for learning the generative model in a collection data database; a pre-examination unit for generating de-identified data by performing a pre-inspection of de-identifying a portion of the collected data with a preset de-identification token; a data generation unit that performs fine-tuning learning based on the de-identified data on the generation model, and performs text data augmentation for a specific domain from the generation model that has performed fine-tuning learning to generate verification data; an inspection unit for deriving result data based on inspection information input from an inspector for the generated inspection data; and a restoration unit that generates final data obtained by restoring the de-identification token included in the result data based on preset rule information and stores it in the result data database.

In addition, the generative model according to the embodiment is characterized in that it has been previously learned using arbitrary text data.

In addition, the pre-inspection unit according to the embodiment specifies at least any one or more of information that is difficult for a generation model to recognize among the contents of the collected data, information having regularity, and information having an exposure risk requiring security for specific information. It is characterized in that de-identified data is generated by de-identifying it with a token.

In addition, the de-identification token according to the embodiment is characterized in that it is restored with random information.

In addition, an embodiment of the present invention provides a data augmentation processing method using a generative model, comprising: a) collecting, by a collecting unit, collected data for learning a generative model and storing it in a collection data database; b) generating de-identified data by performing a pre-examination in which the pre-examination unit de-identifies some of the collected data with a preset de-identification token (token); c) the data generation unit performs fine-tuning learning based on the de-identified data of the generation model, and performing text data augmentation for a specific domain from the generation model on which the fine-tuning learning is performed to generate verification data; d) extracting result data based on the inspection information input from the inspector for the inspection data generated by the inspection unit; and e) generating, by the restoration unit, final data restored by the de-identification token included in the result data based on preset rule information, and storing it in the result data database.

In addition, among the contents of the de-identified data collection data of step b) according to the above embodiment, at least among the information difficult to recognize by the generation model, information having regularity, and information having a risk of exposure requiring security for specific information It is characterized in that any one or more pieces of information are de-identified with a specific token.

In addition, the generation model of step c) according to the above embodiment is characterized in that it has been previously learned based on a plurality of text data collected from one or more external web pages accessed through the communication network N.

In addition, the de-identification token of step e) according to the embodiment is characterized in that it is restored with random information.

The present invention has the advantage of reducing the cost of inspection when generating data.

In addition, the present invention has an advantage in that the performance of the generative model can be further improved by de-identifying parts that the generative model does not understand.

In addition, the present invention has the advantage of more effectively utilizing GPU resources by reducing the overall text length through de-identification.

In addition, the present invention has the advantage of being able to prevent the problem of personal information leakage in advance when the non-identified part is set as personal information.

1 is a conceptual diagram illustrating a text data augmentation technique using a conventional generative model.
2 is a conceptual diagram illustrating a text data augmentation technique using a generative model of the present invention.
3 is a block diagram of a text data augmentation processing system using a generative model of the present invention;
4 is a flowchart of a text data augmentation processing method using a generative model of the present invention;

Hereinafter, the present invention will be described in detail with reference to preferred embodiments of the present invention and the accompanying drawings.

Prior to describing the specific contents for carrying out the present invention, it should be noted that components not directly related to the technical gist of the present invention are omitted within the scope of not disturbing the technical gist of the present invention.

In addition, the terms or words used in the present specification and claims have meanings and concepts consistent with the technical idea of the invention based on the principle that the inventor can define the concept of an appropriate term to best describe his invention. should be interpreted as

In this specification, the expression that a part "includes" a certain element does not exclude other elements, but means that other elements may be further included.

Also, terms such as “… unit”, “… group”, and “… module” mean a unit that processes at least one function or operation, which may be divided into hardware, software, or a combination of the two.

In addition, the term "at least one" is defined as a term including the singular and the plural, and even if the term at least one does not exist, each element may exist in the singular or plural, and may mean the singular or plural. will be self-evident.

In addition, that each component is provided in singular or plural may be changed according to an embodiment.

Hereinafter, a preferred embodiment of a data augmentation processing system and method using a generative model according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

2 is a conceptual diagram illustrating a text data augmentation technique using a generative model of the present invention.

Referring to Figure 2, the text data augmentation technique using the generative model of the present invention, by performing a pre-examination process of de-identifying a part of the original text of the collected data, the time required for the inspection of the result data derived by the generative model and It is a method that can save manpower.

Referring to the drawing, in order to operate a generative model (2) such as a pre-trained GPT model or a BERT model using a large amount of text data (1) to meet a specific purpose, the collected data ( 5) is taught to the generative model 2 and fine-tuning is performed according to the purpose.

At this time, the text data augmentation technique of the present invention performs a pre-examination of the collected data 5 collected for the learning of the generative model 2 .

Specifically, the prior inspection de-identifies information that is difficult to recognize by the generation model 2 in the contents of the collected data 5 or information with a degree of exposure risk (the degree to which security is required for information) as a specific token. It is an operation to generate unidentified and de-identified data (6).

For example, the contents of the collected data (5) above are “Donghae Mountain Heavy Snow Warning and Cold Wave Warning have come into effect, so please do your best for facility management and traffic safety. In the case of “Donghae City (Contact: 033-530-2119, 033-530-2119)”, deleting at least any one or more of information that is difficult to recognize, information with regularity, and information with a risk of exposure , Change the above content to “Donghae Mountain Heavy Snow Warning and Cold Wave Warning has come into effect, so please do your best for facility management and traffic safety (non-identification part 0)”.

Here, the de-identified part of the original content is the part of “Donghae City (Contact: 033-530-2119, 033-530-2119)”. The portion was de-identified as “de-identified portion 0”.

For another example, if the content of the collected data (5) is “Today around 13:24, a rooftop fire in 1101 Eunbit Village, 961 Hwajeong-dong, Goyang-si. By changing the content to “Today around 13:24 (Unidentified Part 1) rooftop fire occurred. Please detour this area and be aware of safety accidents for nearby residents.” 1101" is de-identified as "de-identification unit 1" to generate de-identified data 6 .

Furthermore, in addition to content that the model is difficult to understand, such as a phone number or an address, de-identification of a specific text area can be performed, and through this, the generation model 2 can be fine-tuned in various ways.

In addition, when creating an open domain chatbot with memory, if an area containing memorable information is de-identified and learned in the text, post-processing can be performed more actively by using the de-identification token.

As described above, the generation model 2 learns the de-identified data 6 generated by de-identifying a part of the contents of the collected data 5 to perform fine adjustment according to the purpose.

Here, as described above, the generative model is a known generative model such as the GPT model and the BERT model.

Then, the generation model 2 that has undergone fine adjustment generates the verification data 7 using the de-identified data 6 .

In this case, the de-identified data 6 de-identifies at least any one or more of information difficult to recognize by the generation model 2, regular information, and information with exposure risk as a specific token. As the (unidentified) data, the generation model (2) generates the verification data (7) using the de-identified data (6) including these de-identified tokens.

Next, when the inspection data 7 is generated, the inspection data 7 based on the inspection information input by the inspector V is lacking in detail (detail) or made incompletely among the inspection data 7 ) can be selectively removed to extract the result data (8).

For example, in the case of lack of detail or incompleteness of the inspection data 7, for a sentence in which regularity is determined, such as an address or phone number included in the inspection data 7, the inspection data 7 is nonsensical or rules in case it goes out

In addition, an example of such a case is that in the case of address data, unconventional result data of “Gwangjin-gu, Incheon Special City, Gyeonggi-do” is generated, or in the case of phone number data, inspection data (7) that deviate from specific rules such as “Contact: 110-9920-222” ) is generated.

At this time, the above inspection data (7) is generated using the collection data (5) containing the de-identification token. The difficulty in performing the inspection by V) is formed to be low, so the time and manpower required for the inspection can be reduced, and further, the quality of the result data (8) can be improved.

Next, the final data 9 may be generated by restoring the de-identification token included in the result data 8 based on preset rule information.

As an example of such restoration, the above-mentioned collected data (5) is “Donghae Mountain Heavy Snow Warning and Cold Wave Warning have been in effect, so please do your best for facility management and traffic safety. If the de-identified part of the contents of “Donghae City (Contact: 033-530-2119, 033-530-2119)” is “Donghae City (Contact: 033-530-2119, 033-530-2119)”, this de-identified The phone number part is restored based on a preset restoration rule.

For example, "non-identification unit 0", which de-identifies a phone number, can be restored to an arbitrary random phone number or "xxx-xxx-xxxx".

In addition, the contents of the collected data (5) were de-identified from the contents of "Today around 13:24, a rooftop fire in 1101, Eunbit Village, 961, Hwajeong-dong, Goyang-si, Goyang-si. If the part is "1101-dong, Eunbit Village, 961 Hwajeong-dong, Goyang-si", the unidentified address part can be restored with arbitrary random address information or "aaa-city bbb-dong xx apartment".

Next, a system and method for performing text data augmentation processing using the generative model of the present invention as described above will be described.

3 is a block diagram of a text data augmentation processing system using the generative model of the present invention.

Referring to FIG. 3 , the text data augmentation processing system using the generation model of the present invention includes a collection unit 10 , a dictionary inspection unit 20 , a data generation unit 30 , an inspection unit 40 , The restoration unit 50 may be included.

In addition, the text data augmentation processing system using the generative model operates a generative model 31 such as a GPT model or a BERT model previously learned using a large amount of text data to meet a specific purpose. In one or more external web pages 100 accessed through (N), it is possible to collect a large amount of text data using a crawling technique or the like.

The collection unit 10 receives the collection data 3 related to a specific domain for learning the generative model 31 and stores it in the collection data database 200 .

The pre-examination unit 20 generates de-identified data 6 by performing a pre-examination of de-identifying a portion of the collected data 3 with a preset de-identification token (token).

The de-identified data 6 is at least any one of information that is difficult for a generation model to recognize in the contents of the collected data 5, information having regularity, and information with a risk of exposure, which is the degree to which information security is required. It is data in which the above information is de-identified with a specific token.

As described above, the portion that is de-identified using the de-identification token may include, for example, address information or phone number information with regularity, specific information that is difficult for a model to understand, and information with a risk of exposure. have.

The data generator 30 performs fine-tuning learning based on unidentified data 6 using the generative model 31 , and texts for a specific domain from the generative model 31 on which the fine-tuning learning is performed. Data augmentation may be performed to generate inspection data (7).

The generation model 31 may perform prior learning based on a plurality of text data collected from one or more external web pages accessed through the communication network N, and may be a natural language learning model such as GPT or BERT.

The inspection unit 40 may extract the result data 8 based on the inspection information input from the inspector with respect to the inspection data 7 generated by the data generating unit 30 .

That is, if the inspection unit 40 is insufficient or incomplete detail of the inspection data 7, for example, the inspection data ( 7) is nonsensical or out of the rules, and an example of such a case is when, for example, nonsensical data such as address data “Gwangjin-gu, Incheon, Gyeonggi-do” is generated.

In addition, the derivation of the result data 8 can be performed by the inspector V as in the prior art, but the content of the inspection data 7 itself is difficult to recognize, information with regularity, information with a risk of exposure Since it does not have at least any one or more of the information, the difficulty in performing the inspection through the inspector (V) is significantly lowered, thereby reducing the time and manpower required for the overall inspection, and through this, the result data (8) Quality can also be improved.

The restoration unit 50 generates the final data 9 restored by the de-identification token included in the result data 8 based on preset rule information and stores it in the result data database 300 .

4 is a flowchart of a text data augmentation processing method using a generative model of the present invention.

3 and 4 , in the text data augmentation processing method using a generative model according to an embodiment of the present invention, the collection unit 10 collects the collection data 5 for learning the generative model 31 . (S1), and the collected data is stored in the collection data database (200).

The pre-inspection unit 20 generates de-identified data 6 by performing a pre-inspection of de-identifying some of the collected data 5 collected in step S1 with a preset de-identification token (token) ( S2) can be done.

In the content of the collected data 5 in the step S2, at least any one or more of information that is difficult to recognize by the generation model, information with regularity, and information with a degree of exposure risk, which is the degree to which information security is required, is added to a specific token ( token) to generate unidentified data.

Subsequently, the data generator 30 may perform fine-tuning learning based on the de-identified data 6 using the generation model 31 ( S3 ).

The fine-tuning learning in step S3 is to learn so that the generation model 31 can perform data augmentation for a specific domain by using the collected data for the specific domain.

The generative model may be a known generative model such as a GPT model and a BERT model.

In addition, the data generation unit 30 generates (S4) verification data 7 by performing text data augmentation for a specific domain from the generation model 31 on which fine-tuning learning has been performed.

The inspection unit 40 extracts the result data 8 based on the inspection information input from the inspector for the inspection data 7 generated in step S4 (S5).

That is, when the verification unit 40 augments the desired text data using the fine-tuning generation model 31, the generation model 31 performs data augmentation including the de-identification token, so The result data (8) can be selected by performing inspection on the part except for the identification token.

The restoration unit 50 generates the final data 9 in which the de-identification token included in the result data 8 is restored based on preset rule information in the result data 8 extracted in step S5 ( S6) and the generated final data 9 is stored in the result data database 300 .

Accordingly, the data augmentation processing system and method using the generation model can reduce the cost of inspection when generating data.

In addition, the performance of the generative model can be further improved by de-identifying the part that the generative model does not understand.

In addition, since the overall text length is reduced through de-identification, GPU resources can be more effectively utilized.

In addition, if the de-identified part is set as personal information, the problem of personal information leakage can be prevented in advance.

As described above, although described with reference to the preferred embodiment of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

In addition, the reference numbers described in the claims of the present invention are only described for clarity and convenience of description, and are not limited thereto, and in the process of describing the embodiment, the thickness of the lines shown in the drawings or the size of components, etc. may be exaggerated for clarity and convenience of explanation.

In addition, the above-mentioned terms are terms defined in consideration of functions in the present invention, which may vary depending on the intention or custom of the user or operator, so the interpretation of these terms should be made based on the content throughout this specification. .

In addition, even if it is not explicitly shown or described, a person of ordinary skill in the art to which the present invention pertains can make various modifications including the technical idea according to the present invention from the description of the present invention. Obviously, this still falls within the scope of the present invention.

In addition, the above embodiments described with reference to the accompanying drawings have been described for the purpose of explaining the present invention, and the scope of the present invention is not limited to these embodiments.

10: Collector
20: Pre-Inspection Department
30: result data generation unit
40: inspection department
50: restoration unit
100: external web page
200: collected data database
300: result data database

Claims (8)

a collection unit 10 for storing the collection data 5 collected for learning the generation model 31 in the collection data database 200;
Performing a pre-inspection of de-identifying some of the collected data 5 with a preset de-identification token (token) to generate de-identified data 6,
Among the contents of the collected data 5, one or more information among information that is difficult to recognize by the generation model 31, information having regularity, and information with a risk of exposure requiring security for specific information is converted to a specific token. a pre-checking unit 20 for generating de-identified data 6 by de-identification;
The generative model 31 performs fine-tuning learning based on the de-identified data 6, and de-identification including a de-identification token for a specific domain from the generative model 31 on which fine-tuning learning is performed. Data generation unit 30 for generating verification data (7) by performing text data augmentation based on the data (6);
Inspection unit 40 for deriving result data (8) based on inspection information input from the inspector for the generated inspection data (7); and
The de-identification token included in the result data 8 is restored using any one of random numbers and characters based on preset rule information to generate the final data 9 and store it in the result data database 300 A data augmentation processing system using a generative model comprising a; The method of claim 1,
The generative model 31 is a data augmentation processing system using a generative model, characterized in that it has been previously learned using arbitrary text data. delete delete a) collecting, by the collecting unit 10, collecting data 5 for learning of the generating model 31 and storing the collected data 5 in the collected data database 200;
b) The pre-inspection unit 20 generates de-identified data 6 by performing a pre-inspection that de-identifies some contents of the collected data 5 with a preset de-identification token (token), but the collected data Among the contents of (5), one or more information among information difficult to recognize by the generation model 31, information having regularity, and information having a risk of exposure requiring security for specific information is de-identified as a specific token. generating de-identified data (6);
c) the data generation unit 30 performs fine-tuning learning based on the de-identified data 6 on the generative model 31, and de-identification of a specific domain from the generative model 31 on which the fine-tuning learning is performed generating verification data (7) by performing text data augmentation based on de-identified data (6) containing a currency token;
d) extracting the result data (8) based on the inspection information input from the inspector with respect to the inspection data (7) generated by the inspection unit 40; and
e) The restoration unit 50 generates the final data 9 restored by using any one of random numbers and characters based on preset rule information for the de-identification token included in the result data 8, and results Data augmentation processing method using a generative model comprising; storing in the data database (300). delete 6. The method of claim 5,
The data augmentation processing method using a generative model, characterized in that the generation model 31 in step c) is pre-learned based on a plurality of text data collected from one or more external web pages accessed through the communication network N. delete

Images