JP2023018624A - Data generation method using language model, computer device, and computer program - Google Patents

Abstract

To provide a data generation method, a computer device, and a computer program that use a language model.SOLUTION: A data generation method using a language model comprises the steps of: constructing a prompt that is an input sentence of a language model using original data; and inputting the prompt into the language model and generating new data and label information for the new data from the language model.SELECTED DRAWING: Figure 3

Description

There is an application for exception to loss of novelty

The following description relates to techniques for generating text data.

In order to build an NLP (natural language processing) model, data necessary for model learning is secured through data design and data collection. After learning the model based on the secured data, it repeats the improvement work of data design and model learning while evaluating the performance.

Since the emergence of the pretrained language model (PLM), fine tuning has been applied using an NLP model that solves a specific domain or task based on the PLM.

Furthermore, data augmentation techniques have been used to efficiently train NLP models.

As an example, technology that superficially manipulates phrases to generate text data for model learning, such as EDA (easy data augmentation), has a low grammatical nature of the generated sentences, and the meaning of the sentences does not correspond to the intended meaning. There may be large differences between

As another example, a technique for generating similar sentences using a machine translation model like retranslation (Back-translation) utilizes a translation model learned by a translation corpus with specific linguistic characteristics, There is a disadvantage that the generated writing style cannot reflect the linguistic characteristics of the existing text data (which is the object of generation), and general-purpose use is impossible.

To provide a data augmentation technology capable of deriving a natural language generation result by using a large-scale language model trained by a corpus of various language characteristics and extracting new data from the derived generation result.

A data generating method performed by a computing device, said computing device including at least one processor configured to execute computer readable instructions contained in a memory, said data generating method comprising said at least one a processor constructing a prompt that is an input sentence of a language model using original data; and the at least one processor inputting the prompt to the language model and generating new data from the language model. and generating label information for the new data.

According to one aspect, the generating may include generating soft labels representing label distributions using a probability distribution for natural words corresponding to labels in the prompt.

According to another aspect, the data generation method may further include the at least one processor selecting the original data from a training data set according to semantic diversity of text.

According to another aspect, the selecting step may select the original data corresponding to the number of label types from the learning data set.

According to yet another aspect, the configuring step may configure the prompt in a format that includes a text type and a label type.

According to another aspect, the constructing step may construct the prompt in a format including a text type, a label type, and a label verbalizer.

According to another aspect, the constructing step may process the original data to construct the prompt in a natural language form having the same format as the original data.

According to yet another aspect, the constructing step may construct the prompt by combining a task specification, the original data, and a prompt template.

According to another aspect, the generating step is an auto-regressive method of transferring a probability distribution of previous tokens to natural words corresponding to text and labels in the prompt as input of the next token. extracting the new data and the label information using .

According to still another aspect, the step of extracting is performed by heuristic beam search to extract the probability of the upper part of the probability distribution of the previous token from the input of the next token. and transmitting as

A computer program is provided for causing the computer device to execute the data generation method.

A computing device comprising at least one processor configured to execute computer readable instructions contained in a memory, wherein the at least one processor utilizes original text data to serve as input sentences for a language model. A computing device is provided for processing the steps of constructing a prompt, inputting the prompt into the language model, and generating new data and label information for the new data from the language model.

According to the embodiment of the present invention, the language model is used to transform or expand the original data to generate new data, and the knowledge recognized by the language model is transferred through the new data to collect data. The input man-hours can be significantly reduced, and the effect of model weight reduction can be achieved.

It is a block diagram for explaining an example of an internal configuration of a computer device in one embodiment of the present invention. FIG. 4 is a diagram for explaining the concept of text extension using a large-scale language model in one embodiment of the present invention; 4 is a flow chart showing an example of a data generation method that can be executed by a computer device in one embodiment of the present invention; FIG. 4 is a diagram for explaining the process of constructing an input prompt for a language model in one embodiment of the present invention; FIG. 4 is a diagram for explaining a data extension process in one embodiment of the present invention; FIG. 4 is a diagram for explaining a data extension process in one embodiment of the present invention; FIG. 4 is a diagram for explaining the process of generating new sentences and label information using a language model in one embodiment of the present invention; FIG. 4 is a diagram for explaining the process of generating distributed soft labels in an embodiment of the present invention; FIG. 4 is a diagram showing an example of original data and an example of new data generated from the original data in one embodiment of the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiments of the present invention relate to text data generation technology using language models.

Embodiments including the matters specifically disclosed in this specification are consistent with the characteristics of existing text data by using a large-scale language model, and have high grammaticality and naturalness. You can generate sentences. In addition, high quality label information can also be generated for the relevant sentences.

FIG. 1 is a block diagram illustrating an example of a computing device in one embodiment of the invention. For example, the data generation system according to the embodiment of the present invention may be realized by computer device 100 shown in FIG.

As shown in FIG. 1, computer device 100 includes memory 110, processor 120, communication interface 130, and input/output interface 140 as components for executing the data generation method according to the embodiment of the present invention. good.

The memory 110 is a computer-readable storage medium and may include random access memory (RAM), read only memory (ROM), and permanent mass storage devices such as disk drives. Here, a permanent mass storage device such as a ROM or disk drive may be included in computer device 100 as a separate permanent storage device separate from memory 110 . Also stored in memory 110 may be an operating system and at least one program code. Such software components may be loaded into memory 110 from a computer-readable medium separate from memory 110 . Such other computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, memory cards, and the like. In other embodiments, software components may be loaded into memory 110 through communication interface 130 that is not a computer-readable medium. For example, software components may be loaded into memory 110 of computing device 100 based on computer programs installed by files received over network 160 .

Processor 110 may be configured to process computer program instructions by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 120 by memory 110 or communication interface 130 . For example, processor 120 may be configured to execute received instructions according to program code stored in a storage device, such as memory 110 .

Communication interface 130 may provide functionality for computer device 100 to communicate with other devices over network 160 . As an example, requests, commands, data, files, etc. generated by processor 120 of computer device 100 in accordance with program code recorded in a recording device such as memory 110 may be sent to others via network 160 under the control of communication interface 130 . device. Conversely, signals, instructions, data, files, etc. from other devices may be received by computing device 100 through communication interface 130 of computing device 100 over network 160 . Signals, instructions, data, etc., received through communication interface 130 may be transmitted to processor 120 and memory 110, and files, etc., may be stored in a recording medium (permanent recording device described above) that computing device 100 may further include. may be recorded.

The communication method is not limited, and not only the communication method using the communication network that can be included in the network 160 (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network), but also the short distance between devices. Wireless communication may be included. For example, the network 160 includes a PAN (personal area network), a LAN (local area network), a CAN (campus area network), a MAN (metropolitan area network), a WAN (wide area network), a BBN (broadband network), and the Internet. Any one or more of the networks may be included. Additionally, network 160 may include any one or more of network topologies including, but not limited to, bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks, and the like. will not be

Input/output interface 140 may be a means for interfacing with input/output device 150 . For example, input devices may include devices such as microphones, keyboards, cameras, or mice, and output devices may include devices such as displays, speakers, and the like. As another example, the input/output interface 140 may be a means for interfacing with a device that integrates functions for input and output, such as a touch screen. Input/output device 150 may be one device with computing device 100 .

Also, in other embodiments, computing device 100 may include fewer or more components than the components of FIG. However, most prior art components need not be explicitly shown in the figures. For example, computing device 100 may be implemented to include at least some of the input/output devices 150 described above, and may also include other components such as transceivers, cameras, various sensors, databases, and the like. It's okay.

The large-scale language model used in the present invention refers to a language model that can be inferred without fine-tuning using a method such as Few-shot Learning (FSL). and has more than 10 times more parameters (for example, more than 100 billion parameters) compared to conventional general language models. For example, large-scale language models such as GPT-3 (Generative Pre-trained Transformer 3) and HyperClova are excellent Few-shot learners that can be controlled by natural text prompts, and are capable of processing small amounts of data by prompts. In-context learning is possible, which is the ability to understand patterns and solve NLP problems with only one person.

This embodiment relates to a new data augmentation technique that utilizes a large-scale language model to generate new data from original data. In addition, it is possible to leverage soft labels predicted by the language model to effectively distill knowledge in the large-scale language model while simultaneously generating textual perturbations.

FIG. 2 is a diagram for explaining the concept of text extension using a large-scale language model in one embodiment of the present invention.

Referring to FIG. 2, in this embodiment, a large scale language model 210 is used as the backbone for generating the synthetic text data required for model training.

According to the present embodiment, the large language model 210 can be used to generate synthetic yet hyper-realistic new data from the original data.

When there is text data in a form with or without a label, the corresponding data is converted into a prompt input sentence in a natural language form, and the converted prompt input sentence is given to the language model 210 as an input to generate the subject natural language. You can derive results. The derived results may be analyzed to extract new data, where the new data is in the same form as the original data, with or without a label. The extracted new data is added to the original text data to be used for data collection, or the performance of the model can be improved by generating a model based on the corresponding data.

In other words, the present embodiment aims at data augmentation and uses a prompt-based approach to generate new data in the large-scale language model 210. The new data inspired by the original data and the soft labels predicted by the large language model 210 can be used to train a small classification model, knowledge distillation can be achieved.

FIG. 3 is a flow chart illustrating an example of a data generation method that can be executed by a computing device in one embodiment of the present invention.

The data generation method according to this embodiment may be executed by the computer device 100 described above. In this case, processor 120 of computing device 100 may be implemented to execute control instructions by the code of the operating system and the code of at least one program contained in memory 110 . Here, the processor 120 controls the computer device 100 so that the computer device 100 executes steps 310 to 340 included in the data generation method of FIG. you can

The data generation method according to the present embodiment can generate extremely fluent new data based on data distribution.

Referring to FIG. 3, at step 310, processor 120 may select original data to use for prompts from the data set.

In the following, the method of selecting original data will be described by taking data generation for a text classification task as an example. Given a classification task T, the training data set is a set of text x and label y pairs

となる。

becomes.

As an example, the processor 120 may randomly select k original data from the learning data set D. The processor 120 may select k original data using a uniform distribution (Equation (1)).

The number of original data, k, may be determined in consideration of cost and performance.

As another example, the processor 120 may select original data from the training data set D by considering the semantic diversity of the text. Semantic diversity is a measure of how diverse the meaning of a text is. If the semantic diversity is low, the generated data has a high degree of similarity to the original of the existing dataset, so the data expansion effect is low. increases.

As a method for calculating semantic diversity, a vector of each text is extracted using a sentence vector representation method (eg, bag-of-words, aggregate word2vec, BERT embedding, BLEURT, etc.). After calculating the distance between vectors using similarity (e.g., cosine similarity, etc.), or using a network such as BLEURT to calculate the pairwise semantic distance, the distance is Sampling may be performed by giving higher weights to distant (high semantic distance) pairs.

The processor 120 may select original data corresponding to the number of classes, that is, the types of labels, from the learning data set D. FIG. The processor 120 may preferentially select data having a high semantic distance for each pair of original data with the same label. An α (alpha) multiplier may be applied to the semantic distance, where α corresponds to the hyperparameter that needs to be optimized.

At step 320, the processor 120 may compose a prompt appropriate for inputting the language model. The processor 120 may use the original data selected in step 310 to construct a language model input prompt. The processor 120 may create a dedicated prompt template appropriately reflecting the characteristics of the given NLP problem, and the prompt template may include the definition of the corresponding task and meta information. In other words, the processor 120 may process the original data selected from the data set to construct a prompt in natural language form, where the prompt is produced in a format that can be understood by the language model, Given as input for the language model. The processor 120 designs the prompt such that when the original data is labeled data, the input sentence is generated along with the label information.

The processor 120 extracts the original data sampled from the learning data set D

が与えられるとき、説明ヘッダ（ｄｅｓｃｒｉｐｔｉｏｎ ｈｅａｄｅｒ）、テキスト－ラベル対リスト、拡張接頭辞（ａｕｇｍｅｎｔａｔｉｏｎ ｐｒｅｆｉｘ）で構成されたプロンプトを生成してよい。プロンプトは、言語モデルがデータ分布に対してさらに適切に一般化することが可能なように各タスクの情報を有しており、このようなタスク表示子（ｔａｓｋ ｉｎｄｉｃａｔｏｒ）は、タスクごとに固有であり、課題のメタ情報を提供してよい。

may generate a prompt consisting of a description header, a text-label pair list, and an augmentation prefix. The prompt carries information for each task so that the language model can generalize better to the data distribution, and such task indicators are unique for each task. Yes, and may provide meta information for the issue.

The format of the prompt itself may be configured in various ways, but as an example, the prompt can check the text type (eg, review, article, etc.), the label type (eg, sentiment, classification, etc.), and the label position. A label-token verbalizer may be included.

The text type T is a metatype of the input text x, and for example, the text type corresponds to movie review in movie review sentiment analysis. The label type L is a metatype of the label class y, and for example, in the movie review sentiment analysis, the label type corresponds to emotion. label-token block v

においてプロンプトを公式化するためには、ラベルクラス

To formulate the prompt in the label class

と言語モデルの語彙

and language model vocabulary

で単語トークン間の１対１マッピングが必要となる。

requires a one-to-one mapping between word tokens.

The three pieces of meta information mentioned above are the task specification

を構成する。各タスクＴは、プロンプトを公式化することのできる課題仕様

configure. Each task T is a task specification that can formulate prompts.

を必要とする。プロセッサ１２０は、基本的に、一般タスク仕様である

need. The processor 120 is basically general task specific

を使用してプロンプトを生成してよい。ここで、Ｉは、クラスラベルが語彙

may be used to generate prompts. where I is the class label

に存在すると仮定する識別関数（ｉｄｅｎｔｉｔｙ ｆｕｎｃｔｉｏｎ）を意味する。

We mean the identity function that we assume exists in .

In short, as shown in FIG. 4, for a given task T, the processor 120 considers the task specification 410, the example data 420, which is the original data sampled from the training data set 400, and the characteristics of the given task T. The prompt templates 430 may be combined to construct the language model input prompt.

As can be seen from the specific example of FIG. 5, processor 120 utilizes task specification 410, data example 420, and prompt template 430 to organize and input the language model in a format understandable by the language model. A prompt 540 may be created. An example task specification 410 is shown in Table 1 and an example prompt template 430 is shown in Table 2. As data example 420

が与えられる場合、入力プロンプト５４０は表３のように構成されてよい。

, the input prompt 540 may be constructed as in Table 3.

Referring again to FIG. 3, at step 330, processor 120 may input the prompts constructed at step 320 into the language model and generate natural language containing the new data from the language model. In other words, the processor 120 may obtain the language generation result by the completion function of the language model after inputting the prompt input sentence into the language model.

At step 340, processor 120 may analyze the natural language generation results to extract new data. The language model may generate natural language according to the prompt pattern given as an input sentence, and may extract new data by pattern analysis of the generated natural language.

As shown in FIG. 6, the processor 120 inputs the prompt 540 into the language model 210 and analyzes the natural language patterns generated based on the language model 210 to generate extended data 650 (new sentences, corresponding label information) pairs of sentences may be obtained. As an example, processor 120 may obtain the label distribution using language modeling probability distributions of natural language tokens corresponding to labels in prompt 540 .

In the case of the prompt-based approach method, the extended text x' and the label y' are continuously generated as natural language text after the prompt. A predefined prompt template based on the sampled original data may extract each value by pattern matching to provide an input sentence for the language model to generate a (x', y') structure. Also, joint text and label generation ensures that the generated text is concatenated to the correct label.

The prompt design of this embodiment is

に該当するラベルトークンがテキストｘ以後に生成されるように保障する。プロセッサ１２０は、言語モデルを利用して疑似ラベリング（ｐｓｅｕｄｏ－ｌａｂｅｌｉｎｇ）を実行してよく、拡張テキストｘ’のソフトラベル確率を得るためにラベル－トークンを生成する可能性（ｌｉｋｅｌｉｈｏｏｄ）を正規化してよい。

is generated after the text x. Processor 120 may perform pseudo-labeling utilizing the language model, normalizing the likelihood of generating label-tokens to obtain soft label probabilities for the extended text x′. good.

The pseudo-label probability that the extended text x' is labeled with the label y' is given by Equation (2).

ここで、

here,

は、言語モデリング可能性（ｌａｎｇｕａｇｅ ｍｏｄｅｌｉｎｇ ｌｉｋｅｌｉｈｏｏｄ）を示し、

denotes the language modeling likelihood,

は、与えられたタスク仕様を構成する関数である。

is a function that composes a given task specification.

In this embodiment, text perturbation, pseudo-labeling, and knowledge distillation can be effectively combined in a single expansion task. In practice, pseudo-labeled new data are trained with the original data using cross-entropy loss.

Referring to FIG. 7, the processor 120 may generate new data based on the probability distribution according to the completion function of the language model 210. At this time, the token corresponding to the label in the pattern analysis is generated based on the label-token segment. Probabilities may be used to generate soft labels, ie labels with a distribution.

More specifically, the processor 120 converts all natural language provided as prompt input sentences into a particular tokenization scheme (e.g., constant form index, etc.) and then inputs them to the language model 210. . In order to obtain a (new sentence, label information of the corresponding sentence) pair, the probability distribution of the previous token is transmitted as the input of the next token based on autoregressive. The top n probabilities are used as input for the next token by a beam search using a heuristic. By using the top n probabilities among the joint probabilities obtained by multiplying the own probability distribution of each token by the probability distribution of the previous token, a sequence with a high probability value can be extracted.

For example, as shown in FIG. 8, processor 120 may construct soft labels using the probability of tokens falling under the label. In other words, soft labels may be extracted from the normalized label token distribution predicted by language model 210 .

Referring to FIG. 9, when data example 1 (910) classified as positive and data example 2 (920) classified as negative are given as original data, new data are obtained from the language generation result using the language model 210. 930 may be extracted.

Processor 120 analyzes the results generated by the prompts and extracts synthesized new data and label information. By repeating this process several times, it becomes possible to obtain diverse data and accurate classification information that were not originally present in the data. It can be used to fine-tune NLP models. The new data 930 is output data generated by appropriately referring to the linguistic and structural features of the two given original data 910, 920, and the appropriate new data 930 is mixed with positive and negative. By doing so, it becomes completely new data that does not exist in the past.

Adding new data 930 to the training data set, ie data augmentation, can improve the final classifier performance.

The original data 910 and 920 may be generated as hard-labeled data with a single label attached, and the new data 930 may be generated as soft-labeled data with at least two or more label distributions.

Since a soft label can be converted into a hard label by max operation, the original data 910 and 920 and the new data 930 can all be used in hard label form. Since the learning process uses a loss function such as cross entropy, both hard label and soft label forms can be utilized.

Thus, according to the embodiment of the present invention, extended data is generated by transforming or extending existing data based on the language model, and knowledge recognized by the language model is transferred through the extended data. As a result, the number of man-hours input for data collection can be significantly reduced, and the effect of reducing the weight of the model can be achieved.

The apparatus described above may be realized by hardware components, software components, and/or a combination of hardware and software components. For example, the devices and components described in the embodiments include processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, Or may be implemented using one or more general purpose or special purpose computers, such as various devices capable of executing and responding to instructions. The processing unit may run an operating system (OS) and one or more software applications that run on the OS. The processor may also access, record, manipulate, process, and generate data in response to executing software. For convenience of understanding, one processing device may be described as being used, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. You will understand. For example, a processing unit may include multiple processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include computer programs, code, instructions, or a combination of one or more of these, to configure a processor to operate at its discretion or to independently or collectively instruct a processor. You can Software and/or data may be embodied in any kind of machine, component, physical device, computer storage medium, or device for interpretation by, or for providing instructions or data to, a processing device. good. The software may be stored and executed in a distributed fashion over computer systems linked by a network. Software and data may be recorded on one or more computer-readable recording media.

The method according to the embodiments may be embodied in the form of program instructions executable by various computer means and recorded on a computer-readable medium. Here, the medium may continuously record the computer-executable program or temporarily record it for execution or download. In addition, the medium may be a variety of recording means or storage means in the form of a combination of single or multiple hardware, and is not limited to a medium that is directly connected to a computer system, but distributed over a network. It may exist in Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc., and may be configured to store program instructions. Other examples of media include recording media or storage media managed by application stores that distribute applications, sites that supply or distribute various software, and servers.

As described above, the embodiments have been described based on the limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations based on the above description. For example, the techniques described may be performed in a different order than in the manner described and/or components such as systems, structures, devices, circuits, etc. described may be performed in a manner different from the manner described. Appropriate results may be achieved when combined or combined, opposed or substituted by other elements or equivalents.

Accordingly, different embodiments that are equivalent to the claims should still fall within the scope of the appended claims.

210: Language model 220: PLM

Claims (20)

A data generation method executed by a computer device, comprising:
The computing device includes at least one processor configured to execute computer readable instructions contained in memory;
The data generation method includes:
the at least one processor constructing a prompt that is an input sentence of a language model using original data; and the at least one processor inputting the prompt into the language model to convert the language model into generating new data and label information for said new data from. The generating step includes:
2. The data generation method of claim 1, further comprising: generating soft labels representing label distributions using a probability distribution for natural words corresponding to labels in the prompt. The data generation method includes:
3. The data generating method of claim 1 or 2, further comprising: selecting the original data from a training data set according to the semantic diversity of the text by the at least one processor. The selecting step includes:
4. The data generation method according to claim 3, wherein the original data corresponding to the number of label types are selected from the learning data set. The configuring step includes:
The data generation method according to any one of claims 1 to 4, characterized in that the prompt is configured in a format including a text type and a label type. The configuring step includes:
The data generation method according to any one of claims 1 to 4, characterized in that the prompt is constructed in a format including a text type, a label type, and a label verbalizer. The configuring step includes:
The data generation method according to any one of claims 1 to 4, wherein the original data is processed to compose the prompt in the same natural language form as the original data. The configuring step includes:
The data generation method according to any one of claims 1 to 4, wherein the prompt is constructed by combining a task specification, the original data, and a prompt template. The generating step includes:
extracting the new data and the label information using an auto-regressive method in which the probability distribution of the previous token is transferred as the input of the next token to the natural language corresponding to the text and label in the prompt; A data generation method according to any one of claims 1 to 8, comprising the step of: The extracting step includes:
10. The data of claim 9, comprising communicating, as an input for the next token, a probability of an upper part of the probability distribution of the previous token by a beamsearch using a heuristic. generation method. A computer program for causing a computer device to execute the data generation method according to any one of claims 1 to 10. A computer device,
at least one processor configured to execute computer readable instructions contained in memory;
The at least one processor
A process of constructing a prompt as an input sentence of a language model using original data, and a process of inputting the prompt into the language model and generating new data and label information for the new data from the language model. , computer equipment. The at least one processor
13. The computer according to claim 12, wherein a soft label indicating label distribution is generated using a probability distribution for natural words corresponding to labels in the prompt. The at least one processor
14. The computer device according to claim 12 or 13, wherein said original data is selected from a learning data set according to semantic diversity of text. The at least one processor
15. The computer apparatus according to claim 14, wherein said original data corresponding to the number of label types are selected from said learning data set. The at least one processor
A computer device according to any one of claims 12 to 15, characterized in that it organizes the prompts in a format that includes a text type and a label type. The at least one processor
16. The computer device according to any one of claims 12 to 15, wherein said prompt is configured in a format including text type, label type, and label position segment molecule. The at least one processor
16. The computer device according to any one of claims 12 to 15, wherein the prompt is constructed by combining the task specification, the original data, and the prompt template. The at least one processor
extracting the new data and the label information using an autoregressive method that transfers the probability distribution of the previous token as the input of the next token to the natural language corresponding to the text and label in the prompt. The computer device according to any one of claims 12 to 18, wherein: The at least one processor
20. The computer apparatus of claim 19, wherein a beam search using heuristics communicates probabilities in the upper part of the probability distribution of the previous token as input for the next token.

Images