KR20230160619A - A Method for Constructing Foreign Language Parallel Corpus Source Data and a Storage Medium Recording a Program for Executing the Same - Google Patents

Abstract

The method of constructing foreign language parallel corpus source data according to the present invention includes step (a) of selecting a data storage unit of the management server according to preset criteria, receiving a plurality of raw corpora consisting of a plurality of words, and storing them in a database, management Step (b), where the data processing unit of the server classifies each raw corpus stored in the database according to n detailed topics, and the data processing unit extracts the raw corpus classified by detailed topics in step (b) and converts it into a conversation pattern. Step (c) of refining the raw data set as a set, (d) uploading the raw data set to the authoring cloud server in which the data storage unit is pre-built, and the machine translation unit of the authoring cloud server to the authoring cloud server. Step (e) of generating a machine-translated data set by performing machine translation on the raw data set uploaded to a server, wherein the data storage unit is an expert who has performed expert translation on the raw data set uploaded to the authoring cloud server. Step (f) of uploading the translation data set to the authoring cloud server, and (g) where the data construction unit of the authoring cloud server performs quality evaluation for each item on the machine translation data set and the expert translation data set. and a step (h) in which the data processing unit of the management server downloads the machine translation data set and the expert translation data set whose quality has been evaluated from the authoring cloud server to construct foreign language parallel corpus source data.

Description

A method for constructing foreign language parallel corpus source data and a storage medium recording a program for executing it {A Method for Constructing Foreign Language Parallel Corpus Source Data and a Storage Medium Recording a Program for Executing the Same}

The present invention relates to a method for constructing foreign language parallel corpus source data and a storage medium recording a program for executing the same. More specifically, it aims to promote the global expansion of the language culture content business and revitalize the artificial intelligence-based language culture content industry. This is about a method of constructing source data for a foreign language parallel corpus that can provide basic data for this purpose, and a storage medium that records a program to execute it.

As the global economic growth rate has been sluggish due to the recent international pandemic, interest in Korean culture and the Korean language is rapidly spreading as Korean Wave content has become a global trend.

Accordingly, in response to the future of the 4th Industrial Revolution, where information and communication and convergence technologies are becoming more sophisticated and content distribution platforms are diversifying, the expansion of language and culture content businesses centered on the two axes of 'digital' and 'global' is expected to help the Korean economy. It is predicted that it will be part of the innovation strategy to be pursued.

Therefore, based on the relationship between the Korean Wave and Korean language education, language education based on culture must be provided for the globalization of the Korean language, and the establishment of a digital environment that can be accessed anywhere in the world needs to be actively promoted.

In particular, in foreign language education, contrastive linguistic methodology that examines the differences and similarities between the target language and the learner's native language is useful, and it is very important to objectively extract correspondence pairs between languages.

To this end, there is an emerging need to build foreign language parallel corpus source data that can provide basic data that can be used for various contents. However, no such attempt has been made to date, so efforts are required to conduct related research and build data. It's a situation.

Korean Patent No. 10-2140983

The present invention is an invention made to solve the problems of the prior art described above, and promotes the global spread of language culture content business by efficiently constructing foreign language parallel corpus source data and revitalizing the artificial intelligence-based language culture content industry. The purpose is to provide basic data for

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

The method of constructing foreign language parallel corpus source data of the present invention to achieve the above purpose is to select the data storage unit of the management server according to preset standards, receive input of a plurality of raw corpora consisting of a plurality of words, and store them in the database. Step (a), where the data processing unit of the management server classifies each raw corpus stored in the database according to n detailed topics, and where the data processing unit classifies the raw corpus classified into detailed topics by step (b). Step (c) of extracting and refining the raw data set consisting of conversation patterns, (d) uploading the raw data set to the authoring cloud server in which the data storage unit is pre-built, and the machine of the authoring cloud server Step (e) in which the translation unit performs machine translation on the raw data set uploaded to the authoring cloud server to generate a machine translation data set, wherein the data storage unit performs machine translation on the raw data set uploaded to the authoring cloud server. Step (f) of uploading an expert translation data set that has undergone expert translation to the authoring cloud server, wherein the data construction unit of the authoring cloud server evaluates the quality of each item for the machine translation data set and the expert translation data set. Step (g) of performing and the data processing unit of the management server downloads the machine translation data set and the expert translation data set whose quality has been evaluated from the authoring cloud server, and constructs foreign language parallel corpus source data (h) ) steps are included.

At this time, in step (a), the data storage unit collects everyday conversation, messenger conversation, and semi-spoken language included in spoken language content, and linguistic data included in written content in a uniform ratio and can receive input as a raw corpus.

In addition, in step (b), the data processing unit may set the construction amount for each of the n detailed topics at a uniform ratio.

In addition, the step (c) is a step (c-1) in which the data processing unit randomly extracts the raw corpus into which detailed topics are classified by the step (b), and the data processing unit extracts the raw corpus by the step (c-1). Step (c-2), which combines the two raw corpora to form a single-turn, which is a conversation pattern, and the data processing unit combines three turns composed by step (c-2) It may include step (c-3) of refining as one raw data set.

In addition, the step (g) is a step (g-1) in which the data construction unit receives evaluation data that has undergone expert evaluation for the machine translation data set and the expert translation data set, and the data construction unit bases the evaluation data on the data set. Step (g-2) of performing a quality evaluation for each item on the machine translation data set and the expert translation data set, wherein the data construction unit has a quality standard score higher than or equal to a preset quality standard score in the machine translation data set and the expert translation data set. Step (g-3) of assigning a pass label to items and assigning a fail label to items that have a preset quality standard score or higher, and the machine translation in which the data construction unit performs supplementary processing for items given a fail label. Step (g-4) of re-entering the evaluation data that has undergone expert evaluation for the data set and the expert translation data set, and (g-2) until the data construction unit does not generate any items labeled as failed. It may include step (g-5) of repeating steps through (g-4).

And after step (h), the data storage unit uploads the data construction history of steps (a) to (h) to the artificial intelligence learning model to proceed with learning by the artificial intelligence learning model (i) Further steps may be performed.

Meanwhile, the present invention can be provided in the form of a computer-readable storage medium on which a program for executing the foreign language parallel corpus source data construction method described above is recorded.

The foreign language parallel corpus source data construction method of the present invention to solve the above problems can provide large-capacity, high-quality foreign language parallel corpus source data, and establishes a parallel corpus construction methodology for machine learning and a language data quality inspection and verification system. It has the advantage of not only being able to present standards for research, but also being able to be used as basic data for related research such as linguistics, translation studies, language engineering, and foreign language education.

In particular, the present invention can efficiently build foreign language parallel corpus source data to promote global expansion of language and culture content business and provide basic data for revitalizing the artificial intelligence-based language and culture content industry.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a diagram showing each process of the foreign language parallel corpus source data construction method according to an embodiment of the present invention;
Figure 2 is a diagram showing the configuration of a system for performing a method of constructing foreign language parallel corpus source data according to an embodiment of the present invention;
Figure 3 is a diagram showing the detailed process of step (c) in the foreign language parallel corpus source data construction method according to an embodiment of the present invention;
Figure 4 is a diagram showing the detailed process of step (g) in the foreign language parallel corpus source data construction method according to an embodiment of the present invention; and
Figure 5 is a diagram schematically showing a series of processes of a method for constructing foreign language parallel corpus source data according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention, in which the object of the present invention can be realized in detail, will be described with reference to the attached drawings. In describing this embodiment, the same names and the same symbols are used for the same components, and additional description accordingly will be omitted.

The purpose of the present invention is to build basic data to spread the language and culture content business around the two axes of 'digital' and 'global', and to develop artificial intelligence-based subtitle translators, chatbots for language learning or live commerce, etc. The goal is to design source data that will be used as learning data for learning.

In addition, the method for constructing foreign language parallel corpus source data according to the present invention is performed through a management server in which a program for constructing foreign language parallel corpus source data stored in a storage medium is installed, and can be driven by a processor of the management server.

At this time, the program for constructing foreign language parallel corpus source data can be output through an image output device such as a display module, and visible information can be provided through a graphical user interface visualized on the user's mobile terminal, desktop computer, laptop computer, etc. there is.

In particular, the storage medium storing the program for building foreign language parallel corpus source data can be installed on the management server using a removable disk or a communication network, and the program for building foreign language parallel corpus source data can enable the management server to be operated by various functional means. .

In other words, in the present invention, information processing by software is concretely realized through hardware.

Hereinafter, a method for constructing foreign language parallel corpus source data according to an embodiment of the present invention will be described with reference to the attached FIGS. 1 to 4.

Figure 1 is a diagram showing each process of the foreign language parallel corpus source data construction method according to an embodiment of the present invention.

In addition, Figure 2 is a diagram showing the configuration of a system for performing a method of constructing foreign language parallel corpus source data according to an embodiment of the present invention. In the following description, the codes assigned to each component are based on this diagram.

As shown in Figure 1, the method for constructing foreign language parallel corpus source data according to an embodiment of the present invention includes steps (a) to (i).

Step (a) is a process in which the data storage unit 110 of the management server 100 selects according to preset criteria, receives a plurality of raw corpora consisting of a plurality of words, and stores them in the database 130.

In particular, in this step, the data storage unit 110 can collect daily conversations, messenger conversations, and semi-spoken language included in spoken language content, and linguistic data included in written content at a uniform rate and receive them as a raw corpus.

For example, as shown in Table 1 below, daily conversation, messenger conversation, semi-spoken language, and language data can each be collected at a rate of 25%. In addition, in the example shown in Table 1, daily conversation, messenger conversation, semi-spoken language, and language data each contain 250,000 words, but this figure is provided as an example.

Next, step (b) is performed in which the data processing unit 120 of the management server 100 classifies each raw corpus stored in the database 130 according to n detailed topics.

In step (b), the data processing unit 120 can set the construction amount for each n detailed topics at a uniform ratio. The reason for doing this is because the present invention aims to build and disclose general-purpose data, and is to ensure uniformity by category and individual to prevent data bias.

For example, the ratio of the raw corpus classified by detailed topic in step (b) can be determined as shown in Table 2 below. In this example, 10 detailed topics that met the goal were selected, and the amount of construction for each detailed topic was set at a certain ratio to prevent bias in the data.

Next, step (c) is performed in which the data processing unit 120 extracts the raw corpus classified by detailed topics in step (b) and refines it into a raw data set composed of conversation patterns.

Figure 3 is a diagram showing the detailed process of step (c) in the method for constructing foreign language parallel corpus source data according to an embodiment of the present invention.

As shown in Figure 3, step (c) is in detail, step (c-1) in which the data processing unit 120 randomly extracts the raw corpus into which detailed topics have been classified by step (b), and the data processing unit 120 (c-2) step of combining the two raw corpora extracted in step (c-1) to form a single-turn, which is a conversation pattern, and the data processing unit 120 performs (c-2) It includes step (c-3) of combining three word sequences composed by step ) and refining them as one raw data set.

That is, in step (c), a raw corpus in which detailed topics are classified is extracted, two of these raw corpora are combined to form a word sequence, and three such word sequences can be combined to form one raw data set. .

Also, in this purification process, the standards shown in Table 3 below can be applied.

Next, step (d) of uploading the raw data set to the authoring cloud server 200 in which the data storage unit 110 is pre-built is performed. That is, through this process, the raw data set constructed in step (c) is registered in the authoring cloud server 200.

And in step (e), the machine translation unit 210 of the authoring cloud server 200 performs machine translation on the raw data set uploaded to the authoring cloud server 200 to generate a machine translation data set. This comes true.

At this time, the machine translator used in the machine translation unit 210 in this process may be of various models, including various previously known machine translators and machine translators that will be developed in the future, so a detailed description thereof will be omitted.

In step (f), the data storage unit 110 uploads an expert translation data set that has undergone expert translation for the raw data set uploaded to the authoring cloud server 200 to the authoring cloud server 200. It comes true.

In this process, the expert translation data set is stored in the data storage unit 110 after the raw data set is translated by experts organized according to certain standards. Vocabulary appropriate for the field of use is selected according to the subject area of the original text. , jargon and reference materials can be used to reflect the socio-cultural/lexical normative characteristics of the target language.

Next, step (g) is performed in which the data construction unit 220 of the authoring cloud server 200 performs quality evaluation for each item on the machine translation data set and the expert translation data set.

Figure 5 is a diagram showing the detailed process of step (g) in the foreign language parallel corpus source data construction method according to an embodiment of the present invention. As shown in Figure 5, step (g) may include steps (g-1) to (g-5) in detail.

In step (g-1), the data construction unit 220 receives evaluation data that has undergone expert evaluation for the machine translation data set and expert translation data set, and in step (g-2), such evaluation data is performed. Based on this, the data construction unit 220 performs a quality evaluation for each item on the machine translation data set and the expert translation data set.

At this time, each item for which quality evaluation is made can be selected based on various criteria. In this embodiment, a total of 8 items are selected and evaluated: accuracy/clarity/fluency/cohesion/consistency/significance/succinctness/functionality. It was decided that

Table 4 below provides a detailed explanation of the eight quality evaluation criteria mentioned above.

Next, in step (g-3), the data construction unit 220 assigns a passing label to items with a preset quality standard score or higher in the machine translation data set and the expert translation data set, and to items with a preset quality standard score or higher. A process of assigning a failed label is carried out.

For example, the quality standard score for each language can be determined as shown in Table 4 below. For accurate evaluation, machine translation quality evaluation and post-editing are performed in advance by experts in each language's major department by applying common evaluation standards, then weights for each error are determined by reflecting differences in each language pair, and text genre is determined. A method can be implemented to determine the lowest passing line for data quality, taking into account the situation for each language.

Table 5 below shows examples of quality standard scores set for each item by language.

Next, in step (g-4), the data construction unit 220 re-examines the evaluation data that has undergone expert evaluation on the machine translation data set and the expert translation data set for which supplementary processing has been performed for items assigned a failed label. An input process is carried out, and in step (g-5), the data construction unit 220 repeats steps (g-2) to (g-4) until no items are assigned a failed label. The process takes place.

At this time, in the process of performing step (g-5), if the failed label is maintained for more than the preset number of repetitions, the data construction unit 220 grants a hold label to the corresponding machine translation data set and expert translation data set, and cloud It can be stored as pending data on the server 200.

In this way, the machine translation data set and the expert translation data set stored as pending data are subjected to artificial intelligence learning by uploading the data construction history to the artificial intelligence learning model 300 in step (i), which will be described later, as time passes. Whenever the process of performing learning by the model 300 is completed, a check can be performed and steps (g-2) to (g-4) can be re-performed, and through this, the upgraded check process can be performed again. Processing and re-evaluation can be performed to ensure that items labeled as failing are utilized at a time when they no longer occur.

Next, in step (h), the data processing unit 120 of the management server 100 downloads the machine translation data set and the expert translation data set for which the quality has been evaluated from the authoring cloud server 200, and creates foreign language parallel corpus source data. The process of building is carried out. According to the above process, the present invention can provide large-capacity, high-quality foreign language parallel corpus source data.

Meanwhile, after step (h), the data storage unit 110 uploads the data construction history of steps (a) to (h) to the artificial intelligence learning model 300 and Step (i), which advances learning, can be further performed, and through this, the process of constructing foreign language parallel corpus source data can be operated more precisely and efficiently.

Figure 5 is a diagram schematically showing a series of processes of a method for constructing foreign language parallel corpus source data according to an embodiment of the present invention.

As shown in Figure 5, the overall process of the present invention includes the collection and purification process of steps (a) to (d), the processing process of steps (e) and (f), and steps (g) and (h). ) through a three-step verification process.

In the collection and purification process, topics by type are selected, and the collected texts are preprocessed to select the original text, which is stored as raw data.

In the subsequent processing process, machine translation and expert translation are performed using the uploaded raw data set, and these are stored as labeling data after going through an error evaluation and error correction process.

In addition, in the verification process, verification processes are performed for each machine translation and expert translation, and the verification completed results are constructed as foreign language parallel corpus source data.

As described above, the preferred embodiments according to the present invention have been examined, and the fact that the present invention can be embodied in other specific forms in addition to the embodiments described above without departing from the spirit or scope thereof is recognized by those skilled in the art. It is self-evident to them. Therefore, the above-described embodiments are to be regarded as illustrative and not restrictive, and thus the present invention is not limited to the above description but may be modified within the scope of the appended claims and their equivalents.

100: Management server
110: data storage unit
120: Data processing unit
130: database
200: Cloud server for authoring
210: Machine translation department
220: Data construction department
300: Artificial intelligence learning model

Claims (7)

Step (a) of selecting the data storage unit of the management server according to preset criteria, receiving a plurality of raw corpora consisting of a plurality of words, and storing them in a database;
Step (b) where the data processing unit of the management server classifies each raw corpus stored in the database according to n detailed topics;
Step (c) in which the data processing unit extracts the raw corpus classified by detailed topics in step (b) and refines it into a raw data set composed of conversation patterns;
Step (d) of uploading the raw data set to an authoring cloud server in which the data storage unit is pre-established;
Step (e) of the machine translation unit of the authoring cloud server performing machine translation on the raw data set uploaded to the authoring cloud server to generate a machine translation data set;
Step (f) of the data storage unit uploading an expert translation data set that has undergone expert translation of the raw data set uploaded to the authoring cloud server to the authoring cloud server;
Step (g) of the data construction unit of the authoring cloud server performing quality evaluation for each item on the machine translation data set and the expert translation data set; and
Step (h) of the data processing unit of the management server downloading the machine translation data set and the expert translation data set whose quality has been evaluated from the authoring cloud server to construct foreign language parallel corpus source data;
Including,
Method for constructing foreign language parallel corpus source data. According to paragraph 1,
In step (a),
The data storage unit collects everyday conversations, messenger conversations, and semi-spoken language included in spoken language content, and linguistic data included in written content at a uniform rate and receives them as a raw corpus,
Method for constructing foreign language parallel corpus source data. According to paragraph 1,
In step (b),
The data processing unit sets the construction amount for each of the n detailed topics at a uniform ratio,
Method for constructing foreign language parallel corpus source data. According to paragraph 1,
In step (c),
Step (c-1) in which the data processing unit randomly extracts the raw corpus into which detailed topics have been classified in step (b);
Step (c-2) in which the data processing unit combines the two raw corpora extracted in step (c-1) to form a single-turn, which is a conversation pattern; and
Step (c-3) in which the data processing unit combines three word sequences formed in step (c-2) and refines them into one raw data set;
Including,
Method for constructing foreign language parallel corpus source data. According to paragraph 1,
In step (g),
Step (g-1) where the data construction unit receives evaluation data that has undergone expert evaluation for the machine translation data set and the expert translation data set;
Step (g-2) in which the data construction unit performs an item-specific quality evaluation of the machine translation data set and the expert translation data set based on the evaluation data;
(g-3) wherein the data construction unit grants a pass label to items exceeding a preset quality standard score in the machine translation data set and the expert translation data set, and grants a fail label to items exceeding a preset quality standard score. step;
A (g-4) step in which the data construction unit re-inputs evaluation data that has undergone expert evaluation on the machine translation data set and the expert translation data set on which supplementary processing has been performed for items labeled as failed; and
Step (g-5) in which the data building unit repeats steps (g-2) to (g-4) until no items are labeled as failed;
Including,
Method for constructing foreign language parallel corpus source data. According to paragraph 1,
After step (h) above,
Step (i) is further performed in which the data storage unit uploads the data construction history of steps (a) to (h) to the artificial intelligence learning model to proceed with learning by the artificial intelligence learning model,
Method for constructing foreign language parallel corpus source data. A computer-readable storage medium that records a program for executing the foreign language parallel corpus source data construction method of any one of paragraphs 1 to 6 on a computer.