KR102575752B1 - Examination data classification device and classification method using ensemble classification model - Google Patents

Abstract

The present invention provides a method for classifying examination data based on a plurality of findings, the method comprising: receiving examination data; pre-processing the examination data; generating an ensemble classification model based on the preprocessed examination data; and obtaining a classification result of the examination data using the learned ensemble classification model.
In addition, in the generating step, static embedding and contextualized embedding are performed on the preprocessed checkup data, and the ensemble is performed based on data obtained as a result of performing the static embedding and contextualized embedding. Create a classification model.

Description

Examination data classification device and method using an ensemble classification model

The present invention relates to an apparatus and method for classifying examination data using an ensemble classification model.

As machine learning technology develops, various machine learning techniques have been developed, and these machine learning techniques are being applied to various fields.

Even in the field of natural language processing, various machine learning techniques have been developed, and various methods are being attempted to perform more accurate classification of character data.

Republic of Korea Patent Registration No. 10-1713487, registered on February 28, 2017

An object of the present invention is to provide a examination data classification device and classification method for classifying examination data through an ensemble natural language processing model in which all characteristics of words and contexts are reflected.

Another object of the present invention is to provide an apparatus and method for classifying examination data that provide an ensemble natural language processing model with improved classification performance by using a pretraining language model.

In addition, the present invention extracts sentences containing keywords corresponding to diagnosis names and the number of tokens is less than the maximum number of tokens of the corpus used for pre-training from examination data, and uses the extracted sentences to form a learned ensemble natural language processing model An object of the present invention is to provide a examination data classification device and a classification method.

Another object of the present invention is to provide an apparatus and method for classifying examination data that provide an ensemble natural language processing model learned through a preprocessing process adapted to examination data.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

In order to solve the above problems, a method for classifying examination data according to an embodiment of the present invention is a method for classifying examination data based on a plurality of findings, performed by an apparatus, comprising the steps of receiving examination data; pre-processing the examination data; generating an ensemble classification model based on the preprocessed examination data; and obtaining a classification result of the examination data using the learned ensemble classification model.

In addition, in the generating step, static embedding and contextualized embedding are performed on the preprocessed checkup data, and the ensemble is performed based on data obtained as a result of performing the static embedding and contextualized embedding. to create a classification model.

The generating may include generating at least one first classification model through learning based on fixedly embedded data; generating at least one second classification model through learning based on contextualization-embedded data; and generating the ensemble classification model based on the at least one first classification model and the at least one second classification model.

The generating of the at least one second classification model may include acquiring an initial weight through learning using a pre-obtained corpus set; and generating the at least one second classification model by fine-tuning the initial weights through learning based on contextualization-embedded data.

In addition, at least one of CNN (Convolution Neural Network) and DCNN (Deep Convolution Neural Network) is used to learn the at least one first classification model, and LSTM (Long Short-Term) is used to learn the at least one second classification model. Memory models), Korean Bidirectional Encoder Representations from Transformers (KoBERT), Korean Efficiently Learning an Encoder that Classifies Token Replacements Accurately (KoELECTRA), Bidirectional Encoder Representations from Transformers (BERT), and Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA) one can be used.

In addition, the received checkup data includes text data in which at least one of the maximum and average number of tokens calculated through tokenizing is greater than a preset reference number, and the reference number is This corresponds to the maximum number of tokens of the corpus included in the acquired corpus set.

In addition, the checkup data is calculated for each of a plurality of diagnosis names, and the preprocessing may include searching for a keyword corresponding to each of the plurality of diagnosis names in the text data; extracting a sentence including the searched keyword and having a number of tokens less than or equal to the reference number; and performing pre-processing based on the extracted sentence.

In addition, the examination data includes individual findings text, numerical examination results, and comprehensive findings text, and the examination data is calculated for each of a plurality of diagnosis names.

In addition, the numerical test result is predetermined standardized data corresponding to each of the plurality of diagnosis names, and the comprehensive findings text is at least one of a maximum and an average of tokens calculated through tokenizing. The number may be greater than the reference number, and the reference number may correspond to the maximum number of tokens of the corpus included in the pre-obtained corpus set.

In addition, the pre-processing may include deleting the remaining special characters except for the preset standard special characters from the examination data; uniformizing spacing rules of examination data from which the special characters are deleted, based on a preset standard grammar; digitizing the homogenized examination data; and padding the digitized data.

In addition, the digitizing step may include tokenizing the uniformized examination data; Counting the number of tokens generated through tokenization; and digitizing the uniformized examination data based on the number of tokens.

In addition, the examination data is classified into a plurality of groups each corresponding to the plurality of findings, and the step of digitizing increases the amount of data of the minimum group having the smallest amount of data among the plurality of groups. Include more steps.

The multiplying may include generating a plurality of sentences by dividing the data included in the minimum group into sentence units; and increasing the amount of data of the minimum group by connecting the plurality of sentences to each other.

In addition, the examination data classification apparatus according to an embodiment of the present invention includes a communication unit; database unit; and a control unit classifying examination data stored in the database unit or examination data received from an external server through the communication unit into a plurality of predetermined findings.

In addition, the control unit pre-processes the examination data, performs static embedding and contextualized embedding on the pre-processed examination data, and results data obtained by performing the static embedding and contextualized embedding The ensemble classification model is generated based on , and a classification result of the examination data is obtained using the learned ensemble classification model.

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

According to the present invention as described above, the following effects can be derived.

First, according to the present invention, since examination data is classified through an ensemble natural language processing model in which all characteristics of words and contexts are reflected, classification performance of examination data can be improved.

In addition, according to the present invention, since examination data is classified through an ensemble natural language processing model using a pretraining language model, classification performance of the examination data can be improved.

In addition, according to the present invention, since the natural language processing model in which the characteristics of words are reflected is used together with the natural language processing model in which the characteristics of context are reflected, the number of tokens in the examination data is the number of corpus used for pre-training on the characteristics of context. Accurate classification can be made even when the number of tokens is greater than the maximum number.

In addition, according to the present invention, a sentence containing a keyword corresponding to a diagnosis name and the number of tokens being less than the maximum number of tokens of the corpus used for pre-training is extracted from the examination data, and the extracted sentence is used to learn Ensemble Natural Language Processing Model Through this, the examination data is classified. Through this, even when the number of tokens of the examination data is equal to or greater than the maximum number of tokens of the corpus used for pre-training, accurate classification can be achieved.

Further, according to the present invention, since the examination data is classified through an ensemble natural language processing model learned through a preprocessing process adapted to the examination data, the classification performance of the examination data can be improved.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is an exemplary view showing the configuration of a examination data classification system according to an embodiment of the present invention.
2 is a block diagram showing the configuration of an apparatus for classifying examination data according to an embodiment of the present invention.
FIG. 3 is a block diagram showing the configuration of a control unit in FIG. 2 .
4 is a flowchart illustrating a process of a method for classifying examination data according to an embodiment of the present invention.
5 is a conceptual diagram exemplarily illustrating a process of a method for classifying examination data according to an embodiment of the present invention.
6 is a flowchart illustrating a process of an embodiment of step S10 of FIG. 4 .
FIG. 7 is a flowchart illustrating the process of step S13 of FIG. 6 .
FIG. 8 is a conceptual diagram exemplarily illustrating digitized data in step S133 of FIG. 7 .
9 is a flowchart illustrating the process of step S20 of FIG. 4 .
FIG. 10 is a conceptual diagram illustrating the first classification model of FIG. 9 as an example.
FIG. 11 is a conceptual diagram illustrating the second classification model of FIG. 9 as an example.
FIG. 12 is a flowchart illustrating the process of step S23 of FIG. 9 .
FIG. 13 is a conceptual diagram illustrating the ensemble model of FIG. 4 as an example.
14 is a flowchart illustrating a process of another embodiment of step S10 of FIG. 4 .
FIG. 15 is a conceptual diagram exemplarily illustrating a sentence extracted in step S102 of FIG. 14 .

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Prior to the description, the meaning of the terms used in this specification will be briefly described. However, it should be noted that the description of terms is intended to help the understanding of the present specification, and is not used in the sense of limiting the technical spirit of the present invention unless explicitly described as limiting the present invention.

However, in some embodiments, the examination data classification apparatus 10 may include fewer or more components than those shown in FIGS. 2 and 3 .

The examination data input terminal 2 described in this specification includes, for example, a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), Navigation, slate PC, tablet PC, ultrabook, wearable device (e.g., watch type terminal (smartwatch), glass type terminal (smart glass), HMD (head) mounted display)) may be included.

However, those skilled in the art will readily recognize that the configuration according to the embodiments described in this specification may be applied to fixed terminals such as digital TVs, desktop computers, digital signage, etc., except when applicable only to terminals. .

Referring to FIG. 1 , in the examination data classification system according to an embodiment of the present invention, a server 1 and a plurality of examination data input terminals 2 may be connected to each other to enable communication through a network 3 .

The server 1 performs preprocessing on the examination data received from the plurality of examination data input terminals 2, generates an ensemble classification model based on the preprocessed examination data, and uses the generated ensemble classification model to determine the examination data. Get the classification result.

Referring to FIG. 2 , the server 1 may include a checkup data classification device 10 . However, it is not limited thereto, and the examination data classification device 10 may be provided separately from the server 1.

The examination data classification device 10 includes a communication unit 11, a database unit 12 and a control unit 13. The communication unit 11, the database unit 12, and the control unit 13 may be connected by a system bus connecting each other.

Information exchange between the examination data classification device 10 and the examination data classification device 10, between the examination data classification device 10 and the examination data input terminal 2, or between the examination data classification device 10 and the external server is performed by the communication unit (11) can be performed.

The communication unit may be implemented through at least one of a wired communication module, a wireless communication module, and a short-distance communication module. A wireless Internet module refers to a module for wireless Internet access, and may be built into or external to each device. Wireless Internet technologies include WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), LTE (long term evolution), LTE-A (Long Term Evolution-Advanced) and the like may be used.

The database unit 12 stores various information received through the communication unit 11, various information for performing functions of the control unit 13, and various information calculated by the control unit 13.

The database unit 12 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, It may include a storage medium of at least one type of a magnetic disk and an optical disk. In addition, the database unit 12 may be implemented in the form of a web storage.

The control unit 13 includes a preprocessing module 14 and a classification module 15.

The preprocessing module 14 performs a preprocessing process of converting the received examination data into input data of an ensemble classification model.

The preprocessing process includes standardization, digitization, padding, and data augmentation processes.

To this end, the preprocessing module 14 includes a shaping unit 141, a digitization unit 142, a padding unit 143, and a data augmentation unit 144. The pre-processing process will be described in detail later along with the examination data classification method according to the present invention.

The classification module 15 includes an embedding unit 151 that embeds the preprocessed data.

In addition, the classification module 15 includes a classification unit 152 that performs learning using the embedded data or classifies the embedded data using the learned model.

The classification unit 152 learns an ensemble classification model using an ensemble technique or performs classification on embedded data using the learned ensemble classification model.

In one embodiment, Voting, Bagging, and Boosting methods may be used in Ensemble Learning. However, it is not limited thereto.

That is, the classification unit 152 classifies the examination data through an ensemble technique using various algorithms.

Ensemble techniques include CNN (Convolution Neural Network) and DCNN (Deep Convolution Neural Network), LSTM (Long Short-Term Memory models), KoBERT (Korean Bidirectional Encoder Representations from Transformers), KoELECTRA (Korean Efficiently Learning an Encoder that Classifies Token Replacements Accurately ) At least one algorithm of BERT (Bidirectional Encoder Representations from Transformers) and ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) may be used.

In an embodiment, the classification unit 152 may generate at least one first classification model using at least one algorithm of a convolution neural network (CNN) and a deep convolution neural network (DCNN).

In one embodiment, the classification unit 152 may include Long Short-Term Memory models (LSTM), Korean Bidirectional Encoder Representations from Transformers (KoBERT), Korean Efficiently Learning an Encoder that Classifies Token Replacements Accurately (KoELECTRA) Bidirectional Encoder Representations from BERT Transformers) and ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) to generate at least one second classification model.

In an embodiment, the classification unit 152 may generate an ensemble classification model using the first classification model and the second classification model.

In terms of hardware, the control unit 13 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors ), controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

Also, in terms of software, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. The software code may be implemented as a software application written in any suitable programming language. The software code may be stored in a memory and executed by the control unit 13 .

Hereinafter, a checkup data classification method S1 according to an embodiment of the present invention will be described in detail with reference to FIGS. 4 to 15 .

Referring to FIG. 4, the examination data classification method (S1) includes preprocessing the examination data (S10), learning an ensemble classification model based on the preprocessed examination data (S20), and using the learned ensemble classification model. and classifying the preprocessed examination data (S30).

Referring to FIG. 5, a process in which the examination data classification apparatus 10 creates a classification model for thyroid examination data and derives a classification result according to a plurality of preset findings using the generated classification model is conceptually illustrated. .

In the illustrated embodiment, the examination data may be derived from health examination data. The health examination data may be composed of individual findings texts for various diagnoses that can be confirmed in health examinations, numerical test results related to various abnormalities, and comprehensive findings texts.

Examination data may be composed of individual finding texts for a specific diagnosis name, numerical test results related to a specific diagnosis name, and comprehensive finding texts. In one embodiment, the examination data may be composed of a list of natural words connecting individual finding texts, numerical examination results, and comprehensive finding texts.

In the illustrated embodiment, the examination data is for the thyroid, and is composed of thyroid individual finding text, hormone level test result, and comprehensive finding text. In an embodiment not shown, the checkup data may include checkup data for diagnoses other than thyroid.

When the examination data is received by the examination data classification device 10, the pre-processing module 14 of the examination data classification device 10 pre-processes the examination data (S10).

When the preprocessing is completed, the classification module 15 of the examination data classification device 10 learns an ensemble classification model based on the preprocessed examination data (S20).

When the learning is completed, the classification module 15 of the examination data classification apparatus 10 obtains a classification result for the preprocessed examination data using the learned ensemble classification model (S30).

The classification result is a result of being classified into a plurality of preset findings, and in an embodiment, the plurality of findings may be normal (Class 0) and abnormal findings (Class 1). Also, in one embodiment, the plurality of findings may be normal (Class 0), minor findings (Class 1), and severe findings (Class 2).

Below, with reference to FIGS. 6 to 8, the pre-processing of examination data (S10) will be described in detail.

(1) Description of pre-processing of examination data (S10)

Referring to FIG. 6 , the pre-processing of examination data (S10) includes deleting special characters except for preset special characters (S11), standardizing spacing rules (S12), and digitizing data (S13). ) and data padding (S14).

First, the examination data classification apparatus 10 deletes the remaining special characters except for the preset special characters from the received examination data (S11). Deletion of special characters may be performed by the shaping unit 141 of the preprocessing module 14 .

Since the examination data includes many special characters such as unit of measurement, the sequence of the examination data may be excessively increased. Therefore, in order to reduce the sequence of examination data, the remaining special characters except for the preset special characters are deleted.

In one embodiment, the preset special characters may be 8 special characters such as ", . &' / ~ ² -".

In addition, the numerical examination result included in the examination data is numerical data in a standardized form matched with a diagnosis name, and the formalizing unit 141 may convert the numerical data in a standardized form into text data.

For example, when the examination data is related to the thyroid, the numerical result data may be hormone level data. The shaping unit 141 may convert the hormone level data into a text format such as "hormonal test levels are normal."

When the deletion of the special characters is completed, the checkup data classification device 10 standardizes spacing rules of text included in the checkup data based on a preset grammar rule (S12). Standardization of spacing rules may be performed by the shaping unit 141 .

Depending on the writer, the text included in the examination data may have errors in spacing rules or may use different types of spacing rules. For example, "may", "may" and "may" may be included together. Since all of the corresponding texts are natural language with the same content, they can be unified based on a predetermined spacing rule. For example, "can" and "could" can be unified into "could".

In one embodiment, various grammatical rules other than spaces may be standardized.

Various tokenizers may be used to standardize spacing rules. In an embodiment, at least one tokenizer of Mecab-Ko, WordPiece from KoBERT, WordPiece from KoELECRA, Mecab-Ko & WordPiece may be used. The text included in the examination data is divided into a plurality of tokens by the tokenizer.

When the standardization of spacing rules is completed, the examination data classification apparatus 10 digitizes the uniform examination data (S13).

Referring to FIG. 7 , a detailed process of digitizing examination data (S13) is illustrated.

First, tokenization is performed on examination data by the digitization unit 142 (S131). In an embodiment not shown, the digitization unit 142 may use the result tokenized by the shaping unit 141 without separately performing tokenization.

When the examination data is divided into a plurality of tokens by tokenizing, the digitization unit 142 counts the number of tokens (S132).

When the counting is completed, the digitization unit 142 digitizes the data based on the number of tokens (S133).

Referring to FIG. 8 , the numerical result obtained by step S133 is shown as an example.

The “hormone” token is the 13023rd disposed token among the tokens included in the examination data, and the “examination” token is the 6911th allocated token among the tokens included in the examination data. That is, digitization is performed according to the arrangement order of tokens.

Referring back to FIG. 6 , as digitization is completed, the padding unit 143 performs padding on the examination data for which digitization has been completed.

The padding unit 143 compares sequences of a plurality of digitized examination data and identifies a maximum sequence of the plurality of digitized examination data.

In addition, when the maximum sequence is confirmed, the sequence of a plurality of digitized examination data is increased based on the maximum sequence. That is, the lengths of all of the plurality of digitized examination data are unified.

Pre-processing of the examination data is completed through the above process.

When the preprocessing is completed, the data augmentation unit 144 of the preprocessing module 14 may additionally perform data augmentation.

Examination data used in the learning process of the ensemble classification model may be classified into a plurality of groups each corresponding to a plurality of preset findings. That is, examination data used in the learning process may be labeled as a plurality of groups.

For example, examination data used in the learning process may be classified into normal (Class 0), mild findings (Class 1), and severe findings (Class 2).

However, if the amount of data of the classified specific group is too small, a problem may occur in which the characteristics of the group are not properly reflected in the learning model. Accordingly, the data augmentation unit 144 may perform data augmentation for a group with an excessively small amount of data.

The data augmentation unit 144 divides the data included in the minimum group with the smallest amount of data into sentence units to generate a plurality of sentences, and connects the generated sentences to increase the amount of data in the minimum group. .

For example, when classified as a first sentence, a second sentence, and a third sentence, through a combination of the first and second sentences, a combination of the second and third sentences, and a combination of the first, second, and third sentences. The amount of data can be multiplied.

Since the ensemble classification model is generated through a preprocessing process suitable for examination data, such as omission of special characters, standardization of spacing rules, data digitization, and data padding, the classification accuracy of the generated ensemble classification model can be improved.

(2) Description of the step (S20) of learning an ensemble classification model based on the preprocessed examination data

Referring to FIG. 9 , a detailed process of learning an ensemble classification model based on preprocessed examination data (S20) is illustrated.

First, the examination data classification apparatus 10 performs static embedding and contextualized embedding on the preprocessed examination data. The embedding process may be performed by the embedding unit 151 of the control unit 13 .

Referring to FIG. 10 , fixed embedding is illustrated as an example.

In the illustrated embodiment, fixed embedding based on a preset window size is performed on some tokens among a plurality of tokens included in the inspection data.

In the illustrated embodiment, in the model shown above, fixed embedding is performed for “blood pressure”, “this” and “average” tokens, and “slightly”, “high”, “high” and “.” A fixed embedding is performed on the token.

In the illustrated embodiment, in the model shown below, fixed embedding is performed for “blood pressure”, “this” and “average” tokens, and fixed embedding is performed for “numerical”, “test” and “result” tokens do.

In one embodiment, a window of a certain size may be used for fixed embedding, and windows of various sizes may be used in combination.

Referring to FIG. 11 , contextualization embedding is illustrated by way of example.

In the illustrated embodiment, contextualization embedding is performed on a predetermined number of tokens or more in order to consider context-related characteristics.

In one embodiment, contextualization embeddings may include token embeddings, segment embeddings, and position embeddings.

Token Embedding embeds each character unit, and makes the longest sub-word that appears frequently into one unit. Deterioration in modeling performance can be prevented by OOV (Out Of Vacabulary) processing for words that do not appear frequently.

In addition, segment embedding constructs tokenized words into one sentence again. Use the separator [SEP] between the two sentences and designate the two sentences as one segment and input them.

Position Embedding sequentially encodes tokens.

Referring back to FIG. 9 , the examination data classification apparatus 10 generates a first classification model through learning based on data obtained as a result of fixed embedding (S22).

In an embodiment, the classification unit 152 may generate at least one first classification model using at least one algorithm of a convolution neural network (CNN) and a deep convolution neural network (DCNN).

Referring to FIG. 10, a process of generating a first classification model by CNN and DCNN algorithms is conceptually illustrated.

Referring back to FIG. 9 , the examination data classification apparatus 10 generates a second classification model through learning based on the result data of contextualization embedding (S23).

In one embodiment, the classification unit 152 may include Long Short-Term Memory models (LSTM), Korean Bidirectional Encoder Representations from Transformers (KoBERT), Korean Efficiently Learning an Encoder that Classifies Token Replacements Accurately (KoELECTRA) Bidirectional Encoder Representations from BERT Transformers) and ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) to generate at least one second classification model.

Referring to FIG. 11, a process of generating a second classification model by a Bidirectional Encoder Representations from Transformers (BERT) algorithm is conceptually illustrated.

Since the second classification model in which context characteristics are reflected is used together with the first classification model, the classification performance of the ensemble separation model can be improved.

Referring to FIG. 12 , a detailed process of an embodiment of generating a second classification model (S23) is illustrated.

First, the examination data classification apparatus 10 obtains an initial weight through learning using a pre-obtained corpus set (S231).

When the acquisition of the initial weights is completed, the examination data classification apparatus 10 generates a second classification model by fine-tuning the initial weights through learning based on the data resulting from the culturalization embedding (S232).

In one embodiment, KoBERT (Korean Bidirectional Encoder Representations from Transformers), KoELECTRA (Korean Efficiently Learning an Encoder that Classifies Token Replacements Accurately) BERT (Bidirectional Encoder Representations from Transformers) and ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) At least one algorithm may be used.

Since pre-learning is performed using a previously acquired corpus set, a classification model having high accuracy classification performance can be generated using a relatively small amount of examination data.

However, in the case of learning from a previously acquired corpus set, there is a problem in that data with a length shorter than the maximum number of tokens of the corpus included in the previously acquired corpus set must be input to the second classification model in the fine-tuning or classification process. did.

Accordingly, if the checkup data includes data having a length greater than the maximum number of tokens of the corpus included in the acquired corpus set, the divided data must be divided and used, and desired keywords may not be included in the divided data. In particular, in the case of the comprehensive findings text of the examination data, the length was frequently longer than the maximum number of tokens of the corpus included in the acquired corpus set.

According to the present invention, since the first classification model that is not affected by the maximum number of tokens of the corpus is used together, accurate classification can be made regardless of the length of examination data.

In one embodiment, the checkup data includes text data in which at least one of the maximum and average numbers of tokens calculated through tokenizing is greater than a preset reference number, and the reference number is It may correspond to the maximum number of tokens of the corpus included in the acquired corpus set. For example, at least one of the maximum and average number of tokens of the comprehensive findings text may be greater than the reference number.

Referring back to FIG. 9 , the examination data classification apparatus 10 generates an ensemble classification model based on the generated at least one first classification model and at least one second classification model (S24).

In one embodiment, Voting, Bagging, and Boosting methods may be used in Ensemble Learning. However, it is not limited thereto.

(3) Description of the step (S30) of classifying the preprocessed examination data using the learned ensemble classification model

Referring back to FIG. 4 , when an ensemble classification model is generated, the examination data classification apparatus 10 classifies the preprocessed examination data (S30).

Referring to FIG. 13, a test result of an ensemble classification model using a first classification model generated by the DCNN algorithm and a second classification model generated by the KoELECTRA algorithm using prior learning is shown.

In the illustrated embodiment, an F1 score (Macro F1) calculated based on precision (Macro P) and recall (Macro R) for a plurality of first classification models, a plurality of second classification models, and an ensemble classification model is shown. . It is shown that the precision, recall, and F1 score of the ensemble classification model through the DCNN and KoELECTRA algorithms are improved compared to those using only the first classification model and the second classification model.

In one embodiment, first and second classification models using algorithms such as CNN, DCNN, and LSTM, and a second classification model (KoELECTRA, etc.) learned by using pre-training may be used together as an ensemble classification model.

Through this, by using the second classification model (KoELECTRA, etc.) learned in the method using the pre-training, the accuracy of the classification diagram is improved, and at the same time, accurate classification can be performed without being restricted by the length of the test data.

(4) Description of pre-processing of examination data according to another embodiment (S10)

Referring to FIG. 14 , a process of pre-processing examination data according to another embodiment (S10) is illustrated.

First, the examination data classification apparatus 10 may reprocess examination data based on the reference number, which is the maximum number of tokens of the corpus included in the corpus set, and may perform preprocessing on the reprocessed examination data.

When learning and classifying examination data corresponding to a specific diagnosis name is performed, the examination data classification device 10 extracts sentences satisfying predetermined requirements from the comprehensive findings text, and separates the individual findings text, numerical test results, and extracted sentences. preprocessing can be performed.

First, the examination data classification device 10 searches for keywords corresponding to diagnosis names in the comprehensive findings text (S101).

For example, when the individual finding text is related to the thyroid, a search may be performed for a predetermined keyword related to the thyroid.

When a keyword is searched for, the examination data classification apparatus 10 extracts sentences including the searched keyword and the number of tokens being equal to or less than the reference number (S102).

Referring to FIG. 15 , an extracted sentence is illustrated as an example.

Referring back to FIG. 14 , the examination data classification apparatus 10 performs pre-processing based on the extracted sentence (S103).

In one embodiment, the examination data classification apparatus 10 performs preprocessing on individual finding texts, numerical examination results, and extracted sentences.

Through this, even when the number of tokens of the examination data is greater than or equal to the maximum number of tokens of the corpus used for pre-training, accurate learning and classification can be performed.

The examination data classification method according to the embodiment of the present invention described above is the same content as the examination data classification system and device described through FIGS. 1 to 3 and only the category of the invention is different.

The method according to the embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a server, which is hardware, and stored in a medium.

The aforementioned program is C, C++, JAVA, machine language, etc. It may include a code coded in a computer language of. Such codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, this code may further include a database reference code for determining where additional information or media necessary for the processor of the computer to execute the functions should be referred to from which location (address address) of the internal or external database of the computer. can In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the communication module of the computer to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or database unit, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected through a network, and computer readable codes may be stored in a distributed manner.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. Software modules include RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash database unit (Flash Memory), hard disk, removable disk, CD-ROM, Alternatively, it may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

1: server
2: Examination data input terminal
3: network
10: Examination data classification device
11: Communication department
12: database unit
13: control unit
14: preprocessing module
141: formalization unit
142: digitization unit
143: padding unit
144: data multiplication unit
15: classification module
151: embedding unit
152: classification unit

Claims (10)

In the method of classifying examination data based on a plurality of findings, performed by an apparatus,
Receiving examination data;
pre-processing the examination data;
generating an ensemble classification model based on the preprocessed examination data; and
Acquiring a classification result of the examination data using the learned ensemble classification model;
In the preprocessing step,
deleting the remaining special characters except for the predetermined reference special characters from the examination data;
uniformizing spacing rules of examination data from which the special characters are deleted, based on a preset standard grammar;
digitizing the homogenized examination data; and
Including the step of padding the digitized data,
The step of quantifying,
Tokenizing the uniformized examination data;
Counting the number of tokens generated through tokenization; and
Digitizing the uniformized examination data based on the number of tokens;
The generating step is
performing static embedding and contextualized embedding on the preprocessed examination data;
Generating the ensemble classification model based on data obtained by performing the fixed embedding and contextualized embedding,
method. According to claim 1,
The generating step is
generating at least one first classification model through learning based on fixedly embedded data;
generating at least one second classification model through learning based on contextualization-embedded data; and
Generating the ensemble classification model based on the at least one first classification model and the at least one second classification model,
method. According to claim 2,
The step of generating the at least one second classification model,
acquiring initial weights through learning using a previously acquired corpus set; and
Generating the at least one second classification model by fine-tuning the initial weights through learning based on contextualized embedded data.
method. According to claim 3,
The received examination data includes text data in which at least one of a maximum and an average number of tokens calculated through tokenizing is greater than a preset reference number;
The reference number corresponds to the maximum number of tokens of the corpus included in the previously obtained corpus set.
method. According to claim 4,
The examination data is calculated for each of a plurality of diagnosis names,
In the preprocessing step,
searching the text data for a keyword corresponding to each of the plurality of diagnosis names;
extracting a sentence including the searched keyword and having a number of tokens less than or equal to the reference number; and
Including performing preprocessing based on the extracted sentence,
method. According to claim 3,
The examination data includes individual findings text, numerical test results, and comprehensive findings text,
The examination data is calculated for each of a plurality of diagnosis names,
The numerical test result is standardized data corresponding to each of the plurality of diagnosis names,
In the comprehensive opinion text, at least one of the maximum and average number of tokens calculated through tokenizing is greater than the reference number,
The reference number corresponds to the maximum number of tokens of the corpus included in the previously obtained corpus set.
method. delete According to claim 1,
The examination data is classified into a plurality of groups each corresponding to each of the plurality of findings;
The step of quantifying,
Further comprising increasing the amount of data of the minimum group having the smallest amount of data among the plurality of groups,
The growing step is
generating a plurality of sentences by dividing the data included in the minimum group into sentence units; and
Connecting the plurality of sentences to each other to increase the amount of data of the minimum group,
method. communications department;
database unit; and
A control unit classifying examination data stored in the database unit or examination data received from an external server through the communication unit into a plurality of preset findings;
The control unit,
Preprocessing the examination data,
performing static embedding and contextualized embedding on the preprocessed examination data;
An ensemble classification model is generated based on the data obtained by performing the fixed embedding and the contextualized embedding;
Obtaining a classification result of the examination data using the learned ensemble classification model;
During the preprocessing, the rest of the special characters except for the preset standard special characters are deleted from the examination data, and the spacing rules of the examination data from which the special characters are deleted are uniformized based on the preset standard grammar;
Digitize the uniform examination data,
Padding the digitized data;
When digitizing, tokenizing the uniform examination data, counting the number of tokens generated through tokenizing, and digitizing the uniform examination data based on the number of tokens To do, examination data classification device. A computer program stored in a computer-readable recording medium in order to execute the method of any one of claims 1 to 6 and 8 by being combined with a computer that is hardware.