KR102501793B1 - Method for constructing dataset - Google Patents

Abstract

A method of constructing a dataset for model learning, comprising constructing a plurality of candidate unlabeled data groups from an unlabeled data pool, including one candidate unlabeled data group and one or more label data among the plurality of candidate unlabeled data groups. constructing an input data set to generate, calculating representative eigenvalues corresponding to the input data set by inputting the configured input data set into a first neural network model, and calculating a plurality of representative eigenvalues based on the calculated representative eigenvalues. and determining an unlabeled data group for label switching from among candidate unlabeled data groups, and each of a plurality of representative eigenvalues may correspond to one input data set.

Description

Dataset construction method {METHOD FOR CONSTRUCTING DATASET}

The present invention relates to a method for constructing a dataset, and more particularly, to a method for constructing a dataset for model learning.

In general, in order to detect or classify data through machine learning, securing learning data for machine learning must be preceded. For machine learning, at least 1,000 to 10,000 pieces of learning data must be secured.

In order to learn a huge amount of data, select data that is more effective for learning among unlabeled data, request it to an expert, and construct a dataset for learning with the labeled data according to the request. .

However, among the vast amount of data, not only is the amount of unlabeled data much larger, but also a high cost may occur in securing sufficient training data because specialized personnel are required for data labeling.

Accordingly, there is a need in the art to construct an effective training dataset in learning a model.

Korean Patent Publication No. 2016-0012537 discloses a neural network learning method and apparatus, and a data processing apparatus.

The present disclosure has been made in response to the above background art, and aims to provide a method for constructing a dataset for model learning.

A dataset construction method according to an embodiment of the present disclosure for realizing the above object includes constructing a plurality of candidate unlabeled data groups from an unlabeled data pool, the plurality of candidate unlabeled data groups. Constructing an input data set including one candidate unlabeled data group among label data groups and one or more labeled data; inputting the configured input data set to a first neural network model to set the input data set; calculating a representative eigenvalue corresponding to , and determining an unlabeled data group for label switching from among a plurality of candidate unlabeled data groups based on the calculated plurality of representative eigenvalues. can Each of the plurality of representative eigenvalues may correspond to one input data set.

In an alternative embodiment of the dataset construction method, determining a plurality of unlabeled data included in the determined unlabeled data group for label conversion as data to be labeled, and determining the plurality of unlabeled data as determined as data to be labeled. After labeling is performed, the method may further include configuring the labeled unlabeled data and the one or more labeled data as a training dataset of a neural network model for performing a classification task.

In an alternative embodiment of the dataset construction method, the first neural network model uses the Empirical Neural Tangent Kernel (Empirical NTK) to determine the minimum value, maximum value, or eigenvalues corresponding to the input data of the first neural network model. It may be a model for maximizing the average value.

In an alternative embodiment of the method of constructing a dataset, constructing a plurality of candidate unlabeled data groups from the unlabeled data pool comprises a plurality of candidate unlabeled data groups to include in each of the plurality of candidate unlabeled data groups in the unlabeled data pool. The plurality of candidates are randomly extracted from the unlabeled data pool, or a plurality of unlabeled data to be included in each of the plurality of candidate unlabeled data groups is extracted from the unlabeled data pool based on a preset number. Unlabeled data groups can be configured.

In an alternative embodiment of the dataset construction method, the calculating of the representative eigenvalue corresponding to the input dataset includes first eigenvalues corresponding to each of the one or more label data and the candidate unlabeled data group. calculating second eigenvalues corresponding to each of the plurality of unlabeled data, and calculating a representative eigenvalue corresponding to the input dataset based on the calculated first eigenvalues and second eigenvalues. It may further include steps to do.

In an alternative embodiment of the dataset construction method, calculating a representative eigenvalue corresponding to the input dataset comprises: an eigenvalue having a minimum value or a maximum value among the calculated first eigenvalues and second eigenvalues; may be determined as a representative eigenvalue corresponding to the input dataset, or an average value of the calculated first eigenvalues and second eigenvalues may be determined as a representative eigenvalue corresponding to the input dataset.

In an alternative embodiment of the dataset construction method, the step of determining an unlabeled data group for label switching from among the plurality of candidate unlabeled data groups includes a plurality of eigenvalues calculated corresponding to the plurality of input datasets. selecting an input dataset having the highest eigenvalue among the plurality of input datasets or selecting a plurality of input datasets having a high-level eigenvalue among a plurality of eigenvalues calculated corresponding to the plurality of input datasets; and The method may include determining candidate unlabeled data groups included in one or more selected input datasets as the unlabeled data group for label switching.

A computer program stored in a computer readable storage medium according to an embodiment of the present disclosure for realizing the above object, when the computer program is executed on one or more processors, performs the following steps for constructing a dataset wherein the steps include: constructing a plurality of candidate unlabeled data groups from an unlabeled data pool; an input including one candidate unlabeled data group among the plurality of candidate unlabeled data groups and one or more label data; Constructing a dataset, inputting the constructed input dataset into a first neural network model to calculate a representative eigenvalue corresponding to the input dataset, and calculating a plurality of representative eigenvalues and determining an unlabeled data group for label switching from among a plurality of candidate unlabeled data groups based on the above. Each of the plurality of representative eigenvalues may correspond to one input data set.

A computing device for providing a method for configuring a dataset according to an embodiment of the present disclosure for realizing the above object includes a processor including one or more cores and a memory, wherein the processor includes unlabeled data constructing a plurality of candidate unlabeled data groups from a pool, constructing an input dataset including one candidate unlabeled data group of the plurality of candidate unlabeled data groups and at least one label data; is input into the first neural network model to calculate a representative eigenvalue corresponding to the input dataset, and label switching among a plurality of candidate unlabeled data groups based on the calculated representative eigenvalue. It is possible to determine the unlabeled data group for Each of the plurality of representative eigenvalues may correspond to one input data set.

The present invention can provide a method for constructing a dataset for effective model learning.

In order that the above-mentioned features of the present disclosure may be understood in detail and with reference to the more detailed description and the following embodiments, some of the embodiments are shown in the accompanying drawings. Also, like reference numbers in the drawings are intended to refer to the same or similar function throughout the various aspects. However, it should be noted that the accompanying drawings merely illustrate certain exemplary embodiments of the present disclosure and are not considered to limit the scope of the present invention, and that other embodiments having the same effect may be fully appreciated. Be careful.
1 is a block diagram of a computing device that performs an operation for providing a method for constructing a dataset, according to an embodiment of the present disclosure.
2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.
3 is a flowchart illustrating a process of constructing a dataset according to an embodiment of the present disclosure.
4A and 4B are tables showing performance results between a dataset construction method according to an embodiment of the present disclosure and other methods.
5A and 5B are graphs showing performance results between a method for constructing a dataset according to an embodiment of the present disclosure and another method.
6 is a graph showing convergence rate prediction performance results of a dataset construction method according to an embodiment of the present disclosure.
7 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific details.

The terms “component,” “module,” “system,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure, processor, object, thread of execution, program, and/or computer running on a processor. For example, both the server and the application running on the server can be components. One or more components may reside within a processor and/or thread of execution. A component can be localized within a single computer. A component may be distributed between two or more computers. Also, these components can execute from various computer readable media having various data structures stored thereon. Components may be connected, for example, via signals with one or more packets of data (e.g., data and/or signals from one component interacting with another component in a local system, distributed system) to other systems and over a network such as the Internet. data being transmitted) may communicate via local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless otherwise specified or clear from the context, “X employs A or B” is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, if X uses both A and B, "X uses either A or B" may apply to either of these cases. Also, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the features and/or components are present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. Also, unless otherwise specified or where the context clearly indicates that a singular form is indicated, the singular in this specification and claims should generally be construed to mean "one or more".

In addition, the term “at least one of A or B” should be interpreted as meaning “when only A is included”, “when only B is included”, and “when A and B are combined”.

Those skilled in the art will further understand that the various illustrative logical blocks, components, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or combinations of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure.

The description of the presented embodiments is provided to enable any person skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure. The general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features set forth herein.

In one embodiment of the present disclosure, a computing device may encompass a server and/or user equipment. A computing device in the present disclosure may include any type of device having computing capability. A computing device, as a digital device, may be a digital device equipped with a processor and having an arithmetic capability including a memory, such as a laptop computer, a notebook computer, a desktop computer, a web pad, and a mobile phone. In one embodiment, when a computing device corresponds to a server, the computing device may be a web server that processes services. The types of computing devices described above are only examples and the present disclosure is not limited thereto.

Data used throughout the detailed description and claims of the present invention may refer to data containing values or information in any form, such as video or text.

The term "image" or "image data" as used throughout the description and claims of the present invention refers to multidimensional data composed of discrete image elements (e.g., pixels in the case of a two-dimensional image); in other words, ( A term used to refer to a visible object (e.g., displayed on a video screen) or a digital representation of that object (e.g., a file corresponding to the pixel output of a CT, MRI detector, etc.).

For example, “image” or “image” may refer to computed tomography (CT), magnetic resonance imaging (MRI), fundus imaging, ultrasound, or any other medical imaging known in the art. It may be a medical image of a subject collected by the system. The image does not necessarily have to be provided in a medical context, but may be provided in a non-medical context, such as X-ray imaging for security screening.

Throughout the detailed description and claims of the present invention, the 'DICOM (Digital Imaging and Communications in Medicine)' standard is a term that collectively refers to various standards used for digital image expression and communication in medical devices, The DICOM standard is published by a joint committee formed by the American College of Radiology (ACR) and the National Electrical Manufacturers Association (NEMA).

In addition, throughout the detailed description and claims of the present invention, 'Picture Archiving and Communication System (PACS)' is a term that refers to a system that stores, processes, and transmits according to the DICOM standard, and includes X-ray, CT, , Medical imaging images acquired using digital medical imaging equipment such as MRI are stored in DICOM format and can be transmitted to terminals inside and outside the hospital through a network, and reading results and medical records can be added to them.

Also, throughout this specification, neural network, neural network model, model, neural network, or network function may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may be generally referred to as “nodes”. These “nodes” may also be referred to as “neurons”. A neural network includes at least two or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more “links”.

1 is a block diagram of a computing device that performs an operation for providing a method for constructing a dataset, according to an embodiment of the present disclosure.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may constitute the computing device 100.

The computing device 100 may include a processor 110 , a memory 130 , and a network unit 150 .

In the present disclosure, the processor 110 relates to a method of constructing a dataset for learning a classification model, and selects sample data for label conversion for query from an unlabeled data pool, thereby eigenvalues without additional learning of the classification model. Efficient data for learning can be collected from unlabeled data through (eigenvalue) operation.

In general, in order to train a neural network model, a training dataset for the neural network model must be constructed first. In order to learn the neural network model, a training dataset including at least 1,000 to 10,000 pieces of training data may be required. However, when training a neural network model for a specific field, such as the medical field, there are many cases in which learning data or label data for learning are insufficient. For example, in the case of a rare disease, it may be difficult to collect learning data on a rare disease because the probability of occurrence of the rare disease itself is low.

Furthermore, in teacher learning in the field of artificial intelligence, securing learning data and labeling tasks in which a person writes down answers to the learning data account for a large portion. In particular, in the field of medical deep learning, the subject that labels diagnosis results on medical images is bound to be medical professionals. Considering the high labor cost of medical professionals, increasing efficiency in the process of labeling learning data or constructing learning data plays a major role in increasing the reality of business (ie, product price competitiveness).

Therefore, the dataset construction method of the present disclosure is a dataset construction method for learning a classification model, and can collect data effective for learning from unlabeled data through eigenvalue calculation without additional learning of a classification model, Learning data necessary for rare diseases as well as common diseases can be efficiently collected and collected, and efficiency can be achieved in the process of labeling learning data or constructing learning data.

According to an embodiment of the present disclosure, the processor 110 constructs a plurality of candidate unlabeled data groups from an unlabeled data pool, and selects one candidate unlabeled data group from among the plurality of candidate unlabeled data groups. Constructing an input dataset including a data group and one or more labeled data, and inputting the configured input dataset into a first neural network model to calculate a representative eigenvalue corresponding to the input dataset; An unlabeled data group for label switching may be determined from among a plurality of candidate unlabeled data groups based on the calculated plurality of representative eigenvalues. Here, each of the plurality of representative eigenvalues may correspond to one input data set.

The unlabeled data pool of the present disclosure is a set in which only unlabeled data among data necessary for model learning is classified, and the label data pool may refer to a set in which only label data among data necessary for learning is classified. . The candidate unlabel data group of the present disclosure may refer to a group including unlabel data extracted from an unlabel data pool. Here, unlabeled data included in the candidate unlabeled data group may be unlabeled data to be labeled later.

The input data set of the present disclosure is input data input to the first neural network model to determine an unlabel data group for label conversion, and one input data set includes a candidate unlabel data group including unlabel data and It can consist of one or more label data. An unlabeled data group for label switching according to the present disclosure may refer to a candidate unlabeled data group including unlabeled data to be labeled for model learning among candidate unlabeled data groups. For example, an unlabeled data group for label switching may be transferred to another user terminal or server that performs labeling.

A representative eigenvalue of the present disclosure may mean an eigenvalue representing an input dataset composed of one candidate unlabel data group including a plurality of unlabel data and one or more label data.

According to an embodiment of the present disclosure, the processor 110 determines a plurality of unlabeled data included in the determined unlabeled data group for label conversion as data to be labeled, and determines the plurality of unlabeled data as data to be labeled. After labeling is performed, the labeled unlabeled data and one or more labeled data may be configured as a training dataset of a second neural network model for performing a classification task.

According to an embodiment of the present disclosure, the first neural network model uses the Empirical Neural Tangent Kernel (Empirical NTK) to control the convergence rate of the second neural network model. It may be a model for maximizing the minimum value, maximum value, or average value of the eigenvalues corresponding to the input data of . Eigenvalues corresponding to the input data may be eigenvalues of label data constituting the matrix and eigenvalues of unlabeled data. Eigenvalues of label data and unlabel data constituting the matrix will be described later.

In an embodiment, the first neural network model may mean the same model as the second neural network model. In another embodiment, in an actual application process, the first neural network model may be different from the second neural network model.

For example, the same neural network model may mean a model in which at least one of a neural network structure, training method, inference method, operation method, variable, or weight is the same. As another example, different neural network models may refer to models in which at least one of a structure, a training method, an inference method, an operation method, a variable, or a weight of a neural network is different.

According to an embodiment of the present disclosure, when configuring a plurality of candidate unlabeled data groups from the unlabeled data pool, the processor 110 may include a plurality of candidate unlabeled data groups to be included in each of the plurality of candidate unlabeled data groups in the unlabeled data pool. It can be constructed by randomly extracting the unlabeled data of The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, when configuring a plurality of candidate unlabeled data groups from the unlabeled data pool, the processor 110 may include a plurality of candidate unlabeled data groups to be included in each of the plurality of candidate unlabeled data groups in the unlabeled data pool. It can be configured by extracting unlabeled data of based on a preset number. Here, the preset number may be set based on the total number of unlabeled data included in the unlabeled data pool. For example, the preset number may be less than or equal to the total number of unlabeled data included in the unlabeled data pool. That is, if the total number of unlabeled data included in the unlabeled data pool is about 5000 and the preset number of extracted candidate unlabeled data is set to about 50, then 50 candidate unlabeled data from the unlabeled data pool are extracted from the unlabeled data pool. By extracting as a set, one candidate unlabeled data group can be formed. Accordingly, if the total number of unlabeled data included in the unlabeled data pool is about 5000 and the number of candidate unlabeled data extraction settings is set to about 50, about 100 or more candidate unlabeled data groups can be configured. Accordingly, the processor 110 of the present disclosure may optimally determine the size of the candidate unlabeled data group in order to improve network performance by efficiently handling a memory problem while minimizing complexity of eigenvalue calculation. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, when configuring the candidate unlabel data group, the processor 110 configures one candidate unlabel data group including candidate unlabel data extracted from the unlabel data pool, and It is checked whether the total number of unlabeled data groups is the maximum, and if the total number of candidate unlabeled data groups is the maximum, configuration of the candidate unlabeled data groups may be terminated. Here, the processor 110 determines whether the total number of candidate unlabel data groups is the maximum number, and if the total number of candidate unlabel data groups is not the maximum number, the processor 110 extracts at least one candidate unlabel data from the unlabel data pool. A candidate unlabel data group may be configured based on the extracted candidate unlabel data. For example, if the total number of unlabeled data included in the unlabeled data pool is about 5000 and the number of candidate unlabeled data extraction settings is set to about 50, then 50 candidate unlabeled data extracted from the unlabeled data pool are included. Up to about 100 or more candidate unlabeled data groups can be configured. Therefore, when the maximum number of candidate unlabeled data groups to be configured is optimally set in advance according to the total number of unlabeled data included in the unlabeled data pool, if the total number of candidate unlabeled data groups is the preset maximum number, the candidate When the configuration of the unlabeled data groups is finished and the total number of candidate unlabeled data groups is not a preset maximum number, the process of forming the candidate unlabeled data groups may be repeated. The foregoing is only an example, and the present disclosure is not limited thereto.

The maximum number of candidate unlabeled data groups may be set based on the total number of unlabeled data included in the unlabeled data pool and the number of candidate unlabeled data extracted from the unlabeled data pool. For example, if the number of candidate unlabeled data extracted from the unlabeled data pool is equal to the total number of unlabeled data included in the unlabeled data pool, the processor 110 determines that the maximum number of candidate unlabeled data groups is one. Can be set to dogs. For example, if the total number of unlabeled data included in the unlabeled data pool is about 50 and the number of candidate unlabeled data extraction settings is set to about 50, then 50 candidate unlabeled data extracted from the unlabeled data pool are included. A maximum of one candidate unlabeled data group can be configured. The foregoing is only an example, and the present disclosure is not limited thereto.

Further, the processor 110 sets the maximum number of candidate unlabel data groups to plural if the number of candidate unlabel data extracted from the unlabel data pool is smaller than the total number of unlabel data included in the unlabel data pool. can be set. For example, if the total number of unlabeled data included in the unlabeled data pool is about 500 and the number of candidate unlabeled data extraction settings is set to about 50, then 50 candidate unlabeled data extracted from the unlabeled data pool are included. It is possible to configure up to 10 or more candidate unlabeled data groups. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, one candidate unlabel data group includes at least one candidate unlabel data extracted from an unlabel data pool, and the candidate unlabel data included in the one candidate unlabel data group may be repeatedly included in another candidate unlabeled data group. For example, the unlabel data included in the first candidate unlabel data group may be duplicatively included in the second candidate unlabel data group. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, when calculating a representative eigenvalue corresponding to an input dataset, the processor 110 includes first eigenvalues corresponding to one or more label data and a candidate unlabeled data group. Second eigenvalues corresponding to each of the plurality of unlabeled data may be calculated, and a representative eigenvalue corresponding to the input dataset may be calculated based on the calculated first eigenvalues and the second eigenvalues. For example, when an input dataset consisting of one candidate unlabeled data group containing 50 unlabeled data and 100 labeled data is input to the neural network model, one representative eigenvalue representing the input dataset is calculated. It can be. Accordingly, when there are 100 candidate unlabeled data groups, about 100 representative eigenvalues each representing about 100 input datasets can be calculated. Here, the reason for calculating the eigenvalue is to select data that can affect the learning curve convergence of the model in order to minimize the classification error of the classification model. That is, the eigenvalue representing one input dataset including one or more labeled data and one candidate unlabeled data group may be a factor that allows the convergence of the learning curve to be different depending on which data is queried. there is. Accordingly, the processor 110 of the present disclosure can effectively control learning dynamics by querying data that can effectively control the rate of convergence. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, when calculating a representative eigenvalue corresponding to an input dataset, the processor 110 may, among the calculated first eigenvalues and second eigenvalues, have a minimum or maximum eigenvalue. A value may be determined as a representative eigenvalue corresponding to the input data set, or an average value of the calculated first eigenvalues and second eigenvalues may be determined as a representative eigenvalue corresponding to the input data set. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, when determining an unlabeled data group for label conversion from among a plurality of candidate unlabeled data groups, the processor 110 uses a plurality of eigenvalues calculated corresponding to a plurality of input datasets. An input dataset having the highest eigenvalue among them may be selected, and a candidate unlabeled data group included in the selected input dataset may be determined as an unlabeled data group for label switching. For example, if one input dataset consisting of one candidate unlabeled data group containing 50 unlabeled data and 100 labeled data is input to the neural network model, one representative unique representative of the corresponding input dataset is generated. value can be calculated. Therefore, when there are 100 candidate unlabeled data groups, when about 100 representative eigenvalues representing each of the about 100 input datasets are calculated, the input dataset having the highest eigenvalue among the about 100 representative eigenvalues is included in the input dataset. A candidate unlabeled data group may be determined as an unlabeled data group for label conversion. The foregoing is only an example, and the present disclosure is not limited thereto.

Also, according to an embodiment of the present disclosure, when determining an unlabeled data group for label conversion from among a plurality of candidate unlabeled data groups, the processor 110 determines a plurality of unique data groups calculated in correspondence with a plurality of input data sets. Among the values, a plurality of input datasets having high-level eigenvalues may be selected, and a candidate unlabeled data group included in the selected plurality of input datasets may be determined as an unlabeled data group for label switching. Here, when selecting a plurality of input datasets having higher-level eigenvalues among a plurality of eigenvalues calculated corresponding to the input dataset, the processor 110 interlocks with the target number of unlabeled data to be labeled, A plurality of input datasets having unique values of levels may be selected. For example, if one input dataset consisting of one candidate unlabeled data group containing 50 unlabeled data and 100 labeled data is input to the neural network model, one representative unique representative of the corresponding input dataset is generated. value can be calculated. Accordingly, when there are 100 candidate unlabeled data groups, about 100 representative eigenvalues each representing about 100 input datasets can be calculated. Here, if it is necessary to select the target number of 150 unlabeled data to be labeled, the highest first level eigenvalue, the second highest second level eigenvalue, and the third highest eigenvalue among about 100 representative eigenvalues. About 150 pieces of unlabeled data to be labeled can be selected by selecting candidate unlabeled data groups included in each input dataset having 3-level eigenvalues. The foregoing is only an example, and the present disclosure is not limited thereto.

In addition, according to an embodiment of the present disclosure, the processor 110 may select unlabeled data maximizing the minimum value among n eigenvalues in order to control the learning convergence rate of the model, and the maximum value among n eigenvalues. Unlabeled data maximizing may be selected, and unlabeled data maximizing an average value for n eigenvalues may be selected. The foregoing is only an example, and the present disclosure is not limited thereto.

Next, according to an embodiment of the present disclosure, the processor 110 determines a plurality of unlabeled data included in the determined unlabeled data group for label conversion as data to be labeled, and determines the plurality of unlabeled data determined as data to be labeled. After labeling of the data is performed, the labeled unlabeled data and one or more labeled data may be configured as a training dataset of a second neural network model for performing a classification task. In one example, the second neural network model herein may include a model identical to or different from the first neural network model.

Further, according to an embodiment of the present disclosure, the processor 110 may update the unlabeled data pool by labeling unlabeled data to be labeled. For example, the label data pool and the unlabel data pool may be updated by labeling unlabel data included in the candidate unlabel data group, removing the unlabel data from the unlabel data pool, and adding the unlabel data pool to the label data pool. The processor 110 of the present disclosure may increase the size of the label data pool and decrease the size of the unlabel data pool while performing the update process.

According to an embodiment of the present disclosure, the processor 110 can effectively control the learning dynamics of the model by querying data capable of controlling the convergence rate. The processor 110 of the present disclosure may more effectively control the convergence rate based on convergence rate prediction requirements including a learning rate or a learning rate schedule of the model.

According to an embodiment of the present disclosure, the processor 110 may input a set of medical images to a classification model to identify and classify medical images. The medical image may be an image of an organ or an organ including pathology. For example, the medical image may be an image obtained by photographing an organ or the like using MRI, CT, or the like. The medical image may be an image of a photographed organ. Alternatively, the medical image may be an image of an organ including a specific pathology. A medical image set may be a set of medical images. A medical image set may include one or more medical images. A set of medical images may be created on an organ or pathological basis. For example, a medical image set including a medical image of a lung and a medical image set including a medical image of a pancreas may be generated. Alternatively, for example, lung cancer may be classified into squamous cell carcinoma, adenocarcinoma, large cell carcinoma, etc. according to pathology, and a set of medical images may be separately generated for each of squamous cell carcinoma, adenocarcinoma, and large cell carcinoma. there is. The detailed description of the aforementioned medical image set is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 performs an eigenvalue operation without additional learning of the neural network model in a situation where training data for labeling is insufficient when learning the neural network model for the medical field. Efficient data for learning can be collected from label data, so learning data necessary for rare diseases as well as general diseases can be efficiently collected.

According to an embodiment of the present disclosure, a training dataset of a second neural network model for performing a classification task can be efficiently constructed by using the first neural network. Accordingly, efficiency may be achieved in a process of labeling training data or constructing training data.

According to an embodiment of the present disclosure, the processor 110 can effectively control the convergence rate by selecting sample data for label conversion for query from an unlabeled data pool through Equation 1 below.

는, Empirical NTK(Neutral Tangent Kernel) matrix K이고, X_L은, 레이블 데이터 풀(labeled data pool)의 전체 레이블 데이터이며, X_U는, 언레이블 데이터 풀(unlabeled data pool)의 전체 언레이블 데이터이고, X_u는, 언레이블 데이터 풀의 후보 언레이블 데이터 그룹이며, λ_i는, Empirical NTK matrix의 고유값이고, i는, i ∈ {1, ….., │X_L ∪ X_u│}이다.here,

is an Empirical Neutral Tangent Kernel (NTK) matrix K, X _L is all label data in the labeled data pool, X _U is all unlabeled data in the unlabeled data pool, , X _u are candidate unlabeled data groups of the unlabeled data pool, λ _i is an eigenvalue of the Empirical NTK matrix, and i is i ∈ {1, . .., │X _L ∪ X _u │}.

Equation 1 above describes an equation for selecting unlabeled data that maximizes the minimum value among n eigenvalues in order to control the learning convergence rate of the model. Unlabeled data maximizing may be selected, or unlabeled data maximizing an average value for n eigenvalues may be selected. The foregoing is only an example, and the present disclosure is not limited thereto.

As an example, the first neural network model in the present disclosure may operate based on Equation 1 above.

The processor 110 of the present disclosure can collect data effective for learning from unlabeled data through eigenvalue calculation without additional learning of a classification model by selecting sample data for label conversion for query from an unlabeled data pool. can

That is, the processor 110 of the present disclosure constructs a plurality of candidate unlabel data groups from the unlabel data pool, and includes one candidate unlabel data group and one or more label data among the plurality of candidate unlabel data groups. A representative eigenvalue corresponding to the input dataset is calculated by inputting the configured input dataset into a first neural network model, and a plurality of candidate unlabels are performed based on the calculated representative eigenvalues. An unlabeled data group for label switching is determined among data groups, a plurality of unlabeled data included in the determined unlabeled data group for label switching is determined as data to be labeled, and a plurality of unlabeled data determined as data to be labeled After labeling is performed, the labeled unlabeled data and one or more labeled data may be configured as a training dataset of a second neural network model for performing a classification task.

The neural network models (eg, the first b-yural network model and the second neural network model) of the present disclosure may be deep neural networks. Throughout this specification, neural network, network function, and neural network may be used interchangeably. A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can reveal latent structures in data. In other words, it can identify the latent structure of a photo, text, video, sound, or music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). . Deep neural networks include a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), and a deep belief network (DBN). , Q network, U network, Siamese network, and the like.

A convolutional neural network is a type of deep neural network and includes a neural network including a convolutional layer. A convolutional neural network is a type of multilayer perceptron designed to use minimal preprocessing. A CNN can consist of one or several convolutional layers and artificial neural network layers combined with them. CNNs can additionally utilize weights and pooling layers. Thanks to this structure, CNNs can make full use of the two-dimensional structure of the input data. Convolutional neural networks can be used to recognize objects in images. A convolutional neural network can process image data by representing it as a matrix with dimensions. For example, in the case of image data encoded in RGB (red-green-blue), each R, G, and B color may be represented as a two-dimensional (eg, two-dimensional image) matrix. That is, the color value of each pixel of the image data can be a component of a matrix, and the size of the matrix can be the same as the size of the image. Therefore, image data can be represented as three 2-dimensional matrices (a 3-dimensional array of data).

A convolutional process (input/output of a convolutional layer) can be performed by moving the convolutional filter in the convolutional neural network and multiplying the convolutional filter with matrix elements at each position of the image. A convolutional filter may be composed of an n × n matrix. Convolutional filters can consist of filters of fixed shape, which are generally smaller than the total number of pixels in an image. That is, when an m × m image is input to a convolutional layer (for example, a convolutional layer whose size of convolutional filter is n × n), a matrix representing n × n pixels including each pixel of the image It can be component multiplied with this convolutional filter (ie, multiplication of each component of the matrix). A component matching the convolutional filter can be extracted from the image by multiplication with the convolutional filter. For example, a 3 × 3 convolutional filter to extract top and bottom linear components from an image would be constructed as [[0,1,0], [0,1,0], [0,1,0]] can When a 3 × 3 convolutional filter for extracting upper and lower linear components from an image is applied to an input image, upper and lower linear components matching the convolutional filter may be extracted and output from the image. The convolutional layer may apply a convolutional filter to each matrix for each channel representing an image (ie, R, G, B colors in the case of an R, G, B coded image). The convolutional layer may extract features matching the convolutional filter from the input image by applying the convolutional filter to the input image. The filter value of the convolutional filter (that is, the value of each element of the matrix) may be updated by backpropagation during the learning process of the convolutional neural network.

A subsampling layer is connected to the output of the convolutional layer to simplify the output of the convolutional layer, thereby reducing memory usage and computation. For example, if the output of the convolutional layer is input to a pooling layer having a 2 × 2 max pooling filter, the image is compressed by outputting the maximum value included in each patch for each 2 × 2 patch in each pixel of the image. can The aforementioned pooling may be a method of outputting a minimum value of a patch or an average value of a patch, and any pooling method may be included in the present disclosure.

A convolutional neural network may include one or more convolutional layers and subsampling layers. A convolutional neural network may extract features from an image by repeatedly performing a convolutional process and a subsampling process (eg, max pooling described above). Through an iterative convolutional process and subsampling process, a neural network can extract global features of an image.

An output of the convolutional layer or the subsampling layer may be input to a fully connected layer. A fully connected layer is a layer in which all neurons in one layer are connected to all neurons in neighboring layers. A fully connected layer may refer to a structure in which all nodes of each layer are connected to all nodes of other layers in a neural network.

The processor 110 of the present disclosure may include one or more cores, and includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and tensors of the computing device 100. It may include a processor for analyzing unstructured data such as a tensor processing unit (TPU) and deep learning (DL). The processor 110 may perform model learning according to an embodiment of the present disclosure by reading a computer program stored in the memory 130 . According to an embodiment of the present disclosure, the processor 110 may select a set of samples for query from an unlabeled data pool. The processor 110 may perform calculations for neural network learning, such as processing input data for learning in deep learning, extracting features from input data, calculating errors, and updating neural network weights using backpropagation. there is. In the processor 110, at least one of a CPU, a GPGPU, and a TPU may process learning of a network function. For example, the CPU and GPGPU can process learning of network functions and model learning using network functions. In addition, in one embodiment of the present disclosure, the learning of the network function and the learning of the model using the network function may be processed by using processors of a plurality of computing devices together. In addition, a computer program executed in a computing device according to an embodiment of the present disclosure may be a CPU, GPGPU or TPU executable program.

According to an embodiment of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150 .

According to an embodiment of the present disclosure, the memory 130 is a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Memory) Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The computing device 100 may operate in relation to a web storage performing a storage function of the memory 130 on the Internet. The above description of the memory is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the network unit 150 may transmit/receive data for data set configuration and model learning to other computing devices, servers, and the like. The network unit 150 may transmit and receive inferred data to other computing devices, servers, etc. in order to construct a dataset and perform model learning. In addition, the network unit 150 enables communication between a plurality of computing devices so that learning of a network function can be performed in a distributed manner in each of the plurality of computing devices. The network unit 150 may enable communication between a plurality of computing devices to distribute analysis data generation using a network function.

According to an embodiment of the present disclosure, the network unit 150 may be configured regardless of its communication mode, such as wired and wireless, and may be configured in a personal area network (PAN) or a wide area network (WAN). ), etc., can be composed of various communication networks. In addition, the network unit 150 may be the known World Wide Web (WWW), and use a wireless transmission technology used for short-range communication such as Infrared Data Association (IrDA) or Bluetooth. may be The techniques described herein may be used in the networks mentioned above as well as other networks.

As such, the present disclosure can collect data effective for learning from unlabeled data through eigenvalue calculation without additional learning of a classification model by selecting sample data for label conversion for query from an unlabeled data pool. .

That is, the present disclosure provides a process of constructing a plurality of candidate unlabeled data groups from an unlabeled data pool, one candidate unlabeled data group among a plurality of candidate unlabeled data groups, and an input data set including at least one label data. A process of constructing , a process of calculating representative eigenvalues corresponding to the input dataset by inputting the configured input dataset into a first neural network model, and a plurality of candidate unlabeled data groups based on the calculated representative eigenvalues. Among them, the process of determining an unlabeled data group for label switching, a process of determining a plurality of unlabeled data included in the determined unlabeled data group for label switching as data to be labeled, and a process of determining a plurality of unlabeled data included as data to be labeled. By performing a process of constructing the labeled unlabeled data and one or more labeled data into a learning dataset of a second neural network model for performing a classification task by performing labeling on the unlabeled data, data effective for learning from the unlabeled data can be obtained. can be obtained

2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.

Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes one or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link may form a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node in an output node relationship with one node may have an input node relationship with another node, and vice versa. As described above, an input node to output node relationship may be created around a link. More than one output node can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of data of the output node may be determined based on data input to the input node. Here, a link interconnecting an input node and an output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. An output node value may be determined based on the weight.

As described above, in the neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship in the neural network. Characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association between the nodes and links, and a weight value assigned to each link. For example, when there are two neural networks having the same number of nodes and links and different weight values of the links, the two neural networks may be recognized as different from each other.

A neural network may be composed of a set of one or more nodes. A subset of nodes constituting a neural network may constitute a layer. Some of the nodes constituting the neural network may form one layer based on distances from the first input node. For example, a set of nodes having a distance of n from the first input node may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be passed through to reach the corresponding node from the first input node. However, the definition of such a layer is arbitrary for explanation, and the order of a layer in a neural network may be defined in a method different from the above. For example, a layer of nodes may be defined by a distance from a final output node.

An initial input node may refer to one or more nodes to which data is directly input without going through a link in relation to other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. Also, the hidden node may refer to nodes constituting the neural network other than the first input node and the last output node.

In the neural network according to an embodiment of the present disclosure, the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as the number of nodes progresses from the input layer to the hidden layer. can In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of the input layer may be less than the number of nodes of the output layer and the number of nodes decreases as the number of nodes increases from the input layer to the hidden layer. there is. In addition, the neural network according to another embodiment of the present disclosure is a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as the number of nodes increases from the input layer to the hidden layer. can A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can reveal latent structures in data. In other words, it can identify the latent structure of a photo, text, video, sound, or music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), and restricted Boltzmann machines (RBMs). machine), a deep belief network (DBN), a Q network, a U network, a Siamese network, a Generative Adversarial Network (GAN), and the like. The description of the deep neural network described above is only an example, and the present disclosure is not limited thereto.

In one embodiment of the present disclosure, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network for outputting output data similar to input data. An auto-encoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between input and output layers. The number of nodes of each layer may be reduced from the number of nodes of the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically with the reduction from the bottleneck layer to the output layer (symmetrical to the input layer). Autoencoders can perform non-linear dimensionality reduction. The number of input layers and output layers may correspond to dimensions after preprocessing of input data. In the auto-encoder structure, the number of hidden layer nodes included in the encoder may decrease as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be conveyed, so more than a certain number (e.g., more than half of the input layer, etc.) ) may be maintained.

The neural network may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Learning of the neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

A neural network can be trained in a way that minimizes output errors. In the learning of the neural network, the learning data is repeatedly input into the neural network, the output of the neural network for the training data and the error of the target are calculated, and the error of the neural network is transferred from the output layer of the neural network to the input layer in the direction of reducing the error. It is a process of updating the weight of each node of the neural network by backpropagating in the same direction. In the case of supervised learning, each learning data is labeled with the correct answer (ie, labeled learning data), and in the case of unsupervised learning, the correct answer may not be labeled in each learning data. That is, for example, learning data in the case of supervised learning for data classification may be data in which each learning data is labeled with a category. Labeled training data is input to the neural network, and an error may be calculated by comparing an output (category) of the neural network and a label of the training data. As another example, in the case of unsupervised learning for data classification, an error may be calculated by comparing input learning data with a neural network output. The calculated error is back-propagated in a reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back-propagation. The amount of change in the connection weight of each updated node may be determined according to a learning rate. The neural network's computation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of iterations of the learning cycle of the neural network. For example, a high learning rate may be used in the early stage of neural network training to increase efficiency by allowing the neural network to quickly obtain a certain level of performance, and a low learning rate may be used in the late stage to increase accuracy.

In neural network learning, generally, training data can be a subset of real data (ie, data to be processed using the trained neural network), and therefore, errors for training data are reduced, but errors for real data are reduced. There may be incremental learning cycles. Overfitting is a phenomenon in which errors for actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network that has learned a cat by showing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing the error of machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing the training data, regularization, inactivating some nodes in the network during learning, and using a batch normalization layer should be applied. can

3 is a flowchart illustrating a process of constructing a dataset according to an embodiment of the present disclosure.

As shown in FIG. 3 , the computing device of the present disclosure may configure a plurality of candidate unlabeled data groups from the unlabeled data pool (S10). When constructing a plurality of candidate unlabeled data groups from an unlabeled data pool, the computing device of the present disclosure randomly extracts a plurality of unlabeled data to be included in each of the plurality of candidate unlabeled data groups from the unlabeled data pool. can be configured. When constructing a plurality of candidate unlabeled data groups from the unlabeled data pool, the computing device of the present disclosure sets a preset number of pieces of unlabeled data to be included in each of the plurality of candidate unlabeled data groups in the unlabeled data pool. It can be extracted and configured based on. Here, the preset number may be set based on the total number of unlabeled data included in the unlabeled data pool. The computing device of the present disclosure may optimally determine the size of a candidate unlabeled data group in order to improve network performance by efficiently handling a memory problem while minimizing complexity of eigenvalue calculation.

In addition, the computing device of the present disclosure constructs an input dataset including one candidate unlabeled data group among a plurality of candidate unlabeled data groups and one or more label data, and uses the configured input dataset as a first neural network model. It is possible to calculate representative eigenvalues corresponding to the input data set by inputting to (S20). The computing device of the present disclosure calculates first eigenvalues corresponding to each of one or more label data and second eigenvalues corresponding to each of a plurality of unlabeled data included in a candidate unlabel data group, and the calculated first eigenvalues correspond to each other. Based on the eigenvalues and the second eigenvalues, a representative eigenvalue corresponding to the input dataset may be calculated. Here, the reason for calculating the eigenvalue is to select data that can affect the learning curve convergence of the model in order to minimize the classification error of the classification model. That is, a set of eigenvalues including a labeled dataset and one candidate unlabeled dataset may be a factor that makes it possible to know that the convergence of a learning curve varies depending on which data is queried. Accordingly, the processor 110 of the present disclosure can effectively control learning dynamics by querying data that can effectively control the rate of convergence. When calculating a representative eigenvalue corresponding to an input dataset, the computing device of the present disclosure, among the calculated first eigenvalues and second eigenvalues, assigns an eigenvalue having a minimum value or a maximum value to a corresponding eigenvalue corresponding to the input dataset. It may be determined as a representative eigenvalue, or an average value of the calculated first eigenvalues and second eigenvalues may be determined as a representative eigenvalue corresponding to the input data set.

Next, the computing device of the present disclosure may determine an unlabel data group for label switching from among a plurality of candidate unlabel data groups based on the calculated plurality of representative eigenvalues (S30). The computing device of the present disclosure selects an input dataset having the highest eigenvalue among a plurality of eigenvalues calculated corresponding to a plurality of input datasets, and labels a candidate unlabeled data group included in the selected input dataset. It can be determined as an unlabeled data group for conversion. In some cases, the computing device of the present disclosure selects a plurality of input datasets having higher-level eigenvalues from among a plurality of eigenvalues calculated corresponding to the plurality of input datasets, and assigns the selected plurality of input datasets to the plurality of input datasets. A candidate unlabeled data group to be included may be determined as an unlabeled data group for label switching. Here, when selecting a plurality of input datasets having higher-level eigenvalues among a plurality of eigenvalues calculated corresponding to the input dataset, the processor 110 interlocks with the target number of unlabeled data to be labeled, A plurality of input datasets having unique values of levels may be selected.

Subsequently, the computing device of the present disclosure determines a plurality of unlabeled data included in the determined unlabeled data group for label conversion as data to be labeled, and performs labeling by labeling the plurality of unlabeled data determined as data to be labeled. The unlabeled data and one or more labeled data may be configured as a training dataset of a second neural network model for performing a classification task.

The computing device of the present disclosure can effectively control the learning dynamics of a model by querying data capable of controlling the rate of convergence. The computing device of the present disclosure may more effectively control the convergence rate based on convergence rate prediction requirements including a learning rate or a learning rate schedule of the model.

The computing device of the present disclosure may perform an eigenvalue operation for controlling a convergence rate using a neural network tangent kernel (NTK). In the neural network tangent kernel, when the number of channels in all layers goes to infinity, training dynamics can be expressed as Equation 2 below.

의 고유값/고유벡터 쌍

과 상수

를 이용하여 위의 미분 방정식의 해를 표현하면 다음 수학식 3과 같다.Here, the NTK matrix

eigenvalue/eigenvector pair of

and constant

Expressing the solution of the above differential equation using Equation 3 is as follows.

는, 고유값이고, v_i는, 고유벡터이며, c_i는 상수일 수 있다.here,

is an eigenvalue, v _i is an eigenvector, and c _i may be a constant.

는, 해당 고유값에 상응하는 데이터 쿼리에 따라 학습곡선의 수렴성이 달라질 수 있음을 보여준다.Eigenvalue

shows that the convergence of the learning curve can vary depending on the data query corresponding to the corresponding eigenvalue.

Therefore, the computing device of the present disclosure can effectively control the learning dynamics by querying data capable of effectively controlling the convergence rate.

The computing device of the present disclosure can effectively control the convergence rate by selecting sample data for label conversion for query from an unlabeled data pool through Equation 4 below.

는, Empirical NTK matrix K이고, X_L은, 레이블 데이터 풀(labeled data pool)의 전체 레이블 데이터이며, X_U는, 언레이블 데이터 풀(unlabeled data pool)의 전체 언레이블 데이터이고, X_u는, 언레이블 데이터 풀의 후보 언레이블 데이터 그룹이며, λ_i는, Empirical NTK matrix의 고유값이고, i는, i ∈ {1, ……, │X_L ∪ X_u│}이다.here,

is the Empirical NTK matrix K, X _L is the total label data of the labeled data pool, X _U is the total unlabeled data of the unlabeled data pool, and X _u is, A candidate unlabeled data group of the unlabeled data pool, λ _i is an eigenvalue of the Empirical NTK matrix, and i is i ∈ {1, . . . … , │X _L ∪ X _u │}.

의 고유값을 산출하기 위하여, 각각의 블록(block) 행렬을 다음 수학식 5와 같이 대체할 수도 있다.

In order to calculate the eigenvalue of , each block matrix may be replaced as shown in Equation 5 below.

는, Empirical NTK(Neutral Tangent Kernel) matrix K로서,

는, 데이터쌍 x, x'에 의해서 계산되는 블록 행렬이며,

는,

의 차원(dimension)으로서, C는 출력 및 클래스 개수일 수 있다.here,

Is the Empirical Neutral Tangent Kernel (NTK) matrix K,

is a block matrix calculated by the data pair x, x',

Is,

As a dimension of , C may be the number of outputs and classes.

According to Equation 5, if Equation 4 is approximated to reduce the dimensionality, the eigenvalue may be negative. In this case, (1) based on the calculated eigenvalue without considering positive and negative numbers, the input dataset The eigenvalue of may be calculated, or (2) the eigenvalue of the input dataset may be calculated based on the positive eigenvalue. In the case of (2), the positive number can have a greater influence than the negative number on the convergence rate control for the model learning curve, so the eigenvalues of the matrix are calculated based on the eigenvalues of the positive numbers without considering the eigenvalues of the negative numbers. By doing so, it is possible to reduce the amount of computation required compared to Equation 4 and improve network performance.

As an example, the first neural network model in the present disclosure may operate based on Equations 2 to 5 above.

The computing device of the present disclosure may calculate an eigenvalue by inputting an input dataset including label data and one candidate unlabeled data group to a neural network model. Here, the reason for calculating the eigenvalue is to select data that can affect the learning curve convergence of the model in order to minimize the classification error of the classification model. That is, the representative eigenvalue of the input dataset including the label data and one candidate unlabeled data group may be a factor that makes it possible to know that the convergence of the learning curve varies depending on which data is queried.

Also, the computing device of the present disclosure may determine an unlabel data group for label switching from among a plurality of candidate unlabel data groups based on a plurality of representative eigenvalues calculated corresponding to the input data set. For example, the computing device of the present disclosure selects one input dataset having the highest eigenvalue among a plurality of eigenvalues calculated corresponding to the input dataset, and selects a candidate word included in the selected one input dataset. A label data group can be determined as an unlabeled data group for label switching. As another example, the computing device of the present disclosure selects a plurality of input datasets having higher-level eigenvalues among a plurality of eigenvalues calculated corresponding to the input dataset, and includes A plurality of candidate unlabeled data groups may be determined as unlabeled data groups for label conversion.

The computing device of the present disclosure may select unlabeled data maximizing the minimum value among n eigenvalues or select unlabeled data maximizing the maximum value among n eigenvalues in order to control the learning convergence rate of the model. In addition, unlabeled data maximizing the average value of n eigenvalues may be selected.

Since the learning data is insufficient when learning the neural network model for the medical field, the computing device of the present disclosure can collect data effective for learning from unlabeled data through eigenvalue calculation without additional learning of the model, and general disease In addition, learning data required for rare diseases can be efficiently collected.

4A and 4B are tables showing performance results between a dataset construction method according to an embodiment of the present disclosure and other methods.

As shown in FIG. 4A, when the query size, which means the quantity of sample data for label conversion selected from the unlabeled data pool, is 50 and 100, the classification accuracy for labeled images is controlled by the convergence rate of the present disclosure ( It can be seen that the Convergence Rate Control (CRC) method is more improved than other existing random methods, entropy methods, confidence methods, and EGL (Expected Gradient Length) methods. Other existing methods are methods of learning with one candidate unlabeled data randomly extracted from an unlabeled data pool, and it can be seen that the classification accuracy for images is lower than that of the convergence rate control method of the present disclosure.

As shown in FIG. 4B, when the query size for SVHN (Street View House Numbers) data is 20, and the number of randomly sampled images per class is 12 in the first query, the classification accuracy for labeled images is It can be seen that the convergence rate control (CRC) method of the present disclosure is more improved than other existing random methods. It can be seen that other existing random methods have low classification accuracy for images compared to the convergence rate control method of the present disclosure.

5A and 5B are graphs showing performance results between a method for constructing a dataset according to an embodiment of the present disclosure and another method.

As shown in FIG. 5A, when the query size is 50, it is a graph comparing the performance of a plurality of learning models according to the increase in the number of labeled images, and the convergence rate control method of the present disclosure is different from the existing random method. , it can be seen that it shows improved performance compared to the entropy method, the confidence method, and the EGL (Expected Gradient Length) method.

And, as shown in FIG. 5B, a graph showing the performance comparison of learning models when the query step for selecting data samples for query is continuously performed, and the convergence rate control method of the present disclosure is different from the existing random method. , the entropy method, the confidence method, and the EGL (Expected Gradient Length) method, it can be seen that it shows stable performance.

6 is a graph showing convergence rate prediction performance results of a dataset construction method according to an embodiment of the present disclosure.

6 is a graph showing the distribution of minimum eigenvalues according to the number of epochs. As shown in FIG. 6, the convergence rate control method of the present disclosure determines the number of epochs required for convergence through the distribution of minimum eigenvalues. Predictable. Epoch means the number of times learning is completed for the entire data set, and since the convergence rate control method of the present disclosure can predict the number of epochs required for convergence through the distribution of the minimum eigenvalue, efficient learning can be performed. .

7 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Although the present disclosure has been described above as being generally embodied by a computing device, those skilled in the art will understand that the present disclosure may be combined with computer-executable instructions and/or other program modules that may be executed on one or more computers and/or may be implemented in hardware and software. It will be appreciated that it can be implemented as a combination.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. It will also be appreciated by those skilled in the art that the methods of the present disclosure may be used in single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like (each of which is It will be appreciated that other computer system configurations may be implemented, including those that may be operative in connection with one or more associated devices.

The described embodiments of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer readable media. Computer readable media can be any medium that can be accessed by a computer, including volatile and nonvolatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example, and not limitation, computer readable media may include computer readable storage media and computer readable transmission media. Computer readable storage media are volatile and nonvolatile media, transitory and non-transitory, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. includes media Computer readable storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device. device, or any other medium that can be accessed by a computer and used to store desired information.

A computer readable transmission medium typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism. Including all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information within the signal. By way of example, and not limitation, computer readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer readable transmission media.

An exemplary environment 1100 implementing various aspects of the present disclosure is shown including a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to system memory 1106 , to processing unit 1104 . Processing unit 1104 may be any of a variety of commercially available processors. Dual processor and other multiprocessor architectures may also be used as the processing unit 1104.

System bus 1108 may be any of several types of bus structures that may additionally be interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112 . A basic input/output system (BIOS) is stored in non-volatile memory 1110, such as ROM, EPROM, or EEPROM, and is a basic set of information that helps transfer information between components within computer 1102, such as during startup. contains routines. RAM 1112 may also include high-speed RAM, such as static RAM, for caching data.

The computer 1102 may also include an internal hard disk drive (HDD) 1114 (eg, EIDE, SATA) - the internal hard disk drive 1114 may also be configured for external use within a suitable chassis (not shown). Yes—a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from or writing to a removable diskette 1118), and an optical disk drive 1120 (e.g., a CD-ROM) for reading disc 1122 or reading from or writing to other high capacity optical media such as DVDs). The hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 are connected to the system bus 1108 by a hard disk drive interface 1124, magnetic disk drive interface 1126, and optical drive interface 1128, respectively. ) can be connected to The interface 1124 for external drive implementation includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of computer readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art can use zip drives, magnetic cassettes, flash memory cards, cartridges, etc. It will be appreciated that other tangible media readable by the computer, such as the like, may also be used in the exemplary operating environment and any such media may include computer executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored on the drive and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented in a variety of commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 1102 through one or more wired/wireless input devices, such as a keyboard 1138 and a pointing device such as a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. Although these and other input devices are often connected to the processing unit 1104 through an input device interface 1142 that is connected to the system bus 1108, a parallel port, IEEE 1394 serial port, game port, USB port, IR interface, may be connected by other interfaces such as the like.

A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface such as a video adapter 1146. In addition to the monitor 1144, computers typically include other peripheral output devices (not shown) such as speakers, printers, and the like.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148 via wired and/or wireless communications. Remote computer(s) 1148 may be a workstation, computing device computer, router, personal computer, handheld computer, microprocessor-based entertainment device, peer device, or other common network node, and generally includes It includes many or all of the components described for, but for simplicity, only memory storage device 1150 is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, such as a wide area network (WAN) 1154 . Such LAN and WAN networking environments are common in offices and corporations and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to worldwide computer networks, such as the Internet.

When used in a LAN networking environment, computer 1102 connects to local network 1152 through wired and/or wireless communication network interfaces or adapters 1156. Adapter 1156 may facilitate wired or wireless communications to LAN 1152, which also includes a wireless access point installed therein to communicate with wireless adapter 1156. When used in a WAN networking environment, computer 1102 may include a modem 1158, be connected to a communicating computing device on WAN 1154, or establish communications over WAN 1154, such as over the Internet. have other means. A modem 1158, which may be internal or external and a wired or wireless device, is connected to the system bus 1108 through a serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored on remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing a communication link between computers may be used.

Computer 1102 is any wireless device or entity that is deployed and operating in wireless communication, eg, printers, scanners, desktop and/or portable computers, portable data assistants (PDAs), communication satellites, wireless detectable tags associated with It operates to communicate with arbitrary equipment or places and telephones. This includes at least Wi-Fi and Bluetooth wireless technologies. Thus, the communication may be a predefined structure as in conventional networks or simply an ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) makes it possible to connect to the Internet without wires. Wi-Fi is a wireless technology, such as a cell phone, that allows such devices, eg, computers, to transmit and receive data both indoors and outdoors, i.e. anywhere within coverage of a base station. Wi-Fi networks use a radio technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, for example, at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band) .

Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, instructions, information, signals, bits, symbols and chips that may be referenced in the above description are voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields s or particles, or any combination thereof.

Those skilled in the art will understand that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein are electronic hardware, (for convenience) , may be implemented by various forms of program or design code (referred to herein as software) or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and the design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

Various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical disks (eg, CDs, DVDs, etc.), smart cards, and flash memory devices (eg, EEPROM, cards, sticks, key drives, etc.), but are not limited thereto. Additionally, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of example approaches. Based upon design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure, and the general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present disclosure is not to be limited to the embodiments presented herein, but is to be interpreted in the widest scope consistent with the principles and novel features presented herein.

Claims (10)

A dataset construction method performed by a computing device comprising at least one processor, comprising:
constructing a plurality of candidate unlabeled data groups from an unlabeled data pool;
constructing an input data set including one candidate unlabeled data group among the plurality of candidate unlabeled data groups and at least one labeled data;
calculating a representative eigenvalue corresponding to the input data set by inputting the constructed input data set to a first neural network model; and
determining an unlabeled data group for label switching from among a plurality of candidate unlabeled data groups based on the calculated plurality of representative eigenvalues;
Including,
Each of the plurality of representative eigenvalues corresponds to one input data set,
How to organize your dataset. According to claim 1,
determining a plurality of unlabeled data included in the determined unlabeled data group for label conversion as data to be labeled; and
After labeling is performed on the plurality of unlabeled data determined as the data to be labeled, a training dataset of a network model for performing a classification task on the labeled unlabeled data and the one or more label data. Consisting of;
Including more,
How to organize your dataset. According to claim 2,
The first neural network model is a model for maximizing a minimum value, a maximum value, or an average value of eigenvalues corresponding to input data of the first neural network model using an Empirical Neural Tangent Kernel (Empirical NTK),
How to organize your dataset. According to claim 1,
Constituting a plurality of candidate unlabeled data groups from the unlabeled data pool comprises:
A plurality of unlabeled data to be included in each of the plurality of candidate unlabeled data groups is randomly extracted from the unlabeled data pool, or included in each of the plurality of candidate unlabeled data groups in the unlabeled data pool. constituting the plurality of candidate unlabeled data groups in a manner of extracting a plurality of unlabeled data based on a preset number;
How to organize your dataset. According to claim 1,
Calculating a representative eigenvalue corresponding to the input dataset is:
calculating first eigenvalues corresponding to each of the one or more label data and second eigenvalues corresponding to each of a plurality of unlabeled data included in the candidate unlabeled data group; and
calculating a representative eigenvalue corresponding to the input dataset based on the calculated first eigenvalues and second eigenvalues;
including,
How to organize your dataset. According to claim 5,
Calculating a representative eigenvalue corresponding to the input dataset is:
An eigenvalue having a minimum or maximum value among the calculated first eigenvalues and second eigenvalues is determined as a representative eigenvalue corresponding to the input dataset, or
Determining an average value of the calculated first eigenvalues and second eigenvalues as a representative eigenvalue corresponding to the input dataset,
How to organize your dataset. According to claim 1,
Determining an unlabeled data group for label switching from among the plurality of candidate unlabeled data groups includes:
The input data set having the highest eigenvalue among the plurality of eigenvalues calculated corresponding to the plurality of input data sets is selected, or the input data set having the highest eigenvalue is selected from among the plurality of eigenvalues calculated corresponding to the plurality of input data sets. selecting a plurality of input data sets having unique values; and
determining a candidate unlabeled data group included in the selected one or more input datasets as the unlabeled data group for label conversion;
including,
How to organize your dataset. According to claim 7,
The number of the plurality of input datasets having the higher-level eigenvalues is linked to the target number of unlabeled data to be labeled.
How to organize your dataset. A computer program stored on a computer readable storage medium, which, when executed on one or more processors, causes the following steps for constructing a dataset, the steps comprising:
constructing a plurality of candidate unlabeled data groups from the unlabeled data pool;
constructing an input data set including one candidate unlabel data group among the plurality of candidate unlabel data groups and at least one label data;
calculating a representative eigenvalue corresponding to the input data set by inputting the configured input data set to a first neural network model; and
determining an unlabeled data group for label switching from among a plurality of candidate unlabeled data groups based on the calculated plurality of representative eigenvalues;
Including,
Each of the plurality of representative eigenvalues corresponds to one input data set,
A computer program stored on a computer readable storage medium. A computing device for providing a method for constructing a dataset,
a processor comprising one or more cores; and
Memory;
including,
The processor:
constructing a plurality of candidate unlabeled data groups from the unlabeled data pool;
constructing an input data set including one candidate unlabel data group among the plurality of candidate unlabel data groups and at least one label data;
inputting the configured input data set to a first neural network model to calculate a representative eigenvalue corresponding to the input data set; and
determine an unlabeled data group for label conversion from among a plurality of candidate unlabeled data groups based on the calculated plurality of representative eigenvalues; and
Each of the plurality of representative eigenvalues corresponds to one input data set,
computing device.

Images