KR20230046516A - Method, apparatus and program to estimate uncertainty based on test-time mixup augmentation - Google Patents

Abstract

The present invention relates to a method, a device and a program for estimating uncertainty by using a test-time mix-up augmentation method. The method includes the following steps of: acquiring test data and learning data; generating augmentation test data by mixing the test data with at least one learning datum belonging to the same class as the test data or a different class such that the data can be overlapped with each other; generating predicted data by inputting the augmentation test data into a neural network model; and estimating uncertainty about the predicted data.

Description

METHOD, APPARATUS AND PROGRAM TO ESTIMATE UNCERTAINTY BASED ON TEST-TIME MIXUP AUGMENTATION

The present invention relates to a method for estimating uncertainty, and more particularly, to a method, apparatus, and program capable of estimating uncertainty using a test-time mixup augmentation method.

In general, when training a deep learning model, data augmentation can be used when a problem of insufficient data occurs.

For data augmentation, data sets can be inflated using various methods, such as rotating, inverting, flipping, stretching, shrinking, and noise.

Among these various methods, a TTA (Test-Time Augmentation) method is a method of performing data augmentation when a model is tested with a test set or a model is actually operated.

For example, in the TTA method, a plurality of images may be created by augmenting an original input image, and then each may be input to a learned model, and a final result may be derived by averaging all results from the model.

As such, the TTA method minimizes the error rate for image data with high uncertainty by deriving the final result by inputting images from various viewpoints augmented with the original image into the model, rather than predicting the result by inputting only one original image into the model. Thus, the prediction performance can be improved.

However, when uncertainty is estimated using the TTA method, there is a problem of being less sensitive to uncertainty in response to OoD (Out-of-Distribution) data.

Therefore, in the future, it is required to develop an uncertainty estimation method capable of increasing the accuracy of uncertainty estimation.

Republic of Korea Patent No. 10-2200212 (2021. 01. 08)

One object of the present invention to solve the above problems is an uncertainty estimation method, apparatus, and method capable of increasing the accuracy of uncertainty estimation by estimating uncertainty using a test time mixed augmentation method in which data of different classes are mixed. to provide the program.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

An uncertainty estimation method according to an embodiment of the present invention for solving the above problems includes obtaining test data and training data, and at least one training data belonging to the same class as the test data or a different class. It is characterized in that it includes generating augmented test data by mixing x to overlap with each other, generating prediction data by inputting the augmented test data to a neural network model, and estimating uncertainty for the prediction data. .

In an embodiment, the acquiring of the test data and the training data includes acquiring a plurality of training data belonging to second to Nth classes different from the first class when one test data of the first class is obtained. to be characterized

In an embodiment, the acquiring of a plurality of training data belonging to the second to Nth classes may include acquiring at least one training data for each class.

In an embodiment, in the acquiring of the test data and the training data, if one test data of the first class is acquired, the training data of the first class and the second to Nth classes different from the first class are stored. It is characterized in that a plurality of belonging training data is acquired.

In an embodiment, the obtaining of the test data and training data may include acquiring training data belonging to a specific class different from the first class when one test data of the first class is acquired.

In an embodiment, the classes of the test data and learning data are characterized in that a plurality of disease classes for specific lesions of the body are classified.

In an embodiment, the generating of the augmented test data may include, when one test data of a first class and a plurality of training data belonging to second to Nth classes are obtained, one test data of the first class and the one test data of the first class. Augmented test data is generated by mixing one training data belonging to the second to Nth classes in an overlapping manner, and when mixing of all training data belonging to the second to Nth classes with the test data is completed, the augmentation test data is completed. Characterized in that the generation of test data is terminated.

In an embodiment, in the generating of the augmented test data, when one test data of a first class and a plurality of training data belonging to a second to Nth class including one test data belonging to the first class are obtained, One test data of the first class and one training data belonging to the first class are mixed to overlap each other to generate first augmented test data, and the one test data of the first class and the second to Nth classes are mixed. Generating second augmented test data by mixing one training data belonging thereto so as to overlap, and terminating the generation of the augmented test data when mixing of all training data belonging to the second to N th classes with the test data is completed. characterized by

In an embodiment, the generating of the augmented test data may include, when one test data of a first class and learning data belonging to a specific class different from the first class are obtained, one test data of the first class and the specific class. One learning data belonging to a class is mixed to overlap to generate augmented test data, and when all learning data belonging to the specific class are mixed with the test data, the generation of the augmented test data is terminated.

In an embodiment, the generating of the augmented test data may include generating the augmented test data by mixing the images and labels of the test data and the images and labels of the training data so as to overlap.

In an embodiment, the generating of the augmented test data may include mixing the augmented learning data and the test data so as to overlap each other when the acquired training data includes the augmented learning data by modifying the test data. and generating augmented test data.

In an embodiment, the step of estimating the uncertainty of the predicted data may include estimating the uncertainty by generating a class histogram for the predicted data, and determining a degree of similarity by checking a distance between classes.

In an embodiment, in the step of estimating the uncertainty of the predicted data, the accuracy of the predicted data increases as the estimated uncertainty value decreases, and the accuracy of the predicted data decreases as the estimated uncertainty value increases. It is characterized by judging that it is.

A computer program stored on a computer readable storage medium according to an embodiment of the present invention, when executed in one or more processors, performs the following operations for estimating uncertainty, the operations comprising: test data and learning Acquiring data, mixing the test data and at least one training data belonging to the same class as the test data or a different class to overlap each other to generate augmented test data, and inputting the augmented test data to a neural network model It is characterized in that it includes an operation of generating prediction data by doing so, and an operation of estimating an uncertainty of the prediction data.

A computing device for providing an uncertainty estimation method according to an embodiment of the present invention includes a processor including one or more cores and a memory, wherein the processor obtains test data and training data, and the test data and the At least one training data belonging to the same class as the test data or a different class is mixed to overlap each other to generate augmented test data, the augmented test data is input to a neural network model to generate prediction data, and the prediction data is It is characterized by estimating the uncertainty about.

A user terminal for uncertainty estimation according to an embodiment of the present invention includes a processor including one or more cores, a memory, and an output unit providing a user interface, wherein the user interface includes test data and the test data. Displaying uncertainty estimation result information in response to augmented test data obtained by overlapping and mixing at least one training data belonging to a class or another class, and obtaining the test data and training data from the uncertainty estimation result information, generating augmented test data by mixing the test data and at least one training data belonging to the same class as the test data or another class so as to overlap each other, and inputting the augmented test data to a neural network model to generate prediction data; It is characterized in that it is generated by estimating the uncertainty of the prediction data.

In an embodiment, the uncertainty estimation result information may include a similarity of a specific class to another class by uncertainty learning, an accuracy of uncertainty estimation, and an uncertainty distribution map of test data.

In an embodiment, the user interface displays the uncertainty estimation result information in response to a user input, and the uncertainty estimation result information acquires the test data and the training data, and the test data and the test data. At least one training data belonging to the same class as the data or a different class is mixed to overlap each other to generate augmented test data, inputting the augmented test data to a neural network model to generate predictive data, and generating uncertainty about the predicted data. It is characterized in that it is generated by estimating.

A computer program providing a hyoid bone motion tracking method according to another embodiment of the present invention for solving the above problems is combined with a computer that is hardware and stored in a medium to perform any one of the above methods.

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

As described above, according to the present invention, the accuracy of uncertainty estimation can be increased by estimating the uncertainty using the test time mixed augmentation method in which data of different classes are mixed.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram of a computing device that performs an operation for providing an uncertainty estimation method according to an embodiment of the present invention.
2 and 3 are schematic diagrams illustrating network functions, according to an embodiment of the present invention.
4 is a diagram for explaining and comparing the accuracy of uncertainty estimation between TTA and TTMA according to an embodiment of the present invention.
5 is a diagram for comparing and explaining uncertainty distributions of correct samples and incorrect answer samples according to an embodiment of the present invention.
6 is a diagram for explaining the similarity of another class to a specific class through uncertainty learning according to an embodiment of the present invention.
7 is a diagram for comparing and explaining accuracy of uncertainty estimation corresponding to a mixed weight alpha value according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Prior to the description, the meaning of the terms used in this specification will be briefly described. However, it should be noted that the description of terms is intended to help the understanding of the present specification, and is not used in the sense of limiting the technical spirit of the present invention unless explicitly described as limiting the present invention.

In this specification, neural networks, artificial neural networks, and network functions may often be used interchangeably.

The term "image" or "image data" as used throughout the description and claims of the present invention refers to multidimensional data composed of discrete image elements (e.g., pixels in the case of a two-dimensional image); in other words, ( A term used to refer to a visible object (e.g., displayed on a video screen) or a digital representation of that object (e.g., a file corresponding to the pixel output of a CT, MRI detector, etc.).

For example, “image” or “image” may refer to computed tomography (CT), magnetic resonance imaging (MRI), hyoid bone imaging, ultrasound, or any other medical imaging known in the art. It may be a medical image of a subject collected by the system. The image does not necessarily have to be provided in a medical context, but may be provided in a non-medical context, such as X-ray imaging for security screening.

Throughout the detailed description and claims of the present invention, the 'DICOM (Digital Imaging and Communications in Medicine)' standard is a term that collectively refers to various standards used for digital image expression and communication in medical devices, The DICOM standard is published by a joint committee formed by the American College of Radiology (ACR) and the National Electrical Manufacturers Association (NEMA).

In addition, throughout the detailed description and claims of the present invention, 'Picture Archiving and Communication System (PACS)' is a term that refers to a system that stores, processes, and transmits according to the DICOM standard, and includes X-ray, CT, , Medical imaging images acquired using digital medical imaging equipment such as MRI are stored in DICOM format and can be transmitted to terminals inside and outside the hospital through a network, and reading results and medical records can be added to them.

Also, throughout this specification, a neural network, a neural network, and a network function may be used with the same meaning. A neural network may consist of a set of interconnected computational units, which may be generally referred to as “nodes”. These “nodes” may also be referred to as “neurons”. A neural network includes at least two or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more “links”.

1 is a block diagram of a computing device that performs an operation for providing an uncertainty estimation method according to an embodiment of the present invention.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present invention, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may constitute the computing device 100.

The computing device 100 may include a processor 110 , a memory 130 , and a network unit 150 .

In the present invention, the processor 110 relates to a method for estimating uncertainty using a test-time mixup augmentation method, and acquires test data and training data, and obtains test data and test data. Augmented test data is generated by mixing at least one training data belonging to the same class or another class so as to overlap with each other, predictive data is generated by inputting the augmented test data to a neural network model, and uncertainty for the predicted data is estimated. can

Here, the test data and the training data may include at least one of video data, audio data, and time series data. That is, any type of data by which a person engaged in the medical industry or a device for diagnosis can determine whether or not a disease exists in the data may be included in the medical data. The image data includes all image data that is output after photographing or measuring a patient's affected part through examination equipment and turning it into an electrical signal. The image data may include image data constituting each frame of a video in a video continuously photographed over time from a medical imaging device. For example, it includes ultrasound examination image data, image data by an MRI device, CT tomography image data, X-ray image data, and the like. Furthermore, when audio data is converted into an electrical signal and output as an image in the form of a graph or time-series data is expressed as visualized data such as a graph, the image or data may be included in the video data. For example, medical data may include CT images. The foregoing example of medical data is only an example and does not limit the present disclosure.

Next, as an embodiment, when acquiring test data and learning data, the processor 110 acquires one test data of the first class, a plurality of data belonging to second to Nth classes different from the first class. Learning data can be acquired.

Here, the processor 110 may obtain at least one training data for each class when acquiring a plurality of training data belonging to the second to Nth classes.

For example, the number of learning data acquired in each class may be the same number.

As another example, the number of learning data acquired in each class may be an uneven number.

Then, as another embodiment, the processor 110, when acquiring test data and learning data, acquires one test data of the first class, the learning data of the first class and second class or classes different from the first class. A plurality of training data belonging to the Nth class may be acquired.

Here, the training data of the first class may be data different from the test data of the first class.

In some cases, the training data of the first class may be a plurality of data different from the test data of the first class.

In addition, the processor 110 may obtain at least one training data for each class when acquiring a plurality of training data belonging to the second to Nth classes.

Here, the number of learning data acquired in each class may be the same number.

In some cases, the number of learning data acquired in each class may be an uneven number.

Next, as another embodiment, when obtaining test data and training data, the processor 110 may obtain training data belonging to a specific class different from the first class if one test data of the first class is obtained. there is.

Here, the processor 110 may acquire a plurality of learning data belonging to a specific class when obtaining learning data belonging to a specific class.

In some cases, when obtaining training data belonging to a specific class, the processor 110 may obtain one training data belonging to a specific class.

In the present invention, the classes of the test data and the training data may be classified into a plurality of disease classes for specific lesions of the body, which is only an example, but is not limited thereto.

Further, when generating the augmented test data, the processor 110 obtains one test data of the first class and a plurality of training data belonging to the second to Nth classes, one test data of the first class and the second class. One training data belonging to the 2nd to Nth class is mixed to overlap to generate augmented test data, and when all training data belonging to the 2nd to Nth class is mixed with the test data, augmented test data is generated. can be terminated

In some cases, when generating the augmented test data, the processor 110 acquires one test data of the first class and a plurality of training data belonging to the second to Nth classes including training data belonging to the first class. Then, one test data of the first class and one training data belonging to the first class are mixed to overlap each other to generate first augmented test data, and one test data of the first class and learning data belonging to second to Nth classes are mixed. Second augmented test data is generated by mixing one piece of data in an overlapping manner, and when all learning data belonging to the second to Nth classes are completely mixed with the test data, generation of the augmented test data may be terminated.

In another case, when the processor 110 generates the augmented test data, when one test data of the first class and learning data belonging to a specific class different from the first class are obtained, one test data of the first class and a specific class are obtained. One training data belonging to a class is mixed to overlap to generate augmented test data, and when all training data belonging to a specific class is mixed with the test data, the generation of augmented test data may be terminated.

Also, when generating the augmented test data, the processor 110 may generate the augmented test data by mixing the images and labels of the test data and the images and labels of the training data so as to overlap each other.

Then, when generating the augmented test data, the processor 110 modifies the test data among the acquired learning data and, when the augmented learning data is included, mixes the augmented learning data and the test data so as to overlap each other for the augmented test. data can be generated.

Here, the transformation of the test data may include performing augmentation through various transformations such as rotation, inversion, flipping, stretching, reduction, noise, etc. of the data set, which is only one embodiment, but is not limited thereto.

Next, when estimating the uncertainty of the prediction data, the processor 110 may generate a class histogram for the prediction data to estimate the uncertainty, and determine the degree of similarity by checking the distance between the classes.

Here, when estimating the uncertainty of the predicted data, the processor 110 determines that the accuracy of the predicted data increases as the estimated uncertainty value decreases, and the accuracy of the predicted data decreases as the estimated uncertainty value increases. can

And, when estimating the uncertainty of the prediction data, the processor 110 may estimate the uncertainty of the prediction data and generate uncertainty estimation result information.

Here, the uncertainty estimation result information may include the similarity of another class to a specific class by uncertainty learning, the accuracy of uncertainty estimation, and the uncertainty distribution of test data, which is only one embodiment, but is not limited thereto. .

Also, the aforementioned neural network model may be a deep neural network. Throughout this specification, neural network, network function, and neural network may be used interchangeably. A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can reveal latent structures in data. In other words, it can identify the latent structure of a photo, text, video, sound, or music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). . Deep neural networks include a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), and a deep belief network (DBN). , Q network, U network, Siamese network, and the like.

A convolutional neural network is a type of deep neural network and includes a neural network including a convolutional layer. A convolutional neural network is a type of multilayer perceptron designed to use minimal preprocessing. A CNN may be composed of one or several convolutional layers and artificial neural network layers combined therewith. CNNs can additionally utilize weights and pooling layers. Thanks to this structure, CNNs can fully utilize the two-dimensional structure of the input data. Convolutional neural networks can be used to recognize objects in images. A convolutional neural network can represent and process image data as a matrix having dimensions. For example, in the case of image data encoded in RGB (red-green-blue), each R, G, and B color may be represented as a two-dimensional (eg, two-dimensional image) matrix. That is, the color value of each pixel of the image data can be a component of a matrix, and the size of the matrix can be the same as the size of the image. Thus, image data can be represented by three two-dimensional matrices (a three-dimensional data array).

In a convolutional neural network, a convolutional process (input/output of a convolutional layer) can be performed by moving the convolutional filter and multiplying the convolutional filter with matrix components at each position of the image. A convolutional filter may be composed of an n*n matrix. A convolutional filter can consist of a fixed shape filter, which is usually smaller than the total number of pixels in an image. That is, when an m*m image is input to a convolutional layer (for example, a convolutional layer whose size of convolutional filter is n*n), a matrix representing n*n pixels including each pixel of the image It can be component multiplied with this convolutional filter (ie, multiplication of each component of the matrix). A component matching the convolutional filter can be extracted from the image by multiplication with the convolutional filter. For example, a 3*3 convolutional filter to extract top and bottom linear components from an image would be constructed as [[0,1,0], [0,1,0], [0,1,0]] can When a 3*3 convolutional filter for extracting up and down linear components from an image is applied to an input image, up and down linear components matching the convolutional filter may be extracted from the image and output. The convolutional layer may apply a convolutional filter to each matrix for each channel representing an image (ie, R, G, B colors in the case of an R, G, B coded image). The convolutional layer may extract features matching the convolutional filter from the input image by applying the convolutional filter to the input image. The filter value of the convolutional filter (that is, the value of each element of the matrix) may be updated by backpropagation during the learning process of the convolutional neural network.

A subsampling layer is connected to the output of the convolutional layer to simplify the output of the convolutional layer, thereby reducing memory usage and computational complexity. For example, if the output of the convolutional layer is input to a pooling layer with a 2*2 max pooling filter, the image is compressed by outputting the maximum value included in each patch for each 2*2 patch in each pixel of the image. can The aforementioned pooling may be a method of outputting a minimum value of a patch or an average value of a patch, and any pooling method may be included in the present invention.

A convolutional neural network may include one or more convolutional layers and subsampling layers. A convolutional neural network may extract features from an image by repeatedly performing a convolutional process and a subsampling process (eg, max pooling described above). Through an iterative convolutional process and subsampling process, a neural network can extract global features of an image.

An output of the convolutional layer or the subsampling layer may be input to a fully connected layer. A fully connected layer is a layer in which all neurons in one layer are connected to all neurons in neighboring layers. A fully connected layer may refer to a structure in which all nodes of each layer are connected to all nodes of other layers in a neural network.

According to an embodiment of the present invention, the processor 110 may be composed of one or more cores, a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU) ), a processor for data analysis and deep learning, such as a tensor processing unit (TPU). The processor 110 may read a computer program stored in the memory 130 and perform data processing for machine learning according to an embodiment of the present invention. According to an embodiment of the present invention, the processor 110 may perform an operation for learning a neural network. The processor 110 performs neural network learning, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating neural network weights using backpropagation. calculations can be performed for At least one of the CPU, GPGPU, and TPU of the processor 110 may process learning of the network function. For example, the CPU and GPGPU can process learning of network functions and data classification using network functions. In addition, in one embodiment of the present invention, the learning of a network function and data classification using a network function may be processed by using processors of a plurality of computing devices together. In addition, a computer program executed in a computing device according to an embodiment of the present invention may be a CPU, GPGPU or TPU executable program.

According to an embodiment of the present invention, the memory 130 may store a computer program for providing an uncertainty estimation method, and the stored computer program may be read and driven by the processor 120 . The memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150 .

According to an embodiment of the present invention, the memory 130 is a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Memory) Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The computing device 100 may operate in relation to a web storage that performs a storage function of the memory 130 on the Internet. The description of the above memory is only an example, and is not limited thereto.

The network unit 150 according to an embodiment of the present invention may transmit and receive uncertainty estimation result information and the like to other computing devices and servers. In addition, the network unit 150 may enable communication between a plurality of computing devices so that operations for uncertainty estimation or model learning may be distributedly performed in each of the plurality of computing devices. The network unit 150 may enable communication between a plurality of computing devices to perform distributed processing of calculations for uncertainty estimation or model learning using a network function.

The network unit 150 according to an embodiment of the present invention may operate based on any type of wired or wireless communication technology currently used and implemented, such as short-distance (short-distance), long-distance, wired, and wireless, and other networks. can also be used in

The computing device 100 of the present invention may further include an output unit and an input unit.

The output unit according to an embodiment of the present invention may display a user interface (UI) for providing uncertainty estimation results. The output unit may output any type of information generated or determined by the processor 110 and any type of information received by the network unit 150 .

In one embodiment of the present invention, the output unit is a liquid crystal display (liquid crystal display, LCD), thin film transistor liquid crystal display (thin film transistor-liquid crystal display, TFT LCD), organic light-emitting diode (organic light-emitting diode, OLED) , a flexible display, and a 3D display. Some of these display modules may be of a transparent type or a light transmissive type so that the outside can be seen through them. This may be referred to as a transparent display module, and a representative example of the transparent display module is TOLED (Transparent OLED) and the like.

The input unit according to an embodiment of the present invention may receive a user input. The input unit may include keys and/or buttons on a user interface for receiving user input, or physical keys and/or buttons. A computer program for controlling a display according to embodiments of the present invention may be executed according to a user input through an input unit.

The input unit according to embodiments of the present invention may detect a user's button operation or touch input and receive a signal, or may receive a user's voice or motion through a camera or microphone and convert it into an input signal. For this purpose, speech recognition technology or motion recognition technology may be used.

The input unit according to embodiments of the present invention may be implemented as an external input device connected to the computing device 100 . For example, the input device may be at least one of a touch pad, a touch pen, a keyboard, or a mouse for receiving a user input, but this is only an example and is not limited thereto.

The input unit according to an embodiment of the present invention may recognize a user touch input. The input unit according to an embodiment of the present invention may have the same configuration as the output unit. The input unit may include a touch screen implemented to receive a user's selection input. The touch screen may use any one of a contact capacitive method, an infrared light sensing method, a surface ultrasonic (SAW) method, a piezoelectric method, and a resistive film method. Detailed description of the touch screen described above is only an example according to an embodiment of the present invention, and various touch screen panels may be employed in the computing device 100 . The input unit configured as a touch screen may include a touch sensor. The touch sensor may be configured to convert a change in pressure applied to a specific portion of the input unit or capacitance generated at a specific portion of the input unit into an electrical input signal. The touch sensor may be configured to detect not only the touched position and area, but also the pressure upon touch. When there is a touch input to the touch sensor, the corresponding signal(s) is sent to the touch controller. The touch controller may process the signal(s) and then transmit corresponding data to processor 110 . Accordingly, the processor 110 can recognize which area of the input unit has been touched.

In one embodiment of the present invention, the server may include other configurations for performing the server environment of the server. The server may include any type of device. The server may be a digital device, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, equipped with a processor and having an arithmetic capability with a memory.

A server (not shown) performing an operation for providing a user interface displaying an uncertainty estimation result to a user terminal according to an embodiment of the present invention may include a network unit, a processor, and a memory.

The server may generate a user interface according to embodiments of the present invention. The server may be a computing system that provides information to clients (eg, user terminals) over a network. The server may transmit the generated user interface to the user terminal. In this case, the user terminal may be any type of computing device 100 capable of accessing the server. The processor of the server may transmit the user interface to the user terminal through the network unit. A server according to embodiments of the present invention may be, for example, a cloud server. The server may be a web server that processes services. The types of servers described above are examples only and are not limited thereto.

Accordingly, the present invention can increase the accuracy of uncertainty estimation by estimating uncertainty using a test time mixed augmentation method in which data of different classes are mixed.

2 and 3 are schematic diagrams illustrating network functions, according to an embodiment of the present invention.

As shown in FIG. 2 , according to the present invention, test data and training data for estimating uncertainty may be obtained using a test-time mixup augmentation method.

Here, in the present invention, when one test data of the first class is obtained, a plurality of training data belonging to second to Nth classes different from the first class may be obtained.

At this time, in the present invention, when obtaining a plurality of learning data belonging to the second to Nth classes, at least one learning data for each class may be obtained.

For example, the number of learning data acquired in each class may be the same number or may not be the same number but may be an uneven number.

In some cases, the present invention, when acquiring one test data of the first class, learning data belonging to the same first class as the test data and a plurality of training data belonging to second to Nth classes different from the first class may also be obtained.

Here, the training data of the first class may be one piece of data different from the test data of the first class, or may be a plurality of data different from the test data of the first class.

In addition, in the present invention, when acquiring a plurality of training data belonging to the second to Nth classes, at least one training data for each class may be obtained.

Here, the number of learning data acquired in each class may be the same number or may not be the same number but may be an uneven number.

Next, the present invention may generate augmented test data by mixing the test data and at least one training data belonging to the same class as the test data or another class so as to overlap each other.

Here, in the present invention, when one test data of the first class and a plurality of training data belonging to the second to Nth classes are acquired, one test data of the first class and training data belonging to the second to Nth classes Augmented test data may be generated by mixing one to overlap, and when all learning data belonging to the second to Nth classes are mixed with the test data, generation of the augmented test data may be terminated.

In some cases, the present invention, when a plurality of training data belonging to the second to Nth classes including one test data of the first class and training data belonging to the first class are obtained, one test data of the first class and One training data belonging to the first class is mixed to overlap to generate first augmented test data, and one test data of the first class and one training data belonging to the second to Nth classes are mixed to overlap each other to generate the second augmented test data. Augmented test data is generated, and when all learning data belonging to the second to Nth classes are completely mixed with the test data, the augmented test data generation may be terminated.

Further, according to the present invention, prediction data may be generated by inputting augmented test data to a neural network model, and uncertainty of the prediction data may be estimated.

Here, the present invention can estimate the uncertainty by generating a class histogram for the prediction data, and determine the degree of similarity by checking the distance between the classes.

That is, according to the present invention, it can be determined that the accuracy of the prediction data increases as the estimated uncertainty value decreases, and the accuracy of the prediction data decreases as the estimated uncertainty value increases.

In some cases, the present invention may estimate uncertainty of prediction data and generate uncertainty estimation result information.

Here, the uncertainty estimation result information may include the similarity of another class to a specific class by uncertainty learning, the accuracy of uncertainty estimation, and the uncertainty distribution of test data, which is only one embodiment, but is not limited thereto. .

As shown in FIG. 3 , when acquiring test data and training data, if one test data of the first class is obtained, training data belonging to a specific class different from the first class may be acquired.

Here, in the present invention, a plurality of learning data belonging to a specific class may be obtained, or one learning data belonging to a specific class may be obtained.

In the present invention, when one test data of the first class and training data belonging to a specific class different from the first class are acquired, one test data of the first class and one training data belonging to the specific class are overlapped and mixed for an augmented test. When data is generated and all training data belonging to a specific class is mixed with test data, augmented test data generation may be terminated.

Therefore, the present invention can determine the similarity of another class to a specific class through uncertainty learning.

That is, in the present invention, the degree of similarity can be determined by checking the distance between classes.

In the present invention, it may be determined that the accuracy of the prediction data increases as the estimated uncertainty value decreases, and the accuracy of the prediction data decreases as the estimated uncertainty value increases.

Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes one or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link may form a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node in an output node relationship with one node may have an input node relationship with another node, and vice versa. As described above, an input node to output node relationship may be created around a link. More than one output node can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of data of the output node may be determined based on data input to the input node. Here, a link interconnecting an input node and an output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. An output node value may be determined based on the weight.

As described above, in the neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship in the neural network. Characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association between the nodes and links, and a weight value assigned to each link. For example, when there are two neural networks having the same number of nodes and links and different weight values of the links, the two neural networks may be recognized as different from each other.

A neural network may be composed of a set of one or more nodes. A subset of nodes constituting a neural network may constitute a layer. Some of the nodes constituting the neural network may form one layer based on distances from the first input node. For example, a set of nodes having a distance of n from the first input node may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be passed through to reach the corresponding node from the first input node. However, the definition of such a layer is arbitrary for explanation, and the order of a layer in a neural network may be defined in a method different from the above. For example, a layer of nodes may be defined by a distance from a final output node.

An initial input node may refer to one or more nodes to which data is directly input without going through a link in relation to other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. Also, the hidden node may refer to nodes constituting the neural network other than the first input node and the last output node.

In the neural network according to an embodiment of the present invention, the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as the number of nodes progresses from the input layer to the hidden layer. can In addition, the neural network according to another embodiment of the present invention may be a neural network in which the number of nodes of the input layer may be less than the number of nodes of the output layer, and the number of nodes decreases as the number of nodes increases from the input layer to the hidden layer. there is. In addition, the neural network according to another embodiment of the present invention is a type of neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as the number of nodes progresses from the input layer to the hidden layer. can A neural network according to another embodiment of the present invention may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can reveal latent structures in data. In other words, it can identify the latent structure of a photo, text, video, sound, or music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), and restricted boltzmann machines (RBMs). machine), a deep belief network (DBN), a Q network, a U network, a Siamese network, a Generative Adversarial Network (GAN), and the like. The description of the deep neural network described above is only an example and is not limited thereto.

In one embodiment of the present invention, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network for outputting output data similar to input data. An auto-encoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between input and output layers. The number of nodes of each layer may be reduced from the number of nodes of the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically with the reduction from the bottleneck layer to the output layer (symmetrical to the input layer). Autoencoders can perform non-linear dimensionality reduction. The number of input layers and output layers may correspond to dimensions after preprocessing of input data. In the auto-encoder structure, the number of hidden layer nodes included in the encoder may decrease as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be conveyed, so more than a certain number (e.g., more than half of the input layer, etc.) ) may be maintained.

The neural network may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Learning of the neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

A neural network can be trained in a way that minimizes output errors. In the learning of the neural network, the learning data is repeatedly input into the neural network, the output of the neural network for the training data and the error of the target are calculated, and the error of the neural network is transferred from the output layer of the neural network to the input layer in the direction of reducing the error. It is a process of updating the weight of each node of the neural network by backpropagating in the same direction. In the case of teacher learning, the learning data in which the correct answer is labeled is used for each learning data (ie, the labeled learning data), and in the case of comparative teacher learning, the correct answer may not be labeled in each learning data. That is, for example, learning data in the case of teacher learning regarding data classification may be data in which each learning data is labeled with a category. Labeled training data is input to a neural network, and an error may be calculated by comparing an output (category) of the neural network and a label of the training data. As another example, in the case of comparative history learning for data classification, an error may be calculated by comparing input learning data with a neural network output. The calculated error is back-propagated in a reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back-propagation. The amount of change in the connection weight of each updated node may be determined according to a learning rate. The neural network's calculation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of iterations of the learning cycle of the neural network. For example, a high learning rate may be used in the early stage of neural network training to increase efficiency by allowing the neural network to quickly obtain a certain level of performance, and a low learning rate may be used in the late stage to increase accuracy.

In neural network learning, generally, training data can be a subset of real data (ie, data to be processed using the trained neural network), and therefore, errors for training data are reduced, but errors for real data are reduced. There may be incremental learning cycles. Overfitting is a phenomenon in which errors for actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network that has learned a cat by showing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing the error of machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing the training data, regularization, inactivating some nodes in the network during learning, and using a batch normalization layer should be applied. can

4 is a diagram for comparing and explaining the accuracy of uncertainty estimation of TTA and TTMA according to an embodiment of the present invention. It is a graph comparing the accuracy of data and the accuracy of test data estimated by the lower k% uncertainty using the TTMA (Test-Time Mixup Augmentation) method of the present invention.

As shown in FIG. 4, it can be seen that the TTMA method of the present invention has higher accuracy for data with higher certainty than the existing TTA method.

For example, the TTMA method of the present invention shows better accuracy than the existing TTA method in subgroups of lower k% uncertainty of 70% or less.

5 is a diagram for comparing and explaining uncertainty distributions of correct samples and incorrect answer samples according to an embodiment of the present invention.

As shown in FIG. 5, when comparing uncertainty distributions of correct and incorrect samples through the TTMA method of the present invention and the existing TTA method, in an ideal state, correct samples are concentrated in low-value uncertainties, and incorrect answer samples are However, it is necessary to separate these two groups as they are driven by high values of uncertainty.

Here, in the case of the conventional TTA method, it is difficult to separate the two groups because the correct and incorrect samples are concentrated in a low value of uncertainty, but in the case of the TTMA method of the present invention, the correct and incorrect samples have a wide range of uncertainties. Distributed, the separation of the two groups can have an advantageous advantage.

6 is a diagram for explaining the degree of similarity of another class to a specific class through uncertainty learning according to an embodiment of the present invention, and is a diagram in which skin lesions are classified into a plurality of disease classes.

As shown in FIG. 6, the TTMA method of the present invention measures the similarity between a specific disease class and other disease classes and arranges disease classes having a high average value from a disease class having a low average value, thereby different disease classes. Similarity can be judged through the distance between fields, and similarity can be judged even between the same disease classes.

7 is a diagram for comparing and explaining the accuracy of uncertainty estimation corresponding to a mixed weight alpha value according to an embodiment of the present invention, when mixing test data and training data, according to a change in the mixed weight alpha value This graph measures the change in uncertainty.

As shown in FIG. 7, the TTMA method of the present invention, in terms of accuracy, shows high accuracy when the alpha value is 0.4 or 1, and in terms of ECE (Expected Calibration Error), it appears good when it is 0.8. Able to know.

As described above, the present invention can increase the accuracy of uncertainty estimation by estimating uncertainty using a test time mixed augmentation method in which data of different classes are mixed.

The method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a server, which is hardware, and stored in a medium.

The aforementioned program is C, C++, JAVA, machine language, etc. It may include a code coded in a computer language of. These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. there is. In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected through a network, and computer readable codes may be stored in a distributed manner.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (20)

An uncertainty estimation method performed by a computing device,
acquiring test data and training data;
generating augmented test data by mixing the test data and at least one training data belonging to the same class as the test data or a different class so as to overlap each other;
generating prediction data by inputting the augmented test data into a neural network model; and
An uncertainty estimation method comprising the step of estimating the uncertainty of the predicted data. According to claim 1,
Acquiring the test data and training data,
An uncertainty estimation method characterized in that when one test data of the first class is acquired, a plurality of training data belonging to a second to Nth class different from the first class is acquired. According to claim 2,
Acquiring a plurality of learning data belonging to the second to Nth classes,
An uncertainty estimation method characterized by acquiring at least one training data for each class. According to claim 1,
Acquiring the test data and training data,
When one test data of the first class is acquired, the learning data of the first class and a plurality of training data belonging to second to Nth classes different from the first class are obtained. According to claim 1,
Acquiring the test data and training data,
An uncertainty estimation method characterized by acquiring learning data belonging to a specific class different from the first class when one test data of the first class is acquired. According to claim 1,
Classes of the test data and training data,
A method for estimating uncertainty characterized by being classified into a number of disease classes for specific lesions in the body. According to claim 1,
Generating the augmented test data,
When one test data of the first class and a plurality of training data belonging to the second to Nth classes are obtained, the one test data of the first class and one training data belonging to the second to Nth classes are overlapped. Mixing as much as possible to generate augmented test data,
and ending the generation of the augmented test data when mixing of all training data belonging to the second to Nth classes with the test data is completed. According to claim 1,
Generating the augmented test data,
When a plurality of training data belonging to the second to N-th classes including one test data of the first class and training data belonging to the first class are acquired, the one test data of the first class and the training data belonging to the first class Mixing one training data to overlap to generate first augmented test data;
generating second augmented test data by mixing one test data of the first class and one learning data belonging to the second to Nth classes in an overlapping manner;
and ending the generation of the augmented test data when mixing of all training data belonging to the second to Nth classes with the test data is completed. According to claim 1,
Generating the augmented test data,
When one test data of the first class and training data belonging to a specific class different from the first class are acquired, the one test data of the first class and one training data belonging to the specific class are mixed so as to overlap the augmented test data. create,
and ending the generation of the augmented test data when mixing of all training data belonging to the specific class with the test data is completed. According to claim 1,
Generating the augmented test data,
The method of estimating uncertainty characterized in that the augmented test data is generated by mixing the images and labels of the test data and the images and labels of the training data so as to overlap. According to claim 1,
Generating the augmented test data,
and generating augmented test data by mixing the pre-augmented learning data and the task data so as to overlap each other when the acquired training data includes the pre-augmented training data by transforming the test data. According to claim 1,
The step of estimating the uncertainty of the prediction data,
Uncertainty estimation method, characterized in that for estimating the uncertainty by generating a class histogram for the predicted data, and determining the degree of similarity by checking the distance between the classes. According to claim 12,
The step of estimating the uncertainty of the prediction data,
The uncertainty estimation method characterized in that it is determined that the accuracy of the prediction data increases as the estimated uncertainty value decreases, and the accuracy of the prediction data decreases as the estimated uncertainty value increases. According to claim 1,
The step of estimating the uncertainty of the prediction data,
An uncertainty estimation method characterized in that for estimating the uncertainty of the predicted data and generating uncertainty estimation result information. According to claim 14,
The uncertainty estimation result information,
An uncertainty estimation method characterized in that it includes a similarity of another class to a specific class by uncertainty learning, accuracy for uncertainty estimation, and an uncertainty distribution map for test data. A computer program stored on a computer readable storage medium, which, when executed on one or more processors, causes the following operations for uncertainty estimation, the operations to:
acquiring test data and training data;
generating augmented test data by mixing the test data and at least one training data belonging to the same class as the test data or a different class so as to overlap each other;
generating prediction data by inputting the augmented test data into a neural network model; and
A computer program stored in a computer readable storage medium comprising an operation of estimating uncertainty of the prediction data. A computing device for providing an uncertainty estimation method, comprising:
a processor comprising one or more cores; and
Memory;
including,
the processor,
Acquiring test data and training data;
generating augmented test data by mixing the test data and at least one training data belonging to the same class as the test data or a different class so as to overlap each other;
inputting the augmented test data into a neural network model to generate predictive data; and
A computing device characterized in that for estimating uncertainty for the prediction data. As a user terminal for uncertainty estimation,
a processor comprising one or more cores;
Memory; and
Includes an output unit providing a user interface;
The user interface,
displaying uncertainty estimation result information in response to augmented test data obtained by overlapping and mixing test data and at least one training data belonging to the same class as the test data or a different class; and
The uncertainty estimation result information,
The test data and the training data are obtained, the test data and at least one training data belonging to the same class as the test data or a different class are mixed to overlap each other to generate augmented test data, and the augmented test data is generated by a neural network. A user terminal characterized in that the generation is generated by generating prediction data by inputting to a model and estimating uncertainty for the prediction data. According to claim 18,
The uncertainty estimation result information,
A user terminal characterized in that it includes a similarity of another class to a specific class by uncertainty learning, an accuracy of uncertainty estimation, and an uncertainty distribution map of test data. According to claim 18,
The user interface,
In response to user input, displaying the uncertainty estimation result information, and
The uncertainty estimation result information,
The test data and the training data are obtained, the test data and at least one training data belonging to the same class as the test data or a different class are mixed to overlap each other to generate augmented test data, and the augmented test data is generated by a neural network. A user terminal characterized in that the generation is generated by generating prediction data by inputting to a model and estimating uncertainty for the prediction data.

Images