KR20190136577A - Method for classifying images using deep neural network and apparatus using the same - Google Patents

Abstract

The present invention relates to a method for classifying images using a deep neural network and an apparatus using the same. More specifically, by the method according to the present invention, a computing device obtains images, generates classification information of the images based on a deep neural network, and provides the classification information to an external entity. The classification information includes the classification to which the images belong and an active map supporting the classification.

Description

Method for classifying images using deep neural network and apparatus using same {METHOD FOR CLASSIFYING IMAGES USING DEEP NEURAL NETWORK AND APPARATUS USING THE SAME}

The present invention relates to a method for classifying an image using a deep neural network and an apparatus using the same. Specifically, according to the method according to the present invention, the computing device obtains the image, generates classification information of the image based on the deep neural network, and provides the classification information to an external entity. The classification information includes a classification to which the image belongs and an active map supporting the classification.

Recently, multiplication neural networks have left many successful cases in classifying images. For example, it is possible to accurately classify images photographed for special purposes such as medical images, especially fundus images. However, such a special purpose photographed image is merely to classify the class to which the photographed image belongs, for example, to accurately diagnose a specific disease, and to provide appropriate visualization and localization to support the validity of the classification. Specifying the relevant area in the image) is not provided. In the present disclosure, it is intended to provide an architecture of a convolutional neural network capable of classifying images in detail and simultaneously localizing the basis of the classification in the images. According to the inventors, the neural network architecture can be trained with regional annotations that allow for better localization and classification in some classes of classifications.

For example, fundus images provide a wealth of visual cues about the state of the eye. In the analysis, the ophthalmologist looks for abnormal visual features called findings from the image and classifies the image based on the found findings, that is, the content of the classification is the diagnostic content.

Nowadays, convolutional neural networks (CNNs) have reached the level of occupational ophthalmologists in diagnosing diabetic retinopathy (DR) and diabetic macular edema (DME). However, the CNNs in the literature have been trained to derive the diagnosis directly, which is different from what an ophthalmologist actually diagnoses. The ophthalmologist looks for visual cues first and then makes a diagnosis based on those visual cues. By the way, there have been several studies for visualizing the findings contributing to the determination of medical images, but the individual findings were not distinguished from each other.

Previously, segmentation methods using hand-crafted feature-extractors have been proposed for the detection of bleeding, hard exudates, drusen deposits, and cotton plaques. However, heuristic feature-extractors contain human designer biases that take into account the visual properties of target findings, so that unpredictable patterns are rarely detected, and their performance is severely limited in real-world applications. CNN for segmentation or detection may improve performance, but passive annotation of lesions is very labor intensive, especially when the lesions are dispersed in images. Therefore, the procedure of data collection becomes very expensive.

In this disclosure, we propose a CNN architecture that can classify an image through local annotation on visual cues from the image and localize the basis of the classification (eg, lesions in a medical image).

KR 10-1848321 B

The present invention aims to more accurately localize the basis for classification through training using guidance through regional cues, as well as to improve the performance of the classification itself.

To this end, the present invention aims to enable regional guidance so that the neural network can learn the correct patterns of classification determination instead of the bias in the image.

The characteristic constitution of the present invention for achieving the object of the present invention as described above and realizing the characteristic effects of the present invention described below is as follows.

According to an aspect of the present invention, there is provided a method for classifying an image using a deep neural network, the method comprising: (a) another device in which a computing device obtains the image or is linked to the computing device; Assisting to obtain; (b) supporting, by the computing device, generating or generating classification information of the image based on the deep neural network; And (c) supporting, by the computing device, providing or providing the classification information to an external entity, wherein the classification information includes a classification to which the image belongs and an activation map supporting the classification. ).

According to another aspect of the present invention, there is also provided a computer program, stored on a machine-readable non-transitory recording medium comprising instructions implemented to perform a method according to the invention.

According to another aspect of the present invention, there is provided a computing device for classifying an image using a deep neural network, the apparatus comprising: a communication unit for acquiring the image; And (i) a process for generating classification information of the image based on the deep neural network or for supporting other devices to be linked through the communication unit, and (ii) providing the classification information to an external entity. And a processor that performs a process of supporting the other device to provide the classification information, wherein the classification information includes a classification to which the image belongs and an activation map supporting the classification.

According to an embodiment of the present invention, there is an effect of improving the classification performance of the neural network by training using guidance through regional cues.

Such an effect according to an embodiment of the present invention can be applied not only to medical images such as fundus images but also to 2D or 3D images of various formats. Of course, it is not dependent on the video or platform.

BRIEF DESCRIPTION OF THE DRAWINGS The following drawings, which are attached to be used in the description of the embodiments of the present invention, are merely some of the embodiments of the present invention, and to those skilled in the art to which the present invention pertains based on these drawings without efforts to reach novel invention Other drawings can be obtained.
1 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device that performs a method of classifying images (hereinafter, referred to as an "image classification method") according to the present invention.
2 is an exemplary block diagram illustrating hardware or software components of a computing device performing an image classification method in accordance with the present invention.
3 is a flowchart illustrating an image classification method according to an exemplary embodiment of the present invention.
4 is a diagram illustrating an example of a neural network architecture according to an embodiment of the present invention.
5 is a table showing the results of comparing the performance of the neural network architecture provided according to an embodiment of the present invention with the conventional architecture.
6 is an exemplary diagram qualitatively comparing the performance of the neural network architecture provided according to an embodiment of the present invention with the conventional architecture.

DETAILED DESCRIPTION The following detailed description of the invention refers to the accompanying drawings that show, by way of illustration, specific embodiments in which the invention may be practiced to clarify the objects, technical solutions and advantages of the invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.

The term "image" or "image data" as used throughout the description and claims of the present invention refers to multidimensional data composed of discrete image elements (e.g., pixels in a two-dimensional image), in other words, ( For example, the term refers to a visually visible object (eg, a file corresponding to a pixel output of a CT, MRI detector, etc.) or a digital representation of that object (as shown on a video screen).

For example, an "image" may be a subject collected by computed tomography (CT), magnetic resonance imaging (MRI), ultrasound or any other medical imaging system known in the art. It may be a medical image of a subject. Imaging is not necessarily provided in a medical context, but may be in a non-medical context.

In the drawings presented for convenience of description, some exemplary image formats are shown, but a person skilled in the art will understand that the image formats used in various embodiments of the present disclosure are not limited to a specific format. .

Throughout the description and claims of the present invention, the DICOM (Digital Imaging and Communications in Medicine) standard is a term used to collectively refer to various standards used for digital image representation and communication in medical devices. The standard is published by a coalition committee composed of the American Academy of Radiology (ACR) and the American Electrical Industry Association (NEMA).

In addition, throughout the description and claims of the present invention, 'PACS (Picture Archiving and Communication System)' is a term for a system for storing, processing, and transmitting in accordance with the DICOM standard. The medical image obtained using a digital medical imaging device such as MRI may be stored in DICOM format and transmitted to a terminal inside or outside a hospital through a network, and reading results and medical records may be added thereto.

And in the description and claims of the invention the word 'localization' and variations thereof are intended to refer to the identification, specification or specification of a location. For example, 'localize an area' refers to specifying the location of that area.

And throughout the description and claims of this invention, the word 'comprises' and variations thereof are not intended to exclude other technical features, additives, components or steps. In addition, "one" or "one" is used in more than one sense, and "other" is limited to at least a second or more.

Other objects, advantages and features of the present invention will become apparent to those skilled in the art, in part from this description and in part from the practice of the invention. The following examples and drawings are provided by way of illustration and are not intended to limit the invention. Accordingly, the details disclosed in this disclosure with respect to particular structures or functions are not to be interpreted in a limiting sense, but merely to provide a guide for a person skilled in the art to variously implement the present invention with any suitable structures. It should be interpreted as representative basic data.

Moreover, the present invention encompasses all possible combinations of the embodiments indicated in this disclosure. It is to be understood that the various embodiments of the invention are different, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention with respect to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled. Like reference numerals in the drawings refer to the same or similar functions throughout the several aspects.

Unless otherwise indicated in the present disclosure or clearly contradicted by context, the item referred to in the singular encompasses the plural unless the context otherwise requires. In addition, in describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

Area annotations containing classifications for macular-centered images, which are images used in the verification of the method according to the present invention described hereinafter, are classified by the annotator through a predetermined interface (ie, type of findings). ) And the corresponding regions. If the eye shown in the image is normal, no comments are made on the image, so it is classified as normal. For example, as disclosed in Korean Patent No. 10-1848321, an image may be divided into eight regions, each of which reflects the regional characteristics of the anatomy and features of the eye according to the patent document. . For example, such areas may include macular area, optic nerve papillary area, optic nerve papillary area, lateral (temporal) area, superotemporal area, inferotemporal area, superonasal area and hyponasal area ( inferonasal) region.

1 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device for performing an image classification method according to the present invention.

Referring to FIG. 1, the computing device 100 according to an embodiment of the present invention includes a communication unit 110 and a processor 120, and directly or indirectly with an external computing device (not shown) through the communication unit 110. Can communicate with each other.

Specifically, the computing device 100 may comprise typical computer hardware (eg, a computer processor, memory, storage, input and output devices, devices that may include components of other conventional computing devices; routers, switches, etc.). Electronic communication devices; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SANs) and computer software (ie, enabling computing devices to function in a particular manner). Combination of instructions) to achieve the desired system performance.

The communication unit 110 of such a computing device may transmit and receive a request and a response with another computing device to which the computing device is interworked. As an example, the request and response may be made by the same transmission control protocol (TCP) session. However, the present invention is not limited thereto, and may be transmitted and received as, for example, a user datagram protocol (UDP) datagram. In addition, in a broad sense, the communication unit 110 may include a keyboard, a mouse, other external input devices, a printer, a display, and other external output devices for receiving commands or instructions.

In addition, the processor 120 of the computing device may include a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU) or a tensor processing unit (TPU), and a cache memory. ), And a hardware configuration such as a data bus. In addition, the operating system may further include a software configuration of an application performing a specific purpose.

2 is an exemplary block diagram illustrating hardware or software components of a computing device performing an image classification method in accordance with the present invention.

2, a brief overview of the configuration of a method and apparatus according to the present invention, the computing device 100 may include an image acquisition module 210 as a component thereof. The image acquisition module 210 is configured to acquire an image to which the method according to the present invention is applied. The image reading module 220 is used to read an image acquired by the image obtaining module 210.

The individual modules illustrated in FIG. 2 may be implemented by, for example, the communication unit 110 or the processor 120 included in the computing device 100, or by interworking with the communication unit 110 and the processor 120. The technician will understand.

For example, the image may be obtained from an external image storage system such as a photographing device or a medical image storage transmission system (PACS) interlocked through the communication unit 110, but is not limited thereto. For example, the image may be obtained by the image acquisition module 210 of the computing device 100 after being photographed by the (medical) imaging apparatus and transmitted to the PACS according to the DICOM standard.

In order to learn about the model of the image reading module 220, the image and the labeling information for the image may be obtained together with the image. In the learning, the image and the labeling information may be read. It is used to train the model in advance. The labeling information may include information about the contents of the classification displayed on the image and an area supporting the classification.

The image reading model in which the training is completed may generate the classification information of the image in the image reading module 220 and transfer the information to the result storing and output module 230. The result storing and output module 230 may have a predetermined value. The classification result of the image may be provided to an external entity through an output device such as a user interface displayed on a display. For convenience of the user, a result editing module 240 capable of editing the previously generated classification information may be further provided.

Herein, an external entity is to be understood as including a user, a manager of the computing device 100, and any subject that requires classification (ie, reading) of the image, that is, classification.

Specific functions and effects of the respective components shown in FIG. 2 will be described later in detail. Although the components illustrated in FIG. 2 are illustrated as being realized in one computing device for convenience of description, the computing device 100 performing the method of the present invention may be configured such that a plurality of devices are interworked with each other. It will be understood that it may be covered by the claims appended hereto.

An embodiment of an image classification method according to the present invention will now be described in more detail with reference to FIG. 3.

3 is a flowchart illustrating an image classification method according to an exemplary embodiment of the present invention.

Referring to FIG. 3, in the image classification method according to the present invention, first, an image acquisition module 210 implemented by the computing device 100 acquires an image or receives the image through the communication unit 110 of the computing device 100. And supporting another device (not shown) to be acquired (S100).

Next, in the image classification method according to the present invention, the image reading module 220 implemented by the computing device 100 supports generating or generating classification information of the image based on a deep neural network (S200). Further comprising, the deep neural network is as exemplified below. The classification information may include classification of an image and visualization information supporting the classification. This visualization information may be provided as an activation map to be described later. The classification of the images may be singular or plural.

In one embodiment, step (S200), specifically, the image reading module 220, the step of supporting the other device to determine whether or not the image is readable or through the communication unit 110 (S220); Not shown).

In order to perform the method according to the present invention, a predetermined user interface may be provided, and an image and whether or not the image may be read may be displayed through the user interface. Whether or not this can be read may be included in the classification information.

In step S220, if the image is not readable, a classification may be generated that is not readable.

In operation S200, when the image is readable, the image reading module 220 displays the read result information on the individual classification detected in the image together with the confidence level of the individual classification. The method may further include generating as information or supporting the other device to generate (S240).

In step S240, the visualization information in the image supporting the individual classification may be calculated as an activation map whose calculation scheme will be described below in detail below, for example, FIG. 6 described below. As shown in

Next, in the image classification method according to the present invention, the result storage and output module 230 implemented by the computing device 100 may provide the classification information to an external entity or support the other device to provide. It may further include (S300). As described above, the classification information may include a classification to which the image belongs and visualization information supporting the classification, for example, an activation map.

However, since the generated classification information may need to be modified or edited, in the image classification method according to the present invention, after the step S300, the result editing module 240 implemented by the computing device 100 may be predetermined. Providing a user interface may further include a step (S400; not shown) for supporting a user to edit or modify the classification information through the user interface. In operation S400, when a correction input is obtained, the result editing module 240 may modify the classification information or support the other device to correct the classification information in response to the correction input.

As an example of a user interface used in the present disclosure, a graphical user interface (GUI) includes one or more display images generated by a display processor, which includes a processor Or user interaction with other devices and associated data acquisition and processing functions. The GUI also includes an executable procedure or executable application. The executable procedure or executable application causes the display processor to generate signals representing the GUI display images according to a condition. These signals are transmitted to a display device displaying images for viewing by the user. The processor manipulates the GUI display images in response to signals received from input devices under the control of an executable procedure or an executable application. In this manner, a user may understand that the user may interact with the display image using input devices, thereby enabling user interaction with the processor or other device.

The user referred to in the present invention may not only refer to the singular but also may refer to a plurality of users for the purpose of obtaining abundant and overlapping medical images and their related data, which includes learning a deep neural network for classifying images. Alternatively, there may be a purpose to secure the integrity of the image in actual use.

Various kinds of classifiers may be used as the reading model of the image reading module 220 used in the method according to the present invention, for example, a deep learning model, a random forest, a Bayesian image acquisition module, or the like. The following describes an exemplary neural network architecture used by the inventors as a readout model with reference to FIG. 4.

Exemplary neural network architecture of the present invention. 4 is a diagram illustrating an example of a neural network architecture according to an embodiment of the present invention.

As shown in FIG. 4, the neural network architecture according to an embodiment of the present invention includes a residual layer (412, 422, 432, 440, 458, 470) (characteristic maps after a residual unit), and an intermediate reduction layer (intermediate). reduction layer; 414, 416, 424, 426, 428, 433, 434, 435, 436) {3x3 product of stride 2, batch-norm, feature maps after ReLU}, average pooling layer 452, 454, 456), an atrous pyramid pooling layer 460, a final reduction layer 470, and a 1 × 1 composite product (depth = 1; 480) layer. Here, the residual layer or the residual unit refers to feature maps in which an input layer is added to an output layer in {synthesis product, batch-norm, ReLU} x 2.

As the layers become deeper, it is advantageous to monotonically increase the number of layers forming each of the convolution blocks 410, 420, and 430. If the number is the same or less, the performance tends to be lowered when they have the same parameters. Meanwhile, in the example of FIG. 4, the total number of convolutional blocks is illustrated as three, but more than that.

Half the height and width double the depth of the layers. The first four or more reduction layers of different sizes are concatenated with average pooling to utilize both low and high level features. That is, according to the neural network architecture of the embodiment, the feature is extracted through a plurality of composite products, and then the information is compressed by adjusting the size, that is, by adjusting the size equally (e.g., the average pooling operation) and matching the resolution in the same direction. By concatenation, various levels of features can be extracted. Referring to the example of FIG. 4, the reduction layers of reference numerals 412, 422, and 432 are combined with reference numerals 452, 454, and 456, respectively, by adjusting the resolution, and the reduction layers of reference numeral 440 are coupled thereto.

The combined feature maps then have a dilation rate of 1, 2, 4, 8 (findings for large size), an expansion rate of 1, 2, 4 (findings for medium size), or 1 , Atrous-pyramid-pooled (460) with an expansion rate of 2 (findings for small size). This is for effectively doubling the receptive fields. That is, atlas pyramid pooling has been employed to combine features into accommodation fields of various sizes. The next layer of the atlas pulling layer is the composite product of stride 2, reducing the resolution to 1/2 to characterize it.

The last layer 480 is a 1x1 composite product layer, which computes the 1x1 composite product of the last final reduction layer 470, which is a linear combination of previous feature maps, such as in a class activation map (CAM). It is noted. With the sigmoid function the values of the layer are normalized to (0, 1), so that subsequent layers can be considered normalized active maps. The devised activity map 490a by the inventor differs from the CAM, in that an additional loss function guides the activity to appear only in the desired areas.

In our experiments, combining 16x16 sized features showed the most accurate activation map. Preferably the last layer may be between 8x8 and 32x32 in size so that the size in the actual fundus image corresponding to 1x1 matches the significant lesion identification size.

In addition, the 1 × 1 convolutional layer is global average pooled (GAP) on one side and normalized with a sigmoid function to yield a classification result, that is, a first output layer that yields a prediction that takes into account its presence. Leads to. The 1 × 1 convolutional layer is followed by a second output layer from which the normalized active map is calculated via sigmoid. Therefore, the activity map is directly related to the prediction in the neural network architecture according to the present invention and no external computation is required for visualization.

In the case of small lesions in medical images, there is a case where the activity is generated as a false positive (bias appears) at the same location on the activity map, which is specified as maximum pooling before global average-pooling (GAP) to prevent this. An operation can be performed to output a non-reduced function value with respect to the maximum value of the active values in the zone.

이 주어지면, 그 영상 I에서의 표적 소견의 존재는

로 인코딩되고, 존재 확률

이 신경망으로부터 출력된다. K개의 이미지가 미니 배치(mini-batch)로서 주어지는 때에, 도 4의 분류 손실에 대한 이진 교차 엔트로피는 아래 수학식 1과 같이 주어진다.An objective function according to an embodiment of the present invention is as follows. video

Given this, the presence of the target findings in that image I

Encoded with a probability of existence

It is output from this neural network. When K images are given as mini-batch, the binary cross entropy for classification loss in FIG. 4 is given by Equation 1 below.

이고

이다.From here,

ego

to be.

의 크기를 가지는 때에, 표적 소견

에 대한 영역 마스크(region mask; 490b)가 레이블로서 주어지고, 활성 맵(

)이 신경망으로부터 생성된다. 크기 k의 미니 매치를 가지고 도 4에서의 안내 손실(guidance loss)은 다음 수학식 2와 같이 주어진다.The last feature map

When we have size, target findings

The region mask 490b for is given as a label, and the active map (

) Is generated from neural networks. Guidance loss in FIG. 4 with a mini-match of size k is given by Equation 2 below.

이고,

이며,

과

는

에 대하여 Mⁱ 및 Aⁱ의 l 번째 픽셀에서의 값이다.

인 때에 수치적인 안정성(numerical stability)을 위하여 로그의 안쪽에

이 추가된다는 점이 주목된다. 간명하게 설명하면, 안내 손실은 마스크의 값이 0인 영역들에서의 어떠한 활성도 억제하며, 마스크 내부의 활성에 대하여는 어떠한 영향도 미치지 않는다. 수학식 2에서

대신에

를 이용하여도 무방하다.

또는

는

로 일반화될 수 있는데, 이

는 치역이 양수에 속하는

에 대한 비감소함수(non-decreasing function; weakly increasing function) 중에서 선택될 수 있다.From here

ego,

Is,

and

Is

For the l th pixel of M ⁱ and A ⁱ .

Inside the log for numerical stability

It is noted that this is added. In short, the guidance loss inhibits any activity in areas where the value of the mask is zero, and has no effect on the activity inside the mask. In equation (2)

Instead of

You can also use.

or

Is

Can be generalized to

The range is positive

It can be chosen from a non-decreasing function (weakly increasing function).

The loss of guidance reduces the active value when the value of the mask is zero. The backpropagation equation of the W value is (1-a) * out when the output value of the previous layer is called out, so that the smaller the value of a, the active value of each pixel, is larger. This has the effect of eliminating artifacts on the edges of the classification loss as it searches for patterns.

Then, the total loss value is obtained by combining a classification loss using a loss function such as binary classification entropy and a guide loss multiplied by lambda, a hyperparameter for compromise with this classification loss, as shown in Equation 3 below.

In other words, [lambda] is a value for balancing two objective functions.

Training example of an exemplary neural network architecture of the present invention. The training method of the deep neural network according to the present invention is a technical feature that is different from the conventional technology using the above-described guide loss. In general, in the method for training a deep neural network according to the present invention, first, the computing device 100 extracts at least a portion of training image data as a mini-batch, and then extracts the extracted mini batch. The loss function value for the deep neural network is calculated as in Equation 3. When the loss function value is obtained, the computing device 100 calculates the slope of the direction in which the loss function value decreases, and updates the weight parameter of the deep neural network in the direction of the slope. This may be repeated until the predetermined end of training condition is met.

On the other hand, it is known that the updating of the weight parameter can be achieved by applying a predetermined learning rate.

Using this trained deep neural network, a computing device may classify an input image and localize an area of the input image corresponding to the classification in accordance with the method of the present invention.

The deep neural network that can be trained using intraocular loss is not limited to that illustrated in FIG. 4, and is defined between (i) the first input layer, (ii) a number of composite product layers following the input layer, and (iii) the composite product layer. It may be widely used in a deep neural network comprising at least one intervening pooling layer and (iv) the last output layer.

Implementation of an exemplary neural network architecture of the present invention for verification of performance. We have clinically associated DR and DME (bleeding, hard exudates, drusen, cotton plaques (CWP)), macular holes, membranes and retinal nerve fiber layer defects (RNFL defects). The results of the classification on the important findings were chosen as optional.

We used an annotated set of images based on the expertise of annotators for the purpose of measuring the performance of the neural network architecture. Among them, the total amount of images included in the training set and the test set was 66,473 and 15,451, respectively.

The training set was divided into 90% of the derivation set and 10% of the verification set. The model of an exemplary neural network architecture in accordance with the present invention has been optimized with this set of derivations until the verification loss is stagnated and exacerbated. The model with the lowest verification loss was tested with the test set considered gold standards. The present inventors determined that there was no target finding when all of the ophthalmologists did not annotate, and that the target finding was present when two or more of three ophthalmologists annotated. The union of annotated areas was provided as regional clues during training.

We have experimented with exemplary CNN architectures of the present invention with or without regional guidance, and have shown that the area under receiver operating characteristic curve (AU-ROC), specificity, sensitivity and activity in regional cues (AIR) The aim was to measure the effect of the guidance loss by comparing the results in terms of. AIR is defined as the sum of the activities within the regional clues divided by the sum of all the activities. AIR was measured for both true and false negatives in the classification when regional cues were available. The inventors have implemented a neural network with or without regional guidance by changing the value of λ in Equation 3 using the neural network architecture illustrated in FIG. 4 (there is no regional guidance when λ = 0).

The original color fundus images, which were used in the experiments, were cropped to center the fundus area, removing black background, and resized to 512x512 for neural network input. Preferably it may be readjusted to a size between 256x256 and 1024x1024. The pixel values of the pixels that make up the image are divided by 255 so that they are in the [0,1] range. No other preprocessing is required. When learning in a situation where enough data is given, it may be meaningful to control only a range of pixel values without any preprocessing in the case of red-green-blue (RGB).

The readjusted images were randomly data-augmented by affine transformation (flip, resize, rotate, translate, shear), and the intensity was randomly scaled (re -scaled). Weights and biases were initialized using Xavier initialization. We used SGD as an optimizer, with nestrov momentum 0.9 and a decaying learning rate. The batch size was set at 32 according to the recommendation that smaller batch sizes are better for generalization. In addition, numerical stability was achieved by setting ε = 10 ⁻³ in Equation 2, and λ = 1 in Equation 3 to treat classification loss and guide loss in the same manner.

On the other hand, for an image having a size of 512x512 or more, it is desirable to reduce the resolution by half through a composite product of successive strides 2, which is for efficient calculation.

However, those skilled in the art will understand that the present invention is not limited to the specific numerical values shown in the exemplary neural network.

Experimental results by implementation of exemplary neural network architecture of the present invention. 5 is a table showing the results of comparing the performance of the neural network architecture provided according to an embodiment of the present invention with the conventional architecture.

FIG. 5 summarizes the results of comparing the performance between the model with guide loss introduced and the model without guide loss. The inventors were able to identify the positive effects of guiding loss on AIR of TP (true positive) and FN (false negative) across all categories of findings. This is desirable, as the neural network can pay attention to the regional cues for classification, so the neural network tends to learn less of the dataset biases. Also, the difference in AIR between the two models is larger in the case of TP than in FN. This is reasonable because FN consists of cases where neural network is difficult to classify, whereas TP can be classified relatively easily with high confidence.

For AU-ROC, significant improvements were made in macular holes and retinal nerve fiber layer defects. Since these findings are observed in certain areas, it is interesting to note that the greatest advantage is given by the regional cues. This can be explained by the fact that the neural network is guided to pay attention to the areas important for classification, making learning easier. On the other hand, findings distributed over a wide area such as bleeding, hard exudates and drusen may have little benefit of regional cues in classification. The inventors assume that the classification of these findings has broad regional cues and that the guidance according to the present invention is somewhat redundant, but that the smaller the lesion, the more important the guidance will be. At higher AU-ROCs, higher sensitivity and lower specificity are observed. However, there is a significant difference in macular hole and retinal nerve fiber layer defect.

6 is an exemplary diagram qualitatively comparing the performance of the neural network architecture provided according to an embodiment of the present invention with the conventional architecture.

6, the activity maps of the neural network with and without guide loss are compared with each other. Before superimposing on the original image, the active maps are upscaled by double linear interpolation, blurred with a 32x32 Gaussian filter for natural visualization, and normalized to [0, 1]. As clearly shown in the figure, the neural network produces a more accurate activity map when trained with regional cues. The activity maps derived by the present invention provide significant information about the location of the findings that may be beneficial to the clinician, even if they are not markedly divided by pixel and in some cases a small number of false positives are present. Without loss of guidance, the active maps spread far larger than the periphery of the lesion and often highlight unrelated areas.

As described above, the present invention utilizes regional information on classification in images for localization and classification, in all embodiments and modifications thereof. Efficient labeling to collect regional annotations on classification has been made possible, so that a neural network architecture that can be classified with localization of the area supporting the classification (ie lesions in fundus images) could be proposed, thereby allowing the user to The performance of classifying images is improved.

Based on the description of the above embodiments, those skilled in the art can realize that the methods and / or processes of the present invention and the steps can be realized in hardware, software or any combination of hardware and software suitable for a particular application. The point can be clearly understood.

The functions and process steps described above may be performed automatically or in response to all or some user commands. Activity (including steps) performed automatically is performed in response to one or more executable instructions or device operation without direct initiation by the user of the action. .

The hardware may include a general purpose computer and / or a dedicated computing device or a particular aspect or component of a particular computing device or a specific computing device. The processes may be realized by one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices having internal and / or external memory. In addition, or alternatively, the processes may be configured to process an application specific integrated circuit (ASIC), a programmable gate array, a programmable array logic (PAL), or electronic signals. It can be implemented in any other device or combination of devices that are present. Furthermore, the objects of the technical solutions of the present invention or the parts contributing to the prior art can be implemented in the form of program instructions that can be executed by various computer components and recorded in a machine-readable recording medium. The machine-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the machine-readable recording medium may be those specially designed and configured for the present invention, or may be known and available to those skilled in the computer software arts. Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, Blu-rays, and magnetic-optical media such as floptical disks. (magneto-optical media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include, but are not limited to storing and compiling or interpreting for execution on any one of the foregoing devices, as well as on a machine capable of executing a heterogeneous combination of a processor, processor architecture or combinations of different hardware and software, or any other program instructions. Can be made using a structural programming language such as C, an object-oriented programming language such as C ++, or a high or low level programming language (assembly, hardware description languages and database programming languages and technologies), including machine code, This includes not only bytecode, but also high-level language code that can be executed by a computer using an interpreter.

Thus, in one aspect according to the present invention, when the methods and combinations described above are performed by one or more computing devices, the methods and combinations of methods can be implemented as executable code that performs each step. In another aspect, the method may be practiced as systems that perform the steps, and the methods may be distributed in various ways across the devices or all the functions may be integrated into one dedicated, standalone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any hardware and / or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.

For example, the hardware device may be configured to operate as one or more software modules to perform a process according to the present invention, and vice versa. The hardware device may comprise a processor, such as an MPU, CPU, GPU, TPU, coupled with a memory, such as a ROM / RAM for storing program instructions, and configured to execute the instructions stored in the memory, the external device and the signal It may include a communication unit that can send and receive. In addition, the hardware device may include a keyboard, a mouse, and other external input devices for receiving instructions written by developers.

Although the present invention has been described in terms of specific embodiments such as specific components and the like, and limited embodiments and drawings, the systems and processes of the accompanying drawings are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the present invention to achieve the same purpose. Although the present invention has been described with reference to specific embodiments, it will be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the design of the present disclosure may be implemented by those skilled in the art without departing from the scope of protection of the present invention. Various systems, ancillary systems, agents, managers, and processes as described in this disclosure can be implemented using hardware components, software components, and / or combinations thereof. .

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the appended claims, fall within the scope of the spirit of the present invention. I will say.

Such equivalent or equivalent modifications will include, for example, logically equivalent methods capable of producing the same results as those of the method according to the invention, the spirit and scope of the invention being Is not to be limited by the foregoing examples, but is to be understood in the broadest sense acceptable by law.

Claims (13)

In the method for classifying images using a deep neural network,
The method,
(a) allowing a computing device to acquire the image or to allow another device to interoperate with the computing device;
(b) supporting, by the computing device, generating or generating classification information of the image based on the deep neural network; And
(c) supporting, by the computing device, providing or providing the classification information to an external entity.
Including,
And the classification information includes a classification to which the image belongs and an activation map supporting the classification. The method of claim 1,
The deep neural network,
A convolution block comprising at least three convolution blocks, the convolution block comprising at least one reduction layer and at least one residual layer subsequent to the reduction layer;
An intermediate reduction layer subsequent to the convolutional block;
A concatenation layer that concatenates layers having the same size and the intermediate reduction layer in a depth direction calculated from a reduction layer of each convolution block;
An atlas pyramid pooling layer subsequent to the bonding layer;
A final reduction layer following the atlas pyramid pulling layer;
A 1 × 1 convolutional layer followed by the final reduction layer; And
An output layer including at least two output layers subsequent to the 1x1 composite product layer, the first output layer outputting guide loss and the second output layer outputting classification loss;
Including,
And the reduction layer of the first product block of the at least three product blocks is an input layer into which the image is input. The method of claim 2,
In step (b),
(b1) Before the image is input to the input layer, the computing device performs (i) cropping and resizing the image so that a fundus region is located at the center of the image. Positioning and resizing the image between 256x256 and 1024x1024; And (ii) normalizing the pixel value such that a pixel value of each channel of the image falls within a range of 0 or more.
Video classification method further comprising. The method of claim 3,
If the size of the image exceeds 512x512, the resolution of the image is reduced in the direction of a size of 256x256 through a continuous product of stride 2 in the process (i). The method of claim 2,
And the number of layers belonging to each of the at least three convolutional block blocks is monotonically increased with subsequent layers. The method of claim 2,
The dilation rate of the atlas pyramid pooling layer is one selected from the group comprising (i) 1, 2, (ii) 1, 2, 4 and (iii) 1, 2, 4, 8. Image classification method. The method of claim 2,
By performing a composite product of stride 2 between the atlas pyramid pooling layer and the final reduction layer, features are extracted from the atlas pyramid pooling layer and resolution of data passing through the atlas pyramid pooling layer is 1 /. The image classification method characterized by decreasing to 2. The method of claim 2,
The size of the 1x1 composite product layer is characterized in that the 8x8 to 32x32. The method of claim 2,
The first output layer calculates the classification by applying a sigmoid function after a global average pooling (GAP) of the 1x1 composite product layer,
And the second output layer calculates the activity map supporting the classification by applying the sigmoid function of the 1 × 1 composite product layer. A computer program stored in a machine-readable non-transitory recording medium comprising instructions implemented to cause a computing device to perform the method of any one of claims 1-9. A computing device for classifying images using a deep neural network,
A communication unit for acquiring the image; And
(i) a process of generating classification information of the image based on the deep neural network or supporting other devices which are interworked through the communication unit; And (ii) a process for providing or assisting to provide the classification information to an external entity.
Including,
And the classification information includes a classification to which the image belongs and an activation map for supporting the classification. The method of claim 11,
The deep neural network,
A convolution block comprising at least three convolution blocks, the convolution block comprising at least one reduction layer and at least one residual layer subsequent to the reduction layer;
An intermediate reduction layer subsequent to the convolutional block;
A concatenation layer that concatenates layers having the same size and the intermediate reduction layer in a depth direction calculated from a reduction layer of each convolution block;
An atlas pyramid pooling layer subsequent to the bonding layer;
A final reduction layer following the atlas pyramid pulling layer;
A 1 × 1 convolutional layer followed by the final reduction layer; And
An output layer including at least two output layers subsequent to the 1x1 composite product layer, the first output layer outputting guide loss and the second output layer outputting classification loss;
Including,
And the reduction layer of the first product block of the at least three product blocks is an input layer to which the image is input. The method of claim 12,
The first output layer calculates the classification by applying a sigmoid function after a global average pooling (GAP) of the 1x1 composite product layer,
And the second output layer calculates the activity map supporting the classification by applying the sigmoid function of the 1 × 1 composite product layer.

Images