WO2024014702A1

WO2024014702A1 - Otitis media diagnosis method and device

Info

Publication number: WO2024014702A1
Application number: PCT/KR2023/007254
Authority: WO
Inventors: 권지훈; 안중호; 채지혜; 박근우; 최연주
Original assignee: 재단법인 아산사회복지재단; 울산대학교 산학협력단
Priority date: 2022-07-13
Filing date: 2023-05-26
Publication date: 2024-01-18

Abstract

An electronic device for diagnosing otitis media, according to one embodiment, comprises: a memory for storing computer-executable instructions, and a trained otitis media diagnosis model including a shared layer, which includes at least one convolution operation, and a plurality of classifier layers, which are connected to the shared layer; a processor accessing the memory so as to execute the instructions; a display electrically connected to the processor; and an image acquisition unit for receiving an otoendoscopic image of a patient, wherein the instructions can receive the otoendoscopic image of the patient, generate an input otoendoscopic image on the basis of a region of interest extracted from the received otoendoscopic image, extract feature data from the input otoendoscopic image on the basis of the shared layer, output, on the basis of a first classifier layer from among the plurality of classifier layers, disease prediction results for diseases belonging to a primary class from the extracted feature data, and individually output, on the basis of a plurality of second classifier layers separated from the first classifier layer from among the plurality of classifier layers, a single disease prediction result for each disease of diseases belonging to a secondary class, from the corresponding second classifier layer from among the plurality of second classifier layers.

Description

Otitis media diagnostic methods and devices

Hereinafter, technology related to a method and device for diagnosing otitis media is provided.

Acute otitis media is a disease that occurs so commonly that 80% of children before the age of 3 experience it, has frequent recurrences, and often requires a lot of antibiotics. Otitis media with effusion is a disease that occurs when exudate accumulates within the eardrum due to sequelae of acute otitis media or poor function of the middle ear ventilation duct, and is known to be the most common cause of hearing loss in children. It is known that 80% of children suffer from otitis media with effusion at least once before the age of 10.

To diagnose otitis media in hospitals, an endoscope is generally used to obtain images by approaching the eardrum through the external auditory canal. It is used in various hospitals, such as pediatrics or family medicine, and is often equipped in private hospitals, etc. Recently, endoscopes in the form of portable devices connected to personal communication devices (handsets or tablets) have been developed, increasing opportunities to obtain images of the eardrum.

However, otitis media has many different types of disease, so there are many cases where it is difficult for even experienced specialists to make an accurate diagnosis. Recently, with the development of deep learning technology, the technology for classifying major diseases has shown high performance, but it cannot support diagnosis for diseases that are not considered the target of learning. Therefore, there is an urgent need to develop a method that can effectively provide information related to the occurrence or abnormality of a disease.

Additionally, in order to accurately diagnose middle ear diseases, a method is needed to classify middle ear diseases into diseases that can co-exist and diseases that cannot co-exist.

The background technology described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known technology disclosed to the general public before this application.

In the otitis media diagnosis electronic device according to an embodiment, a learned otitis media diagnosis model including a shared layer including at least one convolution operation and a plurality of classifier layers connected to the shared layer, and computer-executable instructions Memory that stores (computer-executable instructions); a processor that accesses the memory and executes the instructions; a display electrically connected to the processor; and an image acquisition unit for receiving an otoendoscopic image of a patient, wherein the commands are configured to receive an otoendoscopic image of the patient and select a region of interest extracted from the received otoendoscopic image. Based on this, generate an input otoscope image, extract feature data from the input otoscope image based on the shared layer, and based on a first classifier layer among the plurality of classifier layers, the extracted feature data Outputs disease prediction results for diseases belonging to the primary class, and based on a plurality of second classifier layers separated from the first classifier layer among the plurality of classifier layers, secondary A single disease prediction result for each disease belonging to a class may be individually output from the corresponding second classifier layer among the plurality of second classifier layers.

The processor may receive a video sequence of the patient's tympanic membrane and acquire a plurality of otoscope images corresponding to the number of frames determined by the user of the video sequence. .

The processor removes the patient's personally identifiable information from the otoscope image, extracts the region of interest having a predetermined shape from the otoscope image, and selects the extracted region of interest, 2. It can be placed in the center of the otoscope image in a dimensional form.

The processor obtains probabilities for primary diseases belonging to the primary class from the extracted feature data, based on the first classifier layer, and has the highest probability among the probabilities for the primary diseases. The disease may be output to the user as the disease prediction result.

The processor obtains a probability regarding a secondary disease belonging to the secondary class from the extracted feature data based on each of the second classifier layers, and obtains a probability regarding the secondary disease for each of the second classifier layers. The disease occurrence result based on can be output to the user as a single disease prediction result for each of the second classifier layers.

The processor sets the probability of a disease with the highest probability among diseases excluding the disease corresponding to the disease prediction result in the primary class as the target result, and sets the probability of the disease prediction result and the probability of the target result. Based on a case where the difference between the two is less than a threshold determined by the user, an output suggesting retry of otitis media diagnosis through an otoscope image different from the otoscope image may be provided to the user.

The processor selects second classifier layers related to the secondary class disease in which the disease occurred among the disease occurrence results related to the secondary disease, and determines that at least one of the probabilities related to the secondary disease for the selected layers is determined by the user. Based on the case being less than the threshold determined by , an output suggesting retry of otitis media diagnosis through an otoscope image different from the otoscope image may be provided to the user.

The processor may extract feature data based on skipping at least some of the connections between nodes of the shared layers of the learned otitis media diagnosis model.

The processor extracts first feature data based on skipping the selected first connection among the connections between the nodes of the shared layers, and the first connection and the first connection among the connections between the nodes of the shared layers. Extracting second feature data based on skipping other second connections, and repeating changes in connections skipped between the nodes, a plurality of feature data including the first feature data and the second feature data can be extracted.

The processor obtains probabilities for diseases belonging to the primary class for each of the plurality of feature data, based on the first classifier layer, and, for each of the plurality of feature data, the first classifier. The probabilities for diseases obtained from the classifier layer are converted into disease binary results based on a predetermined threshold, and for each disease belonging to the primary class, the average of the plurality of disease binary results is calculated. A first statistical result representing may be obtained, and the highest first statistical result among diseases belonging to the primary class may be output to the user as the disease prediction result.

The processor applies the plurality of feature data to each of the second classifier layers to obtain probabilities for diseases belonging to the secondary class for each of the plurality of feature data, and the second classifier For each of the layers, the probability corresponding to each of the plurality of feature data is converted into a single disease binary result based on a predetermined threshold, and for each of the second classifier layers, the probability corresponding to each of the plurality of feature data is converted into a single disease binary result. Obtain a second statistical result representing an average for disease binary outcomes, and obtain a second statistical result for each of the second classifier layers, individually in a corresponding second classifier layer among the plurality of second classifier layers. It can be output as the single disease prediction result.

1 is a diagram illustrating an electronic device for diagnosing otitis media according to an embodiment.

Figure 2 is a flow chart illustrating a method for diagnosing otitis media according to an embodiment.

FIG. 3 is a diagram illustrating a disease belonging to a primary class and a disease belonging to a secondary class according to an embodiment.

Figure 4 is a diagram illustrating an otitis media diagnosis model for diagnosing otitis media from an otoscope image according to an embodiment.

Figure 5 is a diagram illustrating prediction results obtained from an otitis media diagnosis model according to an embodiment.

Figures 6A to 6C are diagrams showing prediction results obtained from classifier layers of an otitis media diagnosis model according to an embodiment.

FIG. 7 is a diagram illustrating a method of obtaining a prediction result from a plurality of feature data according to an embodiment.

Figure 8 is a diagram showing McNemar test results of an otitis media diagnosis model according to an embodiment.

Figure 9 is a diagram showing a confusion matrix of an otitis media diagnosis model according to an embodiment.

Figure 10 is a diagram showing ROC curves for the primary class and secondary class according to one embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

The electronic device 100 according to an embodiment may apply an otoendoscopic image of the patient 170 to the otitis media diagnosis model 130 and output a plurality of disease prediction results for otitis media diseases. For example, the electronic device 100 may include a processor 110, a memory 120, an image acquisition unit 140, and a display 150.

The processor 110 may receive an otoscope image of the patient 170 from the otoscope device 180. The processor 110 may generate an input otoscope image based on a region of interest extracted from the received otoscope image. The otoscope image may be an image of the tympanic membrane of the patient 170 captured by the endoscope camera 160 of the otoscope device 180. The processor 110 may apply the input otoscope image to the otitis media diagnosis model 130 to obtain a plurality of disease prediction results for otitis media diseases. Processor 110 may execute software and control at least one other component (e.g., hardware or software component) connected to processor 110. The processor 110 may also perform various data processing or operations. For example, the processor 110 may store the otoscope image received from the otoscope device 180 by the image acquisition unit 140 in the memory 120 . The processor 110 may output a plurality of disease prediction results for an otoscope image through the display 150 as result data using an otitis media diagnosis method described later.

Memory 120 may temporarily and/or permanently store various data and/or information required to perform otitis media diagnosis. For example, the memory 120 may store at least one of an otoscope image, computer-executable instructions, or an otitis media diagnosis model 130.

The otitis media diagnosis model 130 may be a learned machine learning model that outputs prediction results regarding otitis media disease from images or images. A description of the otitis media diagnostic model will be provided later in Figure 4.

The image acquisition unit 140 may receive an otoscope image of the patient 170 from the otoscope device 180. In this specification, an example in which the otoscope image is an image corresponding to at least one frame in a video sequence in which the eardrum of the patient 170 is photographed will mainly be described. However, it is mainly explained that the electronic device 100 receives a single ear endoscope image, but it is not limited to this. For example, the electronic device 100 may receive a video sequence. The electronic device 100 may acquire a plurality of otoscope images from the received video sequence, corresponding to the number of frames determined by the user. The electronic device 100 may apply each of the plurality of acquired otoscope images to the otitis media diagnosis model 130.

The display 150 may visually provide a user (eg, a medical professional) with a plurality of disease prediction results for otitis media diseases of the patient 170. The display 150 may visually output at least one of an otoscope image, an input otoscope image, a disease prediction result, a single disease prediction result, or a notification suggesting retry of otitis media diagnosis. For example, the display 150 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch. However, it is not limited thereto, and the display 150 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device.

In step 210, an electronic device (e.g., electronic device 100 of FIG. 1) receives information from an otoscope device (e.g., otoscope device 180 of FIG. 1) to a patient (e.g., patient 170 of FIG. 1). Ear endoscopy images can be received. The otoscope device may include a CCD camera, a CMOS camera, etc. used in general endoscopes, but is not particularly limited thereto. An otoscope image is a medical image of a patient collected by a capsule endoscope image, ultrasound, or any other medical imaging system known in the art of the present invention, and may be an image converted into a form similar to an otoscope image.

In step 220, the electronic device may generate an input otoscope image based on the region of interest extracted from the received otoscope image. A method of generating an input otoscope will be described later in Figure 4.

In step 230, the electronic device may extract feature data based on the shared layer from the input otoscope image. The feature data may include abstracted values extracted by applying the input otoscope image to the shared layer of the otitis media diagnosis model. For example, a shared layer may include multiple convolutional layers. The convolutional layer may be used to extract a plurality of feature maps from input data (eg, an input otoscope image) using a plurality of convolutional filters. Here, a plurality of feature maps extracted from the shared layer may be feature data. As will be described later, the electronic device may apply feature data to a plurality of classifier layers and output a plurality of disease prediction results.

In step 240a, the electronic device may output disease prediction results for diseases belonging to the primary class based on the first classifier layer from the extracted feature data. For example, the first classifier layer may include layers that output prediction results for diseases belonging to the primary class from feature data extracted from the shared layer. A detailed description of the first classifier layer is described later in FIG. 4.

Diseases belonging to the primary class may include at least one of otitis media with effusion (OME), chronic otitis media (COM), congenital cholesteatoma, or absence of disease. there is. A description of the diseases belonging to the primary class will be provided later in FIG. 3.

The disease prediction result may be a disease with the highest probability among diseases belonging to the primary class. For example, the electronic device may obtain probabilities regarding diseases belonging to the primary class from extracted feature data based on the first classifier layer. The electronic device may obtain at least one of a probability for otitis media with effusion, a probability for chronic otitis media, or a probability for the absence of a disease based on the first classifier layer from the feature data. The electronic device may output to the user the disease with the highest probability among the probabilities of primary diseases as a disease prediction result. Specifically, the electronic device may output to the user the disease with the highest probability among the probability of otitis media with effusion, the probability of chronic otitis media, or the probability of the absence of the disease as a disease prediction result. However, it is not limited to this, and the electronic device may output to the user a set of probabilities including at least one of the probability of otitis media with effusion, the probability of chronic otitis media, or the probability of the absence of the disease as a disease prediction result. there is. Additionally, the disease prediction result is not limited to the probability of diseases belonging to the primary class, but may be a statistical result for each disease belonging to the primary class. A method of obtaining statistical results for diseases belonging to the primary class is described later in FIG. 7.

In step 240b, the electronic device may individually output a single disease prediction result for each disease of diseases belonging to the secondary class, based on the first classifier layer and the plurality of second classifier layers separated. You can. For example, the second classifier layers may include layers that individually output prediction results for each disease belonging to the secondary class from feature data extracted from the shared layer. Each of the plurality of second classifier layers may output a single disease prediction result for at least one disease among diseases belonging to the secondary class. A detailed description of the second classifier layers is described later in FIG. 4.

Diseases belonging to the secondary class may include at least one of the following: Attic Cholesteatoma, Myringitis, Otomycosis, Tympanosclerotic Plague, or Ventilating Tube. A description of diseases belonging to the secondary class will be provided later in FIG. 3.

The single disease prediction result may be the disease occurrence result for each of the diseases belonging to the secondary class. For example, a disease occurrence result may be a result indicating whether a disease has occurred. The electronic device may obtain a probability regarding a secondary disease belonging to the secondary class from feature data based on each of the plurality of second classifier layers. The electronic device may separately obtain the probability for epitympanic cholesteatoma, probability for tympanitis, probability for ear fungus, or probability for ventilator tract separately from a corresponding second one of the second classifier layers. . If the probability for each of the diseases belonging to the secondary class is greater than or equal to a threshold predetermined by the user, the electronic device may determine the disease occurrence result for each of the diseases belonging to the secondary class. The electronic device may output the disease occurrence results for each of the diseases belonging to the secondary class to the user based on the determined disease occurrence results. That is, the electronic device may output a disease occurrence result based on the probability of a secondary disease for each of the second classifier layers to the user as a single disease prediction result for each of the second classifier layers. However, the single disease prediction result is not limited to the disease occurrence result based on the probability for each disease belonging to the secondary class, but may be a statistical result for each disease belonging to the secondary class. A method of obtaining statistical results for diseases belonging to the secondary class will be described later with reference to FIG. 7.

The primary class 310 according to an embodiment refers to diseases that are unlikely to exist and/or develop at the same time among otitis media-related diseases. For example, the primary class 310 includes diseases that cannot occur together during a certain time period. It can be included. Specifically, the possibility of simultaneous existence may be determined by at least one of the user's judgment or statistical results of past medical records. The user's judgment may be an example of a specific disease being determined as a disease belonging to the primary class by a medical professional or medical personnel. The statistical result may be a result indicating the possibility that certain diseases may exist simultaneously in multiple otoscope images or multiple medical records. Therefore, an otoscope image taken of a patient's eardrum may have a low possibility of simultaneously containing otitis media with effusion and chronic otitis media. The secondary class 320 may include diseases related to otitis media that can occur together over a certain period of time. For example, diseases belonging to the secondary class 320 may be diseases that are likely to co-exist in the patient's eardrum. For example, an otoscope image taken of a patient's eardrum may have a high possibility of simultaneously containing epitympanic cholesteatoma and myringitis.

An electronic device (e.g., the electronic device 100 of FIG. 1) according to an embodiment may apply the otoscope image 410 to the learned otitis media diagnosis model 440. Specifically, the electronic device may remove the patient's personally identifiable information from the otoscope image 410. The electronic device may extract a region of interest 415 having a predetermined shape from the otoscope image 410. The electronic device may place the extracted region of interest 415 at the center of the two-dimensional otoscope image. The electronic device may generate the input otoscope image 430 by placing the region of interest 415 at the center of the otoscope image. The electronic device may apply the input otoscope image 430 generated based on the region of interest 415 extracted from the otoscope image 410 to the learned otitis media diagnosis model 440. The electronic device may obtain a plurality of disease prediction results by applying the input otoscope image 430 to the learned otitis media diagnosis model 440. The electronic device may obtain a plurality of disease prediction results based on feeding forward the input otoscope image 430 to the learned otitis media diagnosis model 440.

By way of example, the otitis media diagnosis model 440 may include a neural network. A neural network includes layers, and each layer can include nodes. A node may have a node value determined based on an activation function. A node in an arbitrary layer may be connected to a node in another layer (e.g., another node) through a link (e.g., a connection edge) with a connection weight. A node's node value can be propagated to other nodes through links. In the inference operation of a neural network, node values may be forward propagated from the previous layer to the next layer. For example, in the otitis media diagnosis model 440, the forward propagation operation may represent an operation that propagates node values based on input data in the direction from the input side of the shared layer 442 toward the classifier layer. The node value of that node can be propagated (e.g., forward propagation) to the node of the next layer (e.g., next node) connected to the node through a connection line. For example, a node may receive a value weighted by a connection weight from a previous node (eg, multiple nodes) connected through a connection line. The node value of a node may be determined based on applying an activation function to the sum of weighted values received from previous nodes (e.g., a weighted sum). Parameters of the neural network may exemplarily include the connection weights described above. The parameters of the neural network may be updated so that the objective function value, which will be described later, changes in the targeted direction (e.g., the direction in which loss is minimized).

The electronic device may extract feature data based on the shared layer 442 by applying the input otoscope image 430 to the learned otitis media diagnosis model 440. The electronic device may output a plurality of disease prediction results based on the classifier layers by applying the extracted feature data to the classifier layers.

The input otoscope image 430 has the region of interest placed in the center and may be an RGB image reformatted to 256 × 256 × 3. The area of interest 415 may be an area where only a certain area (eg, a circle) including the patient's eardrum is selected in the otoscope image. The electronic device may estimate and extract the area corresponding to the patient's eardrum as the user's area of interest 415. The electronic device may place the region of interest 415 at the center of the image. The electronic device may generate an input otoscope image 430, which is an image in which the region of interest is located at the center.

The learned otitis media diagnosis model 440 may represent a model learned through machine learning, and in detail, may be a learned machine learning model that outputs a prediction result regarding otitis media disease from an image or video. The learned otitis media diagnosis model 440 may output a plurality of disease prediction results from the input otoscope image 430.

A machine learning model (eg, the learned otitis media diagnosis model 440) may be created through machine learning. Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. A machine learning model may include multiple artificial neural network layers. Specifically, the learned otitis media diagnosis model 440 includes a shared layer 442 including at least one convolution operation and a plurality of classifier layers connected to the shared layer 442 (e.g. , task-specific layers). Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. For reference, this specification mainly describes an example in which the shared layer 442 of the otitis media diagnosis model 440 is a convolutional neural network (CNN) including at least one convolution operation. For example, the otitis media diagnosis model 440 may be an EfficientNet B-4 model. Additionally, in the case of supervised learning, the above-described machine learning model has a training input (e.g., an otoscope image of a patient for learning (e.g., patient 170 in FIG. 1)) and a training output mapped to the training input. It may be learned based on training data including pairs of (eg, a plurality of disease prediction results for learning). For example, a machine learning model can be trained to output training output from training input. A machine learning model being trained may produce temporary outputs in response to training inputs, and may be trained such that loss between the temporary outputs and the training outputs (e.g., targets of training) is minimized. During the learning process, parameters of the machine learning model (e.g., connection weights between nodes/layers in a neural network) may be updated according to the loss. This learning may be performed, for example, in the electronic device itself on which the machine learning model is performed, or may be performed through a separate server. The machine learning model on which training has been completed (eg, the learned otitis media diagnosis model 440) may be stored in a memory (eg, the memory 120 of FIG. 1).

For reference, the electronic device may use Equation 1 as an objective function for learning the otitis media diagnosis model.

[Equation 1]

here,

may represent the number of training inputs used for learning the otitis media diagnosis model,

may represent the number of classes (for example, 5 classes, referring to FIG. 4). also,

Is

Class of training input

can represent the ground truth of

Is

Class of training input

It can represent a temporary output (e.g., output probability). also,

can represent 'categorical cross-entropy loss'.

In other words, the electronic device may set the above-described Equation 1 as an objective function for learning the machine learning model, and use the machine learning model learned through the above-described objective function as the learned otitis media diagnosis model 440.

The shared layer 442 may include a plurality of convolutional layers, which are an input layer and a hidden layer. The electronic device uses either a first convolutional filter (e.g., 'Conv 3×3' in FIG. 4) or a second convolutional filter (e.g., 'MBConv 3x3' in FIG. 4) in the convolution layer. At least one convolution filter can be used. The convolution layer may be a layer that performs a convolution operation between input data (eg, input otoscope image 430) and a convolution filter. For example, the first convolutional filter and the second convolutional filter may represent learned connection weights. The electronic device may extract feature data based on a convolution operation between the input otoscope image 430 and the learned connection weights in the shared layer 442.

The classifier layers include a first classifier layer 444 that outputs disease prediction results for diseases belonging to the primary class and a second classifier layer 444 that individually outputs single disease prediction results for diseases belonging to the secondary class. It may include 2 classifier layers 446. Each of the classifier layers may include separate learned connection weights (eg, connection weights between nodes/layers in each of the classifier layers). The electronic device may obtain a plurality of disease prediction results based on an operation between feature data extracted from the classifier layer and learned connection weights.

The first classifier layer 444 may include separate layers from the second classifier layers 446. The electronic device may output a disease prediction result from the first classifier layer 444 through calculation between the feature data and the learned connection weight of the first classifier layer 444.

The second classifier layers 446 may include at least one layer separate from the first classifier layer. The electronic device may output a single disease prediction result through calculation between the feature data and the learned connection weights of each layer of the second classifier layers 446.

The electronic device may obtain a plurality of disease prediction results by applying one input otoscope image 430 to one learned otitis media diagnosis model 440. The electronic device may apply one feature data to a plurality of classifier layers and output a plurality of otitis media-related disease prediction results. Through the learned otitis media diagnosis model 440, the electronic device can provide high accuracy and convenience to the user compared to a comparison target model (eg, an independent otitis media diagnosis model). For example, the electronic device may simultaneously provide a plurality of disease prediction results to the user by applying one input otoscope image 430 to one learned otitis media diagnosis model 440. A user can simultaneously obtain a plurality of disease prediction results by applying one input otoscope image 430 to the learned otitis media diagnosis model 440 once. The learned otitis media diagnosis model 440 can simultaneously output a plurality of disease prediction results through a plurality of classifier layers. Additionally, the learned otitis media diagnosis model 440 may have superior performance compared to the model being compared. For example, the learned otitis media diagnosis model 440 may output a result closer to the correct answer than the comparison target model for the same input otoscope image. The accuracy of the learned otitis media diagnosis model 440 will be described later in FIGS. 8 to 10 below.

An electronic device (e.g., the electronic device 100 of FIG. 1) according to an embodiment outputs a plurality of disease prediction results through a learned otitis media diagnosis model (e.g., the learned otitis media diagnosis model 440 of FIG. 4). can do. Specifically, the electronic device may output prediction results for one otitis media disease or multiple otitis media diseases.

The electronic device may output first results 510 to third results 530 for one otitis media disease. For example, if the correct answer to the first result 510 is normal, the electronic device produces a 'None' result indicating the absence of the disease among the diseases belonging to the primary class in the first classifier layer of the otitis media diagnosis model (e.g., The disease prediction result of the first result 510) may be output. Additionally, the electronic device may output a 'False' result (e.g., a single disease prediction result of the first result 510) for each disease belonging to the secondary class in the second classifier layers of the otitis media diagnosis model. . For example, if the correct answer to the second result 520 is Otomycosis, the electronic device produces a 'None' result indicating the absence of the disease among the diseases belonging to the primary class in the first classifier layer of the otitis media diagnosis model. can be output. Additionally, the electronic device may output a 'True' result from the second classifier layer corresponding to ear fungus among the second classifier layers of the otitis media diagnosis model. For example, if the correct answer to the third result 530 is chronic otitis media (COM), the electronic device selects 'chronic otitis media' among the diseases belonging to the primary class in the first classifier layer of the otitis media diagnosis model. COM' results can be output. Additionally, the electronic device may output a 'False' result for each of the diseases belonging to the secondary class in the second classifier layers of the otitis media diagnosis model.

The electronic device may output fourth results 540 and fifth results 550 for a plurality of otitis media diseases. For example, if the correct answer to the fourth result 540 is otitis media with effusion (OME) and Myringitis, the electronic device selects diseases belonging to the primary class in the first classifier layer of the otitis media diagnosis model. The 'OME' result, which indicates otitis media with effusion, can be output. Additionally, the electronic device may output a 'True' result from the second classifier layer corresponding to myringitis among the second classifier layers of the otitis media diagnosis model. For example, if the correct answer to the fifth result 550 is Myringitis and Ventilating tube, the electronic device indicates the absence of the disease among the diseases belonging to the primary class in the first classifier layer of the otitis media diagnosis model. A result of 'None' can be output. Additionally, among the second classifier layers of the otitis media diagnosis model, the electronic device may output a 'True' result from the second classifier layer corresponding to myringitis and a 'True' result from the second classifier layer corresponding to the ventilation duct.

An electronic device (e.g., the electronic device 100 of FIG. 1) according to an embodiment may output a disease prediction result 650a for the input otoscope image 610a of the patient 664a to the user 662a. there is. For example, the electronic device may apply the input otoscope image 610a to the otitis media diagnosis model to obtain a first probability result 640a for diseases belonging to the primary class. Specifically, the electronic device extracts feature data by applying the input otoscope image 610a to the shared layer 620a, and applies the extracted feature data to the first classifier layer 630a among the classifier layers to determine the first classifier layer 630a. A probability result 640a can be obtained. The electronic device may output the disease with the highest probability among the first probability results 640a (for example, chronic otitis media (COM) in FIG. 6A) as the disease prediction result 650a. The electronic device may output the disease prediction result 650a to the user 662a through the display 660a. However, the disease prediction result 650a is not limited to this, and the electronic device may output the disease prediction result to the user 662a for a plurality of input otoscope images of the patient 664a. For example, the electronic device may apply each of a plurality of input otoscope images to an otitis media diagnosis model to obtain probabilities for each of the diseases belonging to the primary class. The electronic device may obtain a first probability result for a plurality of input otoscope images based on a plurality of probabilities obtained for each of the diseases belonging to the primary class. Specifically, the electronic device may calculate at least one of the average, median, or mode of the plurality of probabilities obtained for each of the diseases belonging to the primary class as the probability for each of the diseases belonging to the primary class. The electronic device may output a disease with the highest probability among first probability results for a plurality of input otoscope images as a disease prediction result.

The electronic device according to one embodiment may output single disease prediction results 650b for the input otoscope image 610b of the patient 664b to the user 662b. For example, the electronic device may apply the input otoscope image 610b to the otitis media diagnosis model to obtain second probability results 640b for diseases belonging to the secondary class. Specifically, the electronic device extracts feature data by applying the input otoscope image 610b to the shared layer 620b, and applies the extracted feature data to the second classifier layers 630b among the classifier layers. 2 probability results 640b may be obtained. If the probability for each of the diseases belonging to the secondary class is greater than or equal to a threshold predetermined by the user, the electronic device may determine the disease occurrence result for each of the diseases belonging to the secondary class. The electronic device determines whether the single disease prediction results 650b are greater than or equal to a predetermined threshold (for example, in FIG. 6B, a probability of 50% is set as the threshold) for the second probability results 640b. ) can be output. For example, in FIG. 6B, the electronic device may output a 'True' result for Attic Cholesteatoma because the probability of Attic Cholesteatoma is 80%. Here, a 'True' result for epitympanic cholesteatoma may indicate that the patient 664b has developed epitympanic cholesteatoma disease. Conversely, the electronic device may output a 'False' result for otomycosis because the probability of otomycosis is 45%. Here, a result of 'False' for ear mold disease may indicate that the patient 664b does not develop ear mold disease. The electronic device may output single disease prediction results 650b to the user 662b through the display 660b. However, the single disease prediction results 650b are not limited to this, and the electronic device may output single disease prediction results to the user 662b for a plurality of input otoscope images of the patient 664b. For example, the electronic device may apply each of a plurality of input otoscope images to an otitis media diagnosis model to obtain probabilities for each of the diseases belonging to the secondary class. The electronic device may obtain second probability results for a plurality of input otoscope images based on the plurality of probabilities obtained for each of the diseases belonging to the secondary class. Specifically, the electronic device may calculate at least one of the average, median, or mode of the plurality of probabilities obtained for each of the diseases belonging to the secondary class as the probability for each of the diseases belonging to the secondary class. The electronic device may output single disease prediction results based on whether the second probability results for the plurality of input otoscope images are greater than or equal to a predetermined threshold.

The electronic device according to one embodiment may output a disease prediction result 650c and single disease prediction results 652c for the input otoscope image 610c. For example, the electronic device may apply the input otoscope image 610c to the otitis media diagnosis model 620c and output a disease prediction result 650c and single disease prediction results 652c.

The electronic device may extract feature data by applying the input otoscope image 610c to the shared layer 625c. The electronic device may simultaneously apply the extracted feature data to the first classifier layer 630c and the second classifier layers 632c. First, the electronic device may obtain a first probability result 640c by applying the extracted feature data to the first classifier layer 630c. Additionally, the electronic device may acquire the first probability result 640c and simultaneously obtain second probability results 642c by applying the extracted feature data to the second classifier layers 632c.

The electronic device may output the disease with the highest probability among the first probability results 640c as the disease prediction result 650c. If the probability for each of the diseases belonging to the secondary class is greater than or equal to a threshold predetermined by the user, the electronic device may determine the disease occurrence result for each of the diseases belonging to the secondary class. The electronic device may output single disease prediction results 652c based on whether the second probability results 640c are equal to or greater than a predetermined threshold.

As a result, the electronic device applies one input otoscope image 610c to the otitis media diagnosis model 620c to provide a disease prediction result 650c and single disease prediction results 652c to a user (e.g., a medical professional). Can be printed.

An electronic device (eg, the electronic device 100 of FIG. 1 ) according to an embodiment may extract a plurality of feature data from the input otoscope image 710 . For example, the electronic device may extract feature data by performing a forward propagation operation of the otitis media diagnosis model while skipping at least some of the connections between nodes of some of the shared layers. The electronic device may extract the first feature data 712 by skipping the first connection among the shared layers of the otitis media diagnosis model 720. The first connection may be a connection excluded from the first forward propagation operation among connections between nodes of the shared layer. The first feature data 712 may include abstracted values extracted by applying the input otoscope image 710 to the shared layer in which the first connection was skipped. The electronic device may extract the second feature data 714 from the shared layer of the otitis media diagnosis model 730 by skipping the second connection that is different from the first connection. The second connection may be a connection excluded from the second forward propagation operation among connections between nodes of the shared layer. Connections between nodes that are skipped or excluded in the first forward propagation operation and the second forward propagation operation may vary. The second feature data 714 may include abstracted values extracted by applying the input otoscope image 710 to the shared layer in which the second connection is skipped. In other words, the electronic device can extract a plurality of different feature data from one input otoscope image 710 by skipping any connection in the shared layer.

The electronic device may apply the first feature data 712 to the first classifier layer to obtain the first disease prediction probability 722. The first disease prediction probability 722 may represent occurrence probabilities for diseases belonging to the primary class. The electronic device may convert the first disease prediction probability 722 into a first disease binary result 726 based on a predetermined threshold. Additionally, the electronic device may apply the first feature data 712 to the first classifier layer and simultaneously apply it to the second classifier layers to obtain first single disease prediction probabilities 724. The first single disease prediction probabilities 724 may represent occurrence probabilities for diseases belonging to the secondary class. The electronic device may convert the first single disease prediction probabilities 724 into a first single disease binary result 728 based on a predetermined threshold.

The electronic device may apply the second feature data 714 to the first classifier layer to obtain the second disease prediction probability 732. The second disease prediction probability 732 may represent occurrence probabilities for diseases belonging to the primary class. The electronic device may convert the second disease prediction probability 732 into a second disease binary result 736 based on a predetermined threshold. Additionally, the electronic device may apply the second feature data 714 to the first classifier layer and simultaneously apply it to the second classifier layers to obtain second single disease prediction probabilities 734. The second single disease prediction probabilities 734 may represent occurrence probabilities for diseases belonging to a secondary class. The electronic device may convert the second single disease prediction probabilities 734 into a second single disease binary result 738 based on a predetermined threshold.

The electronic device may obtain a first statistical result representing an average for the first disease binary outcome 726 and the second disease binary outcome 736. For example, the first statistical result may include statistical results for diseases belonging to the primary class. Specifically, the electronic device uses the average of the binary results of chronic otitis media (COM) of the first disease binary outcome 726 and chronic otitis media of the second disease binary outcome 736 as the statistical result of chronic otitis media. You can. The electronic device may average the binary results of otitis media with effusion (OME) of the first disease binary outcome 726 and otitis media with effusion of the second disease binary outcome 736 into a statistical result of otitis media with effusion. . The electronic device may take the average of the binary results of the absence of the disease in the first disease binary result 726 and the absence of the disease in the second binary result 736 as a statistical result of the absence of the disease. As a result, the first statistical result may include statistical results that are 1 for chronic otitis media, 0 for otitis media with effusion, and 0 for absence of disease. Additionally, the electronic device may output the above-described first statistical result to the user as a disease prediction result for the input otoscope image 710.

The electronic device may obtain a second statistical result representing an average for the first single disease binary outcome 728 and the second single disease binary outcome 738. For example, the second statistical result may include statistical results for diseases belonging to a secondary class. Specifically, the electronic device averages the binary outcomes Attic Cholesteatoma of the first single disease binary outcome 728 and Attic Cholesteatoma of the second single disease binary outcome 738, This can be done with statistical results. The electronic device may average the binary outcomes of Myringitis of the first single disease binary outcome 728 and Myringitis of the second single disease binary outcome 738 into a statistical result of Myringitis. The electronic device may average the binary outcomes of Otomycosis of the first single disease binary outcome 728 and Otomycosis of the second single disease binary outcome 738 into a statistical result of Otomycosis. . The electronic device may average the binary results of the ventilating tube of the first single disease binary result 728 and the ventilating tube of the second single disease binary result 738 as a statistical result of the ventilating tube. As a result, the secondary statistical results may include statistical results that are 1 for epitympanic cholesteatoma, 0.5 for tympanitis, 0.5 for ear fungus, and 0 for ventilator tract. Additionally, the electronic device may output the above-described second statistical result to the user as a single disease prediction result for the input otoscope image 710.

However, each of the first and second statistical results is not limited to the disease prediction result for the input otoscope image 710 and the single disease prediction result. For example, the electronic device may output the first disease prediction probability 722 and the second disease prediction probability 732 to the user as a disease prediction result for the input otoscope image 710. Additionally, the electronic device may output first single disease prediction probabilities 724 and second single disease prediction probabilities 734 to the user as a single disease prediction result for the input otoscope image 710.

The electronic device may provide the user with an output suggesting retry of otitis media diagnosis based on at least one of the disease prediction probability or the single disease prediction probability. The disease prediction probability may include at least one of the first disease prediction probability 722 or the second disease prediction probability 732. The single disease prediction probability may include at least one of the first single disease prediction probabilities 724 or the second single disease prediction probabilities 734.

For example, based on the first disease prediction probability 722, the electronic device may provide an output suggesting that the user retry diagnosing otitis media. The electronic device may provide an output suggesting retry of otitis media diagnosis based on the difference between the probability of the disease prediction result and the probability of the target result. Here, the probability of the disease prediction result may be the probability of the disease having the first probability among the first disease prediction probabilities 722, and the probability of the target result may be the probability of the disease having the second priority probability among the first disease prediction probabilities 722. It can be. Referring to FIG. 7, the disease with the first probability may be chronic otitis media (COM), and the disease with the second highest probability may be otitis media with effusion (OME). The electronic device may provide an output to the user suggesting retrying the otitis media diagnosis through another otoscope image if the calculated difference is below a threshold. Here, the threshold may be a threshold for retrying otitis media diagnosis based on the first disease prediction probability 722.

For example, based on the first single disease prediction probabilities 724, the electronic device may provide an output suggesting that the user retry diagnosing otitis media. The electronic device may select second classifier layers related to the secondary class disease in which the disease occurred among the first single disease prediction probabilities 724. The electronic device may provide an output suggesting retry of otitis media diagnosis based on a case where at least one of the probabilities related to the secondary disease for the selected layers is less than a threshold value. Here, the threshold may be a threshold for retrying otitis media diagnosis based on the first single disease prediction probabilities 724. Referring to FIG. 7, the electronic device divides the second classifier layers for Attic Cholesteatoma and Otomycosis among the first single disease prediction probabilities 724 into a second classifier layer in which the disease occurs. You can choose from: The electronic device may determine whether the probability of Attic Cholesteatoma exceeds a threshold determined by the user. Additionally, the electronic device may determine whether the probability of Otomycosis exceeds a threshold determined by the user. If at least one of the probability of Attic Cholesteatoma or Otomycosis does not exceed the threshold, the electronic device prompts the user to retry diagnosing otitis media through another otoscope image. Suggested output can be provided.

An electronic device (eg, the electronic device 100 of FIG. 1 ) according to an embodiment may acquire the performance of an otitis media diagnosis model (eg, the otitis media diagnosis model 130 of FIG. 1 ). For example, the electronic device may obtain performance difference results between the otitis media diagnosis model and the comparison target model. The electronic device can obtain results of the performance difference between the otitis media diagnosis model and the comparative model through the McNemar test results. The model to be compared may represent a model that outputs one disease prediction result (e.g., a prediction result for otitis media with effusion disease) for one input data (e.g., an input otoscope image). The electronic device can obtain the performance difference results between the otitis media diagnosis model and the comparison target model through the DSC (Dice similarity coefficient) results among the McNemar test results. DSC results may primarily indicate differences between correct and predicted results of video images (e.g., input otoscope images). That is, the higher the DSC result of at least one of the otitis media diagnosis model or the comparison target model, the smaller the difference between the correct answer and the model's predicted results may be indicated.

Referring to the table shown in FIG. 8, the otitis media diagnosis model may have a smaller difference between the correct answer and the predicted result for the remaining diseases except for myringitis, compared to the comparison target model. For example, referring to the result 830, the result 830 may include differences between the DSC result 810 of the otitis media diagnosis model and the DSC result 820 of the comparison target model. In other words, the otitis media diagnosis model may be a model that has superior prediction performance for all diseases except myringitis compared to the comparison model.

An electronic device (e.g., the electronic device 100 of FIG. 1) according to an embodiment may acquire a confusion matrix of an otitis media diagnosis model (e.g., the otitis media diagnosis model 130 of FIG. 1). For example, a confusion matrix can represent a matrix containing elements representing the ratio between the prediction results of a machine learning model and the correct answer. Referring to the table shown in FIG. 9, each matrix may include a row regarding the ground truth (GT) class and a column regarding the prediction result class. For example, referring to the confusion matrix for the primary class, the electronic device selects 1,534 images containing chronic otitis media (COM) disease (e.g., an image of the eardrum containing chronic otitis media) ), 1,463 chronic otitis media disease prediction results can be output.

An electronic device (e.g., the electronic device 100 of FIG. 1) according to an embodiment acquires a receiver operating characteristics (ROC) curve of an otitis media diagnosis model (e.g., the otitis media diagnosis model 130 of FIG. 1) and a comparison target model. can do. For reference, in FIG. 10, the otitis media diagnosis model may be shown as a Combined model, and the model to be compared may be shown as a Separate model. For example, the primary class result 1010 may include an ROC curve regarding the primary class of the otitis media diagnosis model and the comparison target model. The primary class result 1010 may include AUC (area under the ROC curve) values for diseases belonging to the primary class of the otitis media diagnosis model and the comparison target model. Referring to FIG. 10, it can be seen that in the results of diseases belonging to the primary class, the otitis media diagnosis model can have a superior AUC value than the comparison model. The secondary class result 1020 may include an ROC curve regarding the secondary class of the otitis media diagnosis model and the comparison target model. The secondary class result 1020 may include AUC (area under the ROC curve) values for diseases belonging to the secondary class of the otitis media diagnosis model and the model to be compared. Referring to FIG. 10, it can be seen that in the results of diseases belonging to the secondary class excluding myringitis disease, the otitis media diagnosis model can have a superior AUC value than the comparison model.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. It may be possible. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims

In the otitis media diagnostic electronic device,

a memory storing computer-executable instructions and a learned otitis media diagnosis model including a shared layer including at least one convolution operation and a plurality of classifier layers connected to the shared layer;

a processor that accesses the memory and executes the instructions;

a display electrically connected to the processor; and

Image acquisition unit that receives the patient's otoendoscopic image

Including,

The above commands are:

Receiving an otoscope image of the patient,

Generating an input otoscope image based on a region of interest extracted from the received otoscope image,

Extracting feature data from the input otoscope image based on the shared layer,

Based on a first classifier layer among the plurality of classifier layers, output disease prediction results for diseases belonging to a primary class from the extracted feature data,

Based on the plurality of second classifier layers separated from the first classifier layer among the plurality of classifier layers, a single disease prediction result for each disease belonging to the secondary class is generated by the plurality of second classifiers. Individually output from the corresponding second classifier layer among the layers,

Otitis media diagnostic electronic device.
According to paragraph 1,

The processor,

Receiving a video sequence of the patient's tympanic membrane,

Obtaining a plurality of otoscope images using the video sequence as many frames as determined by the user.

Otitis media diagnostic electronic device.
According to paragraph 1,

The processor,

Remove the patient's personally identifiable information from the otoscope image,

Extracting the region of interest having a predetermined shape from the otoscope image,

Placing the extracted region of interest at the center of the otoscope image in two-dimensional form.

Otitis media diagnostic electronic device.
According to paragraph 1,

The processor,

Based on the first classifier layer, obtain probabilities regarding primary diseases belonging to the primary class from the extracted feature data,

Outputting the disease with the highest probability among the probabilities of the primary diseases to the user as the disease prediction result,

Otitis media diagnostic electronic device.
According to paragraph 1,

The processor,

Based on each of the second classifier layers, obtain a probability regarding a secondary disease belonging to the secondary class from the extracted feature data,

Outputting disease occurrence results based on the probability of the secondary disease for each of the second classifier layers to the user as a single disease prediction result for each of the second classifier layers,

Otitis media diagnostic electronic device.
According to paragraph 4,

The processor,

Among the diseases excluding the disease corresponding to the disease prediction result in the primary class, the disease with the highest probability is set as the target result,

Based on the case where the difference between the probability of the disease prediction result and the probability of the target result is less than a threshold determined by the user, an output is provided suggesting that the user retry diagnosing otitis media through an otoscope image different from the otoscope image. providing,

Otitis media diagnostic electronic device.
According to clause 5,

The processor,

Among the disease occurrence results related to the secondary disease, select second classifier layers related to the secondary class disease in which the disease occurred,

Based on a case where at least one of the probabilities related to secondary disease for the selected layers is less than a threshold determined by the user, suggesting to the user to retry diagnosing otitis media through an otoscope image different from the otoscope image providing output,

Otitis media diagnostic electronic device.
According to paragraph 1,

The processor,

Extracting feature data based on skipping at least some of the connections between nodes of the shared layers of the learned otitis media diagnosis model,

Otitis media diagnostic electronic device.
According to paragraph 1,

The processor,

Extracting first feature data based on skipping a selected first connection among connections between nodes of the shared layers,

Extracting second feature data based on skipping a second connection different from the first connection among connections between nodes of the shared layers,

Extracting a plurality of feature data including the first feature data and the second feature data while repeating changes in connections skipped between the nodes.

Otitis media diagnostic electronic device.
According to clause 9,

The processor,

Based on the first classifier layer, obtain probabilities for diseases belonging to the primary class for each of the plurality of feature data,

For each of the plurality of feature data, the probabilities for diseases obtained from the first classifier layer are converted into disease binary results based on a predetermined threshold,

For each of the diseases belonging to the primary class, obtain a first statistical result representing the average of the plurality of disease binary outcomes,

Outputting the highest first statistical result among diseases belonging to the primary class to the user as the disease prediction result,

Otitis media diagnostic electronic device.
According to clause 9,

The processor,

For each of the second classifier layers, apply the plurality of feature data to obtain probabilities for diseases belonging to the secondary class for each of the plurality of feature data,

For each of the second classifier layers, the probability corresponding to each of the plurality of feature data is converted into a single disease binary result based on a predetermined threshold value,

For each of the second classifier layers, obtain a second statistical result representing an average for the plurality of single disease binary outcomes,

Outputting a second statistical result for each of the second classifier layers individually as the single disease prediction result in a corresponding second classifier layer among the plurality of second classifier layers,

Otitis media diagnostic electronic device.
In the otitis media diagnosis method performed by a processor,

Receiving an otoscope image of a patient;

generating an input otoscope image based on a region of interest extracted from the received otoscope image;

extracting feature data from the input otoscope image based on a shared layer of an otitis media diagnosis model;

Outputting disease prediction results for diseases belonging to a primary class from the extracted feature data based on a first classifier layer among the plurality of classifier layers of the otitis media diagnosis model; and

Based on the plurality of second classifier layers separated from the first classifier layer among the plurality of classifier layers, a single disease prediction result for each disease belonging to the secondary class is generated by the plurality of second classifiers. Step of individually outputting from the corresponding second classifier layer among the layers

Otitis media diagnosis method comprising.
According to clause 12,

The step of receiving the otoscope image is,

Receiving a video sequence of the patient's tympanic membrane; and

Obtaining a plurality of otoscope images from the video sequence as many frames as determined by the user.

Otitis media diagnosis method comprising.
According to clause 12,

The step of generating the input otoscope image is,

removing the patient's personally identifiable information from the otoscope image;

extracting the region of interest having a predetermined shape from the otoscope image; and

Placing the extracted region of interest at the center of the otoscope image in two-dimensional form.

Otitis media diagnosis method comprising.
According to clause 12,

The step of outputting the disease prediction result is,

Based on the first classifier layer, obtaining probabilities regarding primary diseases belonging to the primary class from the extracted feature data; and

Outputting a disease with the highest probability among the probabilities of the primary diseases to the user as the disease prediction result.

Including,

How to diagnose otitis media.
According to clause 12,

The step of individually outputting the single disease prediction result from a corresponding second classifier layer among the plurality of second classifier layers,

Based on each of the second classifier layers, obtaining a probability regarding a secondary disease belonging to the secondary class from the extracted feature data; and

Outputting a disease occurrence result based on the probability of the secondary disease for each of the second classifier layers to the user as a single disease prediction result for each of the second classifier layers.

Including,

How to diagnose otitis media.
According to clause 15,

The step of outputting to the user is,

setting the probability of a disease with the highest probability as a target result among diseases excluding the disease corresponding to the disease prediction result in the primary class; and

Based on the case where the difference between the probability of the disease prediction result and the probability of the target result is less than a threshold determined by the user, an output is provided suggesting that the user retry diagnosing otitis media through an otoscope image different from the otoscope image. steps provided

Otitis media diagnosis method comprising.
According to clause 16,

The step of outputting to the user is,

selecting second classifier layers related to the secondary class disease in which the disease occurred from among disease occurrence results related to the secondary disease; and

Based on a case where at least one of the probabilities related to secondary disease for the selected layers is less than a threshold determined by the user, suggesting to the user to retry diagnosing otitis media through an otoscope image different from the otoscope image Steps that provide output

Otitis media diagnosis method comprising.
According to clause 12,

The step of extracting the feature data is,

Extracting feature data based on skipping at least some of the connections between nodes of the shared layers of the learned otitis media diagnosis model.

How to include .
According to clause 12,

The step of extracting the feature data is,

extracting first feature data based on skipping a selected first connection among connections between nodes of the shared layers;

extracting second feature data based on skipping a second connection different from the first connection among connections between nodes of the shared layers; and

Extracting a plurality of feature data including the first feature data and the second feature data while repeating changes in skip connections between the nodes.

Otitis media diagnosis method comprising.
According to clause 20,

Based on the first classifier layer, obtaining probabilities for diseases belonging to the primary class for each of the plurality of feature data;

For each of the plurality of feature data, converting the probability of diseases obtained from the first classifier layer into a disease binary result based on a predetermined threshold value;

For each of the diseases belonging to the primary class, obtaining a first statistical result representing an average of the plurality of disease results; and

Outputting the highest first statistical result among diseases belonging to the primary class to the user as the disease prediction result.

A method for diagnosing otitis media further comprising:
According to clause 20,

For each of the second classifier layers, applying the plurality of feature data to obtain probabilities for diseases belonging to the secondary class for each of the plurality of feature data;

For each of the second classifier layers, converting a probability corresponding to each of the plurality of feature data into a single disease binary result based on a predetermined threshold value;

For each of the second classifier layers, obtaining a second statistical result representing an average for the plurality of single disease binary outcomes; and

Outputting a second statistical result for each of the second classifier layers individually as the single disease prediction result in a corresponding second classifier layer among the plurality of second classifier layers.

A method for diagnosing otitis media further comprising:
A computer program combined with hardware and stored in a computer-readable recording medium to execute the method of any one of claims 12 to 22.