KR102595646B1 - Tympanic disease prediction model system using deep neural network and monte carlo dropout - Google Patents

Abstract

The present invention relates to an eardrum disease prediction model system using a deep neural network and Monte Carlo dropout, which includes an eardrum imaging unit that collects and classifies eardrum images, disease names and hearing characteristics corresponding to the diagnostic eardrum images classified by the eardrum imaging unit. A label generation unit that generates a numerical value as a label, an eardrum area extraction unit that recognizes the eardrum area in the eardrum image and extracts it as an eardrum area, and uses global average pooling to extract the eardrum area extracted from the eardrum area extraction unit. It includes a vector conversion unit that converts the undiagnosed eardrum vector into an undiagnosed eardrum vector, and a disease diagnosis unit that performs disease diagnosis on the undiagnosed eardrum vector converted by the vector conversion unit using a Softmax activation function and Monte Carlo dropout. Do it as

Description

Eardrum disease prediction model system using deep neural network and Monte Carlo dropout {TYMPANIC DISEASE PREDICTION MODEL SYSTEM USING DEEP NEURAL NETWORK AND MONTE CARLO DROPOUT}

The present invention relates to an eardrum disease prediction system using an artificial neural network, and more specifically, an eardrum using a deep neural network and Monte Carlo dropout that performs eardrum image analysis using an artificial neural network and a function to determine whether the eardrum is diseased and predicts the disease. It is about a disease prediction model system.

The ear, which is the organ most closely related to human hearing, consists of the outer ear, external auditory canal, tympanic membrane, semicircular canal, middle ear, vestibular organ, cochlea, inner ear, ear canal, and Eustachian tube. The diagnosis can be made by combining the patient's symptoms and ear endoscopy images.

In particular, because abnormal signals from the hearing organ can be felt and felt up close by the patient, it is an important factor in making a diagnosis to properly explain to the doctor the pain, tinnitus, and abnormal phenomena felt by the patient at the time of diagnosis.

However, people who are poor at expressing themselves or have problems expressing their opinions, such as children, the elderly, and the disabled, cannot properly hear expressions of abnormalities in their hearing organs, so diagnosis must be performed only through endoscopic imaging. The accuracy may be lower than that of diagnosis using images and the patient's expression of opinion, and in the case of diagnosis using only endoscopic images, there is a problem that if the person performing the diagnosis makes a mistake, a completely incorrect diagnosis may be performed.

Therefore, a tympanic disease prediction model system that collects endoscopic images with ear diseases from external institutions or external databases and determines the presence of disease through deep learning image analysis of the collected ear disease image data and the endoscopic images of the patient being analyzed. Research is required.

Korean Patent No. 10-2047237

The purpose of the present invention is to determine the presence and type of disease by analyzing images inside the patient's ear taken using an endoscope through machine learning.

In addition, by analyzing the sensing data collected through various types of sensors in the ear endoscopic imaging device that photographs the inside of the patient's ear to diagnose failure, the reliability of the image data can be improved by distinguishing between images collected from imaging devices that have been determined to be malfunctioning. The purpose is to maintain it.

In addition, by using Monte Carlo dropout when performing machine learning, the purpose is to create an ensemble effect that increases accuracy by using multiple models and output more generalized results.

An tympanic disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention includes an tympanic membrane imaging unit that collects and classifies tympanic membrane images, and a tympanic membrane image corresponding to the diagnostic tympanic image classified by the tympanic membrane imaging unit. A label generation unit that generates the disease name and hearing level as a label, an eardrum area extraction unit that recognizes the eardrum area in the eardrum image and extracts it as an eardrum area, and a global average pooling of the eardrum area extracted from the eardrum area extraction unit. It includes a vector conversion unit that converts the undiagnosed eardrum vector into an undiagnosed eardrum vector using a Softmax activation function and a Monte Carlo dropout, and a disease diagnosis unit that performs disease diagnosis of the undiagnosed eardrum vector converted in the vector conversion unit. can do.

In addition, the eardrum imaging unit includes an undiagnosed eardrum image collection unit that collects an undiagnosed eardrum image of a patient taken using an ear endoscopic imaging device, a diagnostic eardrum image that is an ear endoscopic image of a patient whose diagnosis has been completed from the outside, a disease name, and It may include a diagnostic eardrum image collection unit that collects hearing values, and a diagnostic eardrum image classification unit that classifies disease names and hearing values corresponding to the diagnostic eardrum images.

In addition, the eardrum imaging unit analyzes the distribution data of the image histogram of the eardrum image generated by the histogram generator and the histogram generator for generating image histogram distribution data of the eardrum image collected using the ear endoscopic imaging device, thereby providing reliability. , but if the average value (A _err ) calculated by the following [Equation 1] is greater than the preset limit value (S _err ), failure diagnosis to determine that an error has occurred in the ear endoscope imaging device and the eardrum image. It can include more wealth.

[Equation 1]

(Here, A _err is the average error of the image histogram distribution data of the eardrum image, T _aver is the overall average of the image histogram distribution data of the eardrum image, and P _aver is the partial average of the histogram distribution data for the n eardrum image images. , T _σ means the overall standard deviation of the image histogram distribution data of the eardrum image)

In addition, the eardrum area extraction unit recognizes the eardrum area in the eardrum image collected by the eardrum imaging unit, and selects a circular area extracted from the eardrum area using Hough Circle Transform processing technology as a region of interest. It may include a region of interest extractor that extracts a region of interest (Interest) and an eardrum center region extractor that extracts a preset area based on the position of the eardrum within the region of interest as the eardrum center region.

In addition, the disease diagnosis unit performs disease diagnosis corresponding to a plurality of disease labels generated by the label generation unit using the softmax activation function on the undiagnosed eardrum vector, and performs disease diagnosis on the undiagnosed eardrum for which the disease diagnosis has been performed. Each hearing value of the disease label corresponding to the vector can be calculated using the Monte Carlo dropout technique.

According to the present invention, the presence and type of disease can be determined by analyzing images inside the patient's ear taken using an endoscope through machine learning.

In addition, by analyzing the sensing data collected through various types of sensors in the ear endoscopic imaging device that photographs the inside of the patient's ear to diagnose failure, the reliability of the image data can be improved by distinguishing between images collected from imaging devices that have been determined to be malfunctioning. It can be maintained.

Additionally, by using Monte Carlo dropout when performing machine learning, it is possible to output more generalized results by creating an ensemble effect that increases accuracy by using multiple models.

Figure 1 is a block diagram of an eardrum disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention.
Figure 2 is a diagram showing an intermediate block diagram of an eardrum disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention.
Figure 3 is a diagram showing an intermediate block diagram of an eardrum disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention.
Figure 4 is a diagram illustrating the tympanic membrane image extraction process of the tympanic membrane disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention.
Figure 5 is a diagram showing a diagnostic eardrum image collected from outside by an eardrum disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention.
Figure 6 is a diagram illustrating a process in which an eardrum disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention converts the output value into a vector by applying global average pooling.

Specific details, including the problem to be solved by the present invention, the means for solving the problem, and the effect of the invention, are included in the examples and drawings described below. The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings.

The scope of the present invention is not limited to the embodiments described below, and various modifications can be made by those skilled in the art without departing from the technical gist of the present invention.

Hereinafter, the eardrum disease prediction model system using the deep neural network and Monte Carlo dropout of the present invention will be described in detail with reference to the attached FIGS. 1 to 5.

First, Figure 1 is a block diagram of an eardrum disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention, and Figure 2 is a block diagram of a deep neural network and Monte Carlo dropout according to an embodiment of the present invention. It is a diagram showing an intermediate block diagram of an eardrum disease prediction model system using dropout, and Figure 3 shows an intermediate block diagram of an eardrum disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention. It is a diagram, and Figure 4 is a diagram illustrating the process of extracting an eardrum image of an eardrum disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention, and Figure 5 is a diagram illustrating a deep neural network and Monte Carlo dropout according to an embodiment of the present invention. A diagram showing diagnostic eardrum images collected from outside by a tympanic disease prediction model system using a neural network and Monte Carlo dropout, and FIG. 6 is a tympanic disease prediction model system using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention. This diagram shows the process of converting the output value to a vector by applying global average pooling.

Referring to Figure 1, the eardrum disease prediction model system 100 using a deep neural network and Monte Carlo dropout according to an embodiment of the present invention includes an eardrum imaging unit 110, a label generation unit 120, and an eardrum area extraction unit ( 130), a vector conversion unit 140, and a disease diagnosis unit 150.

The eardrum imaging unit 110 can collect and classify eardrum images.

Here, the eardrum imaging unit 110 will be described in more detail with reference to FIG. 2 .

Referring to FIG. 2 , the eardrum imaging unit 110 may include a non-diagnosed eardrum image collection unit 111, a diagnostic eardrum image collection unit 112, and a diagnostic eardrum image classification unit 113.

The undiagnosed eardrum image collection unit 111 may collect images of the patient's undiagnosed eardrum captured using an ear endoscopic imaging device.

Here, the undiagnosed eardrum image is an image taken of the patient's ear using the ear endoscopic imaging device, and is not subject to the determination of the name of the disease and calculation of hearing values because the disease has not been currently diagnosed by a medical professional (such as a doctor) in charge of diagnosing the disease. It can mean a video that is.

The diagnostic eardrum image collection unit 112 may collect the diagnostic eardrum image, which is an ear endoscopic image of a patient whose diagnosis has been completed, the name of the disease, and the hearing level from the outside.

Here, the reason for collecting the diagnostic eardrum images is because the already diagnosed eardrum images are used as basic data for future diagnosis, and the diagnostic eardrum images and the disease names diagnosed in the corresponding images (diagnostic eardrum images) are necessarily collected, Depending on the type of disease, hearing measurements may be collected without necessarily being recorded.

Meanwhile, the diagnostic eardrum image collected through the diagnostic eardrum image collection unit 112 will be described in more detail with reference to FIG. 5 .

Referring to FIG. 5 , the diagnostic eardrum image collected by the diagnostic eardrum image collection unit 112 may mean an eardrum image diagnosed with various types of diseases.

For example, in the diagnostic eardrum image collected by the diagnostic eardrum image collection unit 112, the first diagnostic eardrum image 510 is a moderate hearing loss, and after a doctor diagnoses the disease, the disease name is 'hearing loss (moderate)' and the hearing value is the corresponding diagnosis. The second diagnostic eardrum image 520 is diagnosed as a normal eardrum image, so the disease name can be collected as 'normal' and the hearing level can be collected as '21dBHL'. In addition, the third diagnostic tympanic membrane image 530 is an image of the tympanic membrane after a doctor diagnoses otitis media, with the disease name indicated as 'otitis media (acute)' and the hearing level as '25dbHL' or 'no hearing level test' measured at the time of the diagnosis. can be collected.

In addition, the disease name can be classified into acute otitis media, chronic otitis media, otitis media with effusion, congenital cholesteatoma, traumatic eardrum perforation, hearing loss, normal eardrum, etc.

Meanwhile, the diagnostic eardrum image collection unit 112 classifies the disease according to the name of the diagnosed disease, but can collect all eardrum images for which a diagnosis has already been performed, regardless of the name of the diagnosis.

Referring again to FIG. 2, the diagnostic eardrum image classification unit 113 may classify the disease name and hearing value corresponding to the diagnostic eardrum image.

More specifically, the diagnostic eardrum images collected by the diagnostic eardrum image collection unit 112 may be classified in detail according to the name of the disease or hearing level.

For example, in the case of the first diagnostic eardrum image 510, since 'hearing loss (moderate)' was diagnosed, it is grouped and classified with diagnostic eardrum images diagnosed as 'hearing loss', and the third diagnostic eardrum image 530 In the case of, 'otitis media (acute)' was diagnosed, so it can be classified and grouped with diagnostic tympanic membrane images diagnosed as 'otitis media'.

In addition, the tympanic membrane imaging unit 110 is equipped with an ear endoscopic imaging device including a lighting device and an imaging device having a diameter of about 1 mm to 3 mm, and is inserted into the external auditory canal of a patient (subject to capture images inside the ear). You can photograph the eardrum and save the captured video as an image file.

Here, the eardrum imaging unit 110 may determine whether to create an image file for the image inside the patient's ear captured in response to the control of the control engineer or doctor controlling the ear endoscopic imaging device.

Meanwhile, the eardrum imaging unit 110 may further include a histogram generator (not shown) and a fault diagnosis unit (not shown).

The histogram generator (not shown) calculates the color distribution (RGB color code value, etc.), contrast, brightness, and Histogram distribution data for image characteristics such as the total number of pixels can be generated.

The fault diagnosis unit (not shown) determines reliability by analyzing the image histogram distribution data of the eardrum image generated by the histogram generator (not shown), and determines the reliability by calculating the average value (A _err ) is greater than the preset threshold value (S _err ), it may be determined that an error has occurred in the ear endoscope imaging device and the eardrum image captured from the ear endoscopic imaging device.

[Equation 1]

Here, A _err is the average error of the image histogram distribution data of the eardrum image, T _aver is the overall average of the image histogram distribution data of the eardrum image, P _aver is the partial average of the histogram distribution data for the n eardrum image images, T _σ may mean the overall standard deviation of image histogram distribution data of the eardrum image.

More specifically, T _aver is the overall average of the image histogram distribution data of the eardrum image, and calculates the overall average of the histogram distribution data generated for a single eardrum image image generated during a preset period (ex. one month). means a value, and T _σ means a value calculated from the total standard deviation of the image histogram distribution data of the eardrum image during the preset period (ex. one month).

In addition, P _aver is a partial average of the histogram distribution data for n diagnostic eardrum image images among the eardrum image images generated during a preset period (ex. one month), and is obtained from the ear endoscope imaging device of the image capture unit 112. In the process of collecting captured undiagnosed eardrum image data, histogram distribution data for a preset number (n) of diagnostic eardrum image images is generated, and histogram distribution data for the preset number (n) of diagnostic eardrum images are generated. The average is calculated and can be referred to as a partial average because it corresponds to the average of some histogram distribution data.

At this time, if the estimated average value is calculated with 95% confidence using partial averages, the estimated average value (μ) is It has a range.

Therefore, the average error (A _err ), which is the difference between the upper or lower limit of the estimated average value (μ) and the overall average (T _aver ), can be calculated as in [Equation 1] above.

Therefore, the fact that the average error (A _err ) calculated by [Equation 1] is greater than the preset limit error (S _err ) means that the undiagnosed eardrum image collected through the undiagnosed eardrum image collection unit 111 is Since it means that there is a very high possibility that the input is incorrect due to a failure of the ear endoscope imaging device, etc., the failure diagnosis unit (not shown) collects the undiagnosed eardrum image through the image capture unit 112 when the above conditions are satisfied. Data can be judged as unreliable and displayed to the user.

Meanwhile, the fault diagnosis unit (not shown) compares the average value (A _err ) calculated by [Equation 1] and a preset threshold value (S _err ) to determine whether there is an error in the undiagnosed eardrum image data. At the same time, a temperature sensor that measures the temperature in the ear endoscope imaging device of the undiagnosed eardrum image collection unit 111 is provided in the lighting device so that the user can measure the temperature of the ear endoscope imaging device during ear endoscopy, and the measured temperature is set to the preset temperature. Whether it is out of range, whether the temperature data collected from the temperature sensor shows a periodically repeating pattern within the preset section size, and measuring the humidity around the ear endoscope imaging device of the undiagnosed eardrum image collection unit 111 A humidity sensor is further provided to determine whether the measured humidity is outside the preset range, and the operation time and non-operation of the temperature sensor or humidity sensor under the condition that the temperature sensor or humidity sensor operates only when the lighting device is turned on during ear endoscopy. By analyzing the ratio of time, it is determined whether the ratio is outside the preset ratio range. If at least one of the above conditions is satisfied, it is determined that a malfunction has occurred in the ear endoscope imaging device, and the eardrum image captured by the device is determined. It may be determined that an error has occurred in the data.

Next, when performing fault diagnosis by determining whether there is an error in the temperature sensor provided in the ear endoscope imaging device, it is possible to monitor whether the temperature sensor shows a pattern that is periodically repeated within a preset section size, which is The technical principle is that when a foreign substance enters the ear endoscope imaging device of the undiagnosed eardrum image collection unit 111, the temperature measurement value of the temperature sensor can be repeatedly output in a specific pattern within a preset section size due to the injected foreign substance. is used. At this time, the preset section size can be determined according to [Equation 2] below.

[Equation 2]

A _range = {(T _aver + D _max ) - (T _aver - D _max )}*0.3

A _range is the preset section size, T _aver is the overall average during the preset period, and D _max means the maximum deviation during the preset period.

For example, if the overall average during the preset period is 70 and the maximum deviation is 20, the A _range is 12, so periodically repeating values are output in the range where the difference between the maximum and minimum values does not exceed 12 (ex. 52, 62 , 52, 62, 53, 62, 52, 63, etc.), it can be assumed that a foreign substance has entered the ear endoscope imaging device of the undiagnosed eardrum image collection unit 111 and an error has occurred in the temperature sensor. You can.

Next, in order to measure the humidity of the undiagnosed eardrum image collection unit 111 and determine whether the measured humidity exceeds a preset value, a humidity sensor is installed within a preset distance from the undiagnosed eardrum image collection unit 111. is provided separately, and the humidity of the undiagnosed eardrum image collection unit 111 can be monitored in real time through the humidity sensor. This uses the technical principle that if moisture flows into the undiagnosed eardrum image collection unit 111, it operates abnormally. If the humidity exceeds a preset value, it is considered that moisture has flowed into the undiagnosed eardrum image collection unit 111. You can guess. Therefore, in this case, it can be determined that an error has occurred in the undiagnosed eardrum image collection unit 111, which can be inferred to have occurred in the captured undiagnosed eardrum image data.

Finally, by analyzing the ratio of the operating time and non-operating time of the ear endoscope imaging device of the undiagnosed eardrum image collection unit 111, it is possible to determine whether the ratio is outside the preset ratio range, and the results are Accordingly, it may be determined that an error has occurred in the eardrum image data.

For example, the point at which the undiagnosed eardrum image collection unit 111 installed in the field cannot be detected from the hook by providing a proximity sensor to the hook where the ear endoscope imaging device is located by the user (videographer or doctor, etc.) to capture the eardrum image. When the detection point was recognized as the operating time and the point of detection as the non-operating time, the ratio of the average daily operating time to the non-operating time for at least one month was recorded as 10:1 to 9:1, but as a result of analysis through real-time monitoring, If the ratio is reversed from 1:10 to 1:9 or changes to a significantly different ratio (ex. 5:5, etc.), there may be a malfunction of the ear endoscope imaging device or an error in the proximity sensor located on the hook of the ear endoscope imaging device. It can be judged that it has occurred.

As described above, in one embodiment of the present invention, the fault diagnosis unit (not shown) can determine whether there is an error in the ear endoscopic imaging device and the sensor provided in the ear endoscopic imaging device by considering all of the multiple fault diagnosis methods, The final judgment may be made as the S _total value that reflects all of the multiple fault diagnosis methods as shown in [Equation 3] below.

[Equation 3]

S _total = W1*R _aver + W2*R _tem + W3*R _rep + W4*R _hum + W5*R _rat

Here, S _total is the sum _of multiple error judgment values, and R _aver is the error judgment value ( _error A value between 0 and 1 depending on whether the temperature is outside the preset range), R _tem is a value that determines whether there is an error depending on whether the temperature is outside the preset range (a value between 0 and 1 depending on whether there is an error), R _rep is a value that determines whether there is an error depending on whether a repeating pattern appears (a value between 0 and 1 depending on whether there is an error), and R _hum is a value that determines whether there is an error depending on whether the humidity is outside the preset range (error or not). one of 0 and 1 depending on), R _rat is the value determined for error by analyzing the ratio of operating time and non-operating time (either 0 or 1 depending on whether there is an error), W1 is the R _aver item weight, W2 is the R _tem item weight, and W3 is R. _Rep item weight, W4 means R _hum item weight, and W5 means R _rat item weight, respectively.

For example, if S _total is 4 or more, it can be preset to indicate that there is an error in the ear endoscope imaging device. If the fever temperature is a very important parameter, W2 can be preset to 3, and W1, W3, W4, and W5 are all set to 3. Can be preset to 1.

Under these conditions, as a result of monitoring errors according to the above multiple fault diagnosis methods, if R _aver is 1, R _tem is 1, R _rep is 0, R _hum is 0, and R _rat is 0, then S _total is 4. Therefore, it can be finally determined that there is an error in the ear endoscope imaging device.

Referring again to FIG. 1, the label generator 120 may generate a label with the disease name and hearing value corresponding to the diagnostic eardrum image classified by the eardrum imaging unit 110.

Here, the label means 'answer' or 'result' for diagnosing the undiagnosed eardrum image, and defines the disease name and hearing value corresponding to each diagnostic eardrum image collected from outside and for which diagnosis has been completed as one correct answer group. It can mean something.

At this time, in the case of the undiagnosed eardrum image, since it has not been diagnosed, it may be excluded from the target for creating the corresponding label.

For example, the first diagnostic eardrum image 510 may be created with a label having the disease name and hearing value of 'hearing loss' and '51dBHL'.

Meanwhile, the label generator 120 may further generate an area where a user or an eardrum classification worker divides the eardrum area in the eardrum image as an eardrum area label.

Additionally, the label generation unit 120 may generate the diagnostic eardrum vector converted into a vector in the vector conversion unit 140, the corresponding disease name, and hearing level as a label.

The eardrum area extractor 130 may recognize the eardrum area in the eardrum image and extract it as a preset eardrum area.

Here, the eardrum area extractor 130 will be described in more detail with reference to FIG. 3 .

Referring to FIG. 3, the eardrum area extractor 130 may include an eardrum recognition unit 131, a region of interest extractor 132, and an eardrum central area extractor 133.

Additionally, the undisturbed eardrum area may mean a region of interest extracted by the region of interest extractor 132 or a central area of the eardrum extracted by the eardrum central area extractor 133 with respect to the eardrum image.

The eardrum recognition unit 131 may recognize the eardrum area in the eardrum image collected by the eardrum imaging unit 110.

Here, the eardrum image may mean an eardrum image including both a non-diagnosed eardrum image and a diagnostic eardrum image.

Meanwhile, the eardrum recognition unit 131 performs deep learning based on area information labeled by dividing the diagnostic eardrum images collected by the eardrum imaging unit 110 into eardrum areas in the label generator 120. ) After recognizing and learning the eardrum area object using a convolutional neural network (CNN), one of the algorithms, the eardrum area included in the undiagnosed eardrum image can be recognized.

The region of interest extraction unit 132 recognizes the eardrum area in the eardrum image collected by the eardrum imaging unit 131, and creates a circular area extracted from the eardrum area using Hough Circle Transform processing technology. It can be extracted by Region Of Interest.

More specifically, the region of interest extractor 132 processes the eardrum area recognized by the eardrum recognition unit 131 into a circle using the pixel value corresponding to the eardrum area through Hough Circle Transform processing. The center coordinates and radius information can be detected.

Through this, the region of interest extractor 132 can extract the detected circle as a region of interest at a faster speed and more accurate detection rate.

In addition, the RGB color code value for each pixel in the entire image of the eardrum image collected through the eardrum imaging unit 110 is calculated, and the RGB color code value of three or more pixels among the four pixels adjacent to each pixel is preset. If it is included within the scope, it may be judged to be subject to deletion.

Here, the preset range of the RGB color code value in the eardrum image is 0 to 10 for red (R), 0 to 10 for green (G), and 0 to 10 for blue (B) for the RGB color code value of each pixel. This may mean that the value is measured in the range of 0 to 10.

For example, the eardrum image was collected as a square-sized image based on the center, but the RGB color code value of the pixel was calculated from the end corner of a specific location based on the center to (0, 2, 7), (2). , 0, 5), (3, 3, 10), (10, 10, 10), (13, 20, 5), (80, 0, 53), the RGB color code value is calculated Among the pixels, the color code value of any one of red, green, and blue does not exceed 10 (0, 2, 7), (2, 0 5), (3, 3, 10), (10, 10, 10). The corresponding pixel can be determined as a target for deletion. Here, if three or more pixels among the four pixels adjacent to the pixel classified as the pixel to be deleted are target pixels, a method of automatically deleting them and extracting them as a region of interest can be used.

The eardrum center area extractor 133 may extract a preset area as the eardrum center area based on the eardrum location within the region of interest extracted by the interest region extractor 132.

Here, the preset area refers to an area set to a preset size based on the position of the eardrum. The size of the area may vary depending on the user's settings, and the maximum horizontal pixel number is 70px or less based on the number of pixels. The maximum number of vertical pixels can be set to 80px or less.

At this time, if the eardrum area is larger than the size of the area set by the user, the minimum size that fits the recognized eardrum area may be automatically calculated and set as the central area of the eardrum.

That is, the eardrum central region extractor 133 may extract a smaller eardrum central region centered on the eardrum region from the region of interest extracted by the region of interest extractor 132.

Here, the central area of the eardrum can be extracted as a generally square-shaped image.

The process by which the eardrum region extractor 130 extracts the central region of the eardrum using the diagnostic eardrum image and the non-diagnosed eardrum image from the eardrum imaging unit 110 will be described in more detail with reference to FIG. 4 .

Referring to FIG. 4, the eardrum area extraction unit 130 may recognize the eardrum area in the eardrum image 410 collected by the eardrum imaging unit 110 using the eardrum recognition unit 131. At this time, a circular area with a radius of a certain number of pixels is extracted from the center of the area recognized as the eardrum through the region of interest extractor 132, or each RGB color code value in the eardrum image 410 is 10 for all three color codes. The region of interest image 420 within the eardrum image 410 can be extracted by excluding the following region. Afterwards, the eardrum center area extractor 133 generates a size corresponding to the number of pixels set by the user with a maximum horizontal pixel count of 70px and a vertical maximum pixel count of 80px based on the eardrum area recognized by the eardrum recognition unit 131. The eardrum center area is extracted from a square image, but if the number of pixels set by the user is too small and the recognized eardrum area exceeds the size, the eardrum center area 430 is converted to a rectangular area of the minimum size containing the recognized eardrum area. ) can be extracted.

Referring again to FIG. 1, the vector conversion unit 140 may convert the eardrum region extracted by the eardrum region extractor 130 into an undiagnosed eardrum vector using global average pooling.

Here, the vector conversion method using global average pooling will be described in more detail with reference to FIG. 6.

Referring to FIG. 6, global average pooling is one of the techniques for converting the output value of the last layer in a CNN (Convolutional Neural Network) into vector form, and conversion can be performed without flattening when converting the output value vector.

Through this, global average pooling can classify and convert the extracted eardrum area into an undiagnosed eardrum vector by converting it into vector form while preserving the spatial information of the output value.

That is, the vector conversion unit 140 can average all the characteristics of the eardrum area channel extracted by the eardrum area extractor 130 and compress them into a one-dimensional vector.

At this time, the area converted into a vector by the vector conversion unit 140 may mean an undiagnosed central area of the eardrum among the central areas of the eardrum extracted by the eardrum area extractor 130.

Here, the vector conversion unit 140 converts the image of the patient's eardrum, which is the central area of the undiagnosed eardrum, into a one-dimensional vector and extracts the average value for each channel to prevent overfitting when diagnosing a disease and converts the parameters ( Parameters) can be reduced to lead to faster calculation performance.

Meanwhile, the vector conversion unit 140 can convert the diagnostic eardrum center area, which is the eardrum center area, from the diagnostic eardrum image into a vector.

The disease diagnosis unit 150 can perform disease diagnosis of the undiagnosed tympanic membrane vector converted in the vector conversion unit 140 using a Softmax activation function and Monte Carlo Dropout. there is.

More specifically, the disease diagnosis unit 150 normalizes the value of the undiagnosed eardrum vector converted and input from the vector conversion unit 140 so that the total is 1 using a softmax activation function, and outputs the probability. It can be expressed as

At this time, disease diagnosis can be performed according to the disease name with the highest probability by comparing the normalized output undiagnosed eardrum vector with the eardrum disease labels corresponding to the plurality of diagnostic eardrum vectors generated by the label generator 120.

Meanwhile, the disease diagnosis unit 150 may calculate each hearing value of the disease label corresponding to the undiagnosed eardrum vector for which disease diagnosis was performed using the Monte Carlo dropout technique.

Here, the Monte Carlo dropout technique may refer to a technique of inferring a prediction distribution using a Bayesian neural network to estimate uncertainty about the output calculated through the softmax function.

In particular, the disease diagnosis unit 150 uses the Monte Carlo dropout technique to prevent overfitting for calculating the disease name in the output value where the uncertainty of the undiagnosed eardrum vector is large through the softmax function and reflects the uncertainty. Performance improvement in disease determination can be induced through the ensemble effect.

Through the above process, the eardrum disease prediction model system using the deep neural network and Monte Carlo dropout collects diagnostic eardrum images and undiagnosed eardrum images. In the case of diagnostic eardrum images, the corresponding diagnosis disease name and hearing test performed at the time of diagnosis Results can be further collected and generated as labels on diagnostic eardrum images. In addition, the eardrum area is extracted by recognizing the eardrum area in the undiagnosed eardrum image collected from the patient, and the extracted eardrum area is converted into a vector and softmax activation function and Monte Carlo dropout are used to diagnose and diagnose diseases of the undiagnosed eardrum. Hearing values can be calculated and provided as supplementary data to medical professionals.

According to one embodiment of the present invention, the presence and type of disease can be determined by analyzing images inside the patient's ear taken using an endoscope through machine learning.

In addition, by analyzing the sensing data collected through various types of sensors in the ear endoscopic imaging device that photographs the inside of the patient's ear to diagnose failure, the reliability of the image data can be improved by distinguishing between images collected from imaging devices that have been determined to be malfunctioning. It can be maintained.

Additionally, by using Monte Carlo dropout when performing machine learning, it is possible to output more generalized results by creating an ensemble effect that increases accuracy by using multiple models.

As described above, although one embodiment of the present invention has been described with limited examples and drawings, one embodiment of the present invention is not limited to the above-described embodiment, which is based on common knowledge in the field to which the present invention pertains. Anyone who has the knowledge can make various modifications and variations from this description. Accordingly, one embodiment of the present invention should be understood only by the scope of the claims set forth below, and all equivalent or equivalent modifications thereof shall fall within the scope of the spirit of the present invention.

100: Eardrum disease prediction model system using deep neural network and Monte Carlo dropout
110: eardrum imaging unit
111: Undiagnosed eardrum image collection unit
112: Diagnostic eardrum image collection unit
113: Diagnostic eardrum image classification unit
120: label creation unit
130: eardrum area extraction unit
131: eardrum recognition unit
132: Region of interest extraction unit
133: Eardrum central area extraction unit
140: Vector conversion unit
150: Disease Diagnosis Department
410: Eardrum image
420: Region of interest image
430: Tympanic membrane central area
510: First diagnostic tympanic membrane image
520: Second diagnostic eardrum image
530: Third diagnostic tympanic membrane imaging

Claims (5)

an eardrum imaging unit that collects and classifies eardrum images;
a label generation unit that generates a disease name and hearing value corresponding to the eardrum image classified by the eardrum image unit as a label;
an eardrum area extraction unit that recognizes the eardrum area in the eardrum image and extracts it as a preset eardrum area;
a vector conversion unit that converts the eardrum region extracted by the eardrum region extraction unit in response to an undiagnosed eardrum image for which diagnosis has not been performed among the eardrum images into an undiagnosed eardrum vector using global average pooling;
a disease diagnosis unit that performs disease diagnosis on the undiagnosed eardrum vector converted by the vector conversion unit using a Softmax activation function and Monte Carlo dropout;
Including,
The eardrum imaging unit,
a histogram generator that generates image histogram distribution data of an eardrum image collected using an ear endoscope imaging device; and
Reliability is determined by analyzing the distribution data of the image histogram of the eardrum image generated by the histogram generator, but when the average value (A _err ) calculated by [Equation 1] below is greater than the preset threshold value (S _err ) , a fault diagnosis unit that determines that an error has occurred in the ear endoscope imaging device and the eardrum image;
An eardrum disease prediction model system using a deep neural network and Monte Carlo dropout, further comprising:
[Equation 1]

(Here, A _err is the average error of the image histogram distribution data of the eardrum image, T _aver is the overall average of the image histogram distribution data of the eardrum image, and P _aver is the partial average of the histogram distribution data for the n eardrum image images. , T _σ means the overall standard deviation of the image histogram distribution data of the eardrum image)
According to paragraph 1,
The eardrum imaging unit,
An undiagnosed eardrum image collection unit that collects images of the patient's undiagnosed eardrum captured using an ear endoscope imaging device;
A diagnostic eardrum image collection unit that collects a diagnostic eardrum image, which is an ear endoscopic image of a patient whose diagnosis has been completed from the outside, the name of the disease, and hearing values; and
a diagnostic eardrum image classification unit that classifies disease names and hearing values corresponding to the diagnostic eardrum images;
An eardrum disease prediction model system using a deep neural network and Monte Carlo dropout, characterized by including.
delete According to paragraph 1,
The eardrum region extraction unit,
Region of interest extraction by recognizing the eardrum area in the eardrum image collected by the eardrum imaging unit and extracting the circular area extracted from the eardrum area using Hough Circle Transform processing technology as a region of interest. wealth; and
an eardrum center area extractor that extracts a preset area as the eardrum center area based on the eardrum position within the region of interest;
An eardrum disease prediction model system using a deep neural network and Monte Carlo dropout, characterized by including.
According to paragraph 1,
The disease diagnosis department,
Performing disease diagnosis corresponding to a plurality of disease labels generated by the label generation unit using the softmax activation function on the undiagnosed eardrum vector,
An eardrum disease prediction model system using a deep neural network and Monte Carlo dropout, characterized in that each hearing value of the disease label corresponding to the undiagnosed eardrum vector for which a disease diagnosis was performed is calculated using the Monte Carlo dropout technique.