KR102220573B1 - Method, apparatus and computer program for calculating quality score of fundus image data using artificial intelligence - Google Patents

Abstract

Provided are a method, a device and a computer program for calculating a quality score of fundus oculi image data using artificial intelligence. According to various embodiments of the present invention, the method for calculating quality score of fundus oculi image data, which is performed by a computing device, includes the steps of: collecting fundus oculi image data; extracting one or more feature values from the fundus oculi image data; calculating a probability that the fundus oculi image data belongs to each of a plurality of preset quality grades by using the extracted one or more feature values; and calculating a quality score for the fundus oculi image data by using the probability of belonging to each of the plurality of quality grades calculated.

Description

Method for calculating the quality score of fundus image data using artificial intelligence, device and computer program {METHOD, APPARATUS AND COMPUTER PROGRAM FOR CALCULATING QUALITY SCORE OF FUNDUS IMAGE DATA USING ARTIFICIAL INTELLIGENCE}

Various embodiments of the present invention relate to a method, apparatus, and computer program for calculating a quality score of fundus image data using artificial intelligence.

The age of onset of major eye diseases is gradually increasing, and the number of people with red signals on eye health due to chronic diseases such as high blood pressure and diabetes is increasing. However, unlike major endoscopy, including national examinations, ophthalmologists often do not know which tests to take to protect eye health.

In fact, according to an eye health awareness survey of 1,000 men and women aged 25 years and over nationwide, 76.5% of the subjects were interested in eye health, and 63.5% knew that regular checkups were needed, but 11.3 %.

According to the Korean Academy of Ophthalmology, the first step to maintaining eye health is a “fungus examination”. This is because it is the way to detect macular degeneration, diabetic retinopathy, and glaucoma, which are one of the three major causes of blindness. In this regard, the Korean Ophthalmological Association has set October 10 as the day of the eye and is continuing efforts to include a fundus examination as part of the national examination under the theme of'fundament examination, the beginning of eye health.'

Fundus examination (or fundus imaging examination) is a fundus camera to diagnose and early detect fundus-related diseases by photographing the center of the retina (e.g., the back of the eye and the bottom of the eye) through the pupil and checking the color and thickness of the retinal vessels or optic nerve It is a test performed for.

In general, the quality of the fundus image data acquired in the fundus examination varies greatly depending on the type of the photographing camera and the photographing environment. In particular, it is easy to acquire image data of low quality from images of smart phone-based portable devices.

On the other hand, image data that is difficult to read is selected (e.g., scoring fundus image data) because image data of low quality can have a negative effect on analysis and reading by doctors or computer-aided diagnosis (CAD) systems and artificial intelligence. ) To select the fundus image data of low quality), and screening the selected fundus image data in advance is required.

Korean Patent Publication No. 10-2019-0087272 (2019.07.24)

The problem to be solved by the present invention is a method of calculating the quality score of fundus image data using artificial intelligence that selects fundus image data of low quality and calculates the quality score of fundus image data in order to screen the selected fundus image data in advance , To provide devices and computer programs.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

A method for calculating a quality score of fundus image data according to an embodiment of the present invention for solving the above-described problem, in a method performed by a computing device, comprises: collecting fundus image data, at least one from the fundus image data Extracting a feature value, calculating a probability that the fundus image data belongs to each of a plurality of preset quality classes using the extracted one or more feature values, and using the calculated probability of belonging to each of the plurality of quality classes Thus, it may include calculating a quality score for the fundus image data.

In various embodiments, the plurality of quality grades include a first grade, a second grade lower than the first grade, and a third grade lower than the second grade, and calculating the quality score includes the following It may include calculating a quality score for the fundus image data using Equation 1.

[Equation 1]

Here, prob(G ₁ ) is the probability that the fundus image data belongs to the first grade, prob(G ₂ ) is the probability that the fundus image data belongs to the second grade, and prob(G ₃ ) is the fundus image data Probabilities of falling into the third grade, C ₁ , C _2, and C ₃ may be preset constant values.

In various embodiments, the one or more feature values include a probability that the fundus image data belongs to an abnormal grade, a quality map of the fundus image data, a brightness, a focus value, and an illumination histogram. histogram) and roundness, and calculating the probability of belonging to each of the plurality of quality grades includes the one of the plurality of fundus image data using an artificial intelligence model that has previously been learned as training data. It may include calculating a probability that the fundus image data belongs to each of the plurality of quality classes from the above feature values.

In various embodiments, the extracting of the one or more feature values may include using a first binary classification model that classifies the first class and the second class as a normal class, and classifies the third class as an abnormal class. Calculating a first abnormal grade probability that is a probability that the fundus image data belongs to the abnormal grade, extracting a first quality map visualizing an area determined to be abnormal in the fundus image data, and determining the first grade as a normal grade And calculating a second abnormal grade probability, which is a probability that the fundus image data belongs to the abnormal grade, using a second binary classification model that classifies the second grade and the third grade as abnormal grade, and the fundus image It may include the step of extracting a second quality map visualizing the area determined to be abnormal in the data.

In various embodiments, the extracting of the one or more feature values includes detecting an area corresponding to a disc of the fundus from the fundus image data, and the length and shortening of the major axis of the area corresponding to the detected nipple of the fundus. Extracting a length, and calculating a roundness of the fundus using a ratio of the extracted length of the major axis and the length of the extracted minor axis.

In various embodiments, the sharpness includes a first sharpness and a second sharpness, and the extracting of the one or more feature values may include calculating the first sharpness for the fundus image data using Equation 2 below. And calculating the second sharpness of the fundus image data using Equation 3 below.

[Equation 2]

, grayscale(X)는

으로 상기 안저 영상 데이터의 그레이 스케일 값, X_R는 상기 안저 영상 데이터의 R채널 값, X_G는 상기 안저 영상 데이터의 G채널 값, X_B는 상기 안저 영상 데이터의 B채널 값 및 mean(X)는

일 수 있다.Here, X is the fundus image data value, std(X) is

, grayscale(X) is

As a gray scale value of the fundus image data _{, X R} is an R channel value of the fundus image data _{, X G is a G channel value of the fundus image data, X B} is a B channel value and mean(X) of the fundus image data. Is

Can be

[Equation 3]

In various embodiments, the extracting the one or more feature values may include calculating a focus value for the fundus image data using Equation 4 below.

[Equation 4]

, grayscale(X)는

으로 상기 안저 영상 데이터의 그레이 스케일 값, X_R는 상기 안저 영상 데이터의 R채널 값, X_G는 상기 안저 영상 데이터의 G채널 값, X_B는 상기 안저 영상 데이터의 B채널 값, kernel_x는

으로 x방향의 3x3 커널, kernel_y는

으로 y방향의 3x3 커널일 수 있다.Here, X is the fundus image data value, std(X) is

, grayscale(X) is

As a gray scale value of the fundus image data _{, X R} is an R channel value of the fundus image data _{, X G is a G channel value of the fundus image data, X B} is a B channel value of the fundus image data, and kernel _x is

As a 3x3 kernel in the x direction, kernel _y is

It may be a 3x3 kernel in the y direction.

In various embodiments, the extracting of the one or more feature values includes pre-processing the fundus image data, extracting one or more patches having a preset size from the pre-processed fundus image data, the following Computing a representative value for each of the one or more patches using Equation 5, generating an illuminance map using the calculated representative value, and generating the illuminance histogram using the illuminance map can do.

[Equation 5]

일 수 있다.Here, P is the one or more patch values, P _R is the R channel value _{of the one or more patches, P- G} is the G channel value of the one or more patches, P _B is the B channel value and mean(x ) Is

Can be

An apparatus for calculating a quality score of fundus image data according to another embodiment of the present invention for solving the above-described problem includes a memory storing one or more instructions, and a processor executing the one or more instructions stored in the memory, the The processor may perform the method of calculating a quality score of fundus image data according to an embodiment of the present invention by executing the one or more instructions.

A computer program according to another embodiment of the present invention for solving the above-described problem is combined with a computer that is hardware, so that the computer can perform the method of calculating the quality score of the fundus image data according to an embodiment of the present invention. It can be stored on a readable recording medium.

Other specific details of the present invention are included in the detailed description and drawings.

According to various embodiments of the present invention, one or more feature values are extracted from fundus image data, and a probability that the fundus image data belongs to each of a plurality of preset quality classes is calculated using the extracted one or more feature values, and the calculated probability There is an advantage in that the quality of the fundus image data can be more accurately determined by calculating a quality score for the fundus image data based on.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating a system for calculating a quality score of fundus image data using artificial intelligence according to an embodiment of the present invention.
2 is a diagram illustrating an apparatus 100 for calculating a quality score of fundus image data according to another embodiment of the present invention.
3 is a flowchart of a method of calculating a quality score of fundus image data according to another embodiment of the present invention.
4 is a diagram illustrating a quality map generated by an apparatus for calculating a quality score of fundus image data in various embodiments.
5 is a diagram illustrating a result of detecting a disc of a fundus from fundus image data by an apparatus for calculating a quality score of fundus image data, according to various embodiments.
6 is a flowchart of a method of generating an illuminance histogram performed by an apparatus for calculating a quality score of fundus image data in various embodiments.
7 and 8 are tables exemplarily illustrating fundus image data sets used as training data of a first binary classification model and a second binary classification model in various embodiments.
9 is a diagram illustrating a sample of training data of a semantic partitioning model in various embodiments.
10 is a diagram exemplarily illustrating normal and abnormal data samples for each feature in various embodiments.
11 is a diagram illustrating an illuminance map generated by an apparatus for calculating a quality score of fundus image data according to various embodiments.
12 is a diagram illustrating an illuminance histogram generated by an apparatus for calculating a quality score of fundus image data according to various embodiments.
13 is a diagram illustrating fundus image data according to a quality score in various embodiments.
14 is a flowchart of a method of calculating a quality score threshold for filtering image data according to another embodiment of the present invention.
15 is a diagram illustrating an example of a connection element labeling result according to a Q quartile, in various embodiments.
16 is a hardware configuration diagram of an apparatus for calculating a quality score of fundus image data according to another embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

The term "unit" or "module" used in the specification refers to a hardware component such as software, FPGA or ASIC, and the "unit" or "module" performs certain roles. However, "unit" or "module" is not meant to be limited to software or hardware. The “unit” or “module” may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors. Thus, as an example, "sub" or "module" refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, It includes procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables. Components and functions provided within "sub" or "module" may be combined into a smaller number of components and "sub" or "modules" or into additional components and "sub" or "modules". Can be further separated.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc., as shown in the figure It can be used to easily describe the correlation between a component and other components. Spatially relative terms should be understood as terms including different directions of components during use or operation in addition to the directions shown in the drawings. For example, if a component shown in a drawing is turned over, a component described as "below" or "beneath" of another component will be placed "above" the other component. I can. Accordingly, the exemplary term “below” may include both directions below and above. Components may be oriented in other directions, and thus spatially relative terms may be interpreted according to orientation.

In the present specification, a computer refers to all kinds of hardware devices including at least one processor, and may be understood as encompassing a software configuration operating in a corresponding hardware device according to embodiments. For example, the computer may be understood as including all of a smartphone, a tablet PC, a desktop, a laptop, and a user client and an application running on each device, but is not limited thereto.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Each of the steps described herein is described as being performed by a computer, but the subject of each step is not limited thereto, and at least some of the steps may be performed by different devices according to embodiments.

In addition, in the present specification, the description is based on calculating the quality score for the fundus image data by analyzing the fundus image data generated by photographing the user's fundus, but is not limited thereto, and image data generated by photographing various objects It can be applied to any field that analyzes and calculates a quality score for image data.

1 is a diagram illustrating a system for calculating a quality score of fundus image data using artificial intelligence according to an embodiment of the present invention.

Referring to FIG. 1, a system for calculating a quality score of fundus image data using artificial intelligence according to an embodiment of the present invention includes an apparatus 100 for calculating a quality score of fundus image data, a user terminal 200 and an external server 300. It may include.

Here, the system for calculating the quality score of the fundus image data using artificial intelligence shown in FIG. 1 is according to an embodiment, and its components are not limited to the embodiment shown in FIG. 1, and are added or changed as necessary. Or it can be deleted.

In one embodiment, the apparatus 100 for calculating the quality score of the fundus image data (hereinafter, "computing device 100") is provided with a fundus from the outside (eg, a fundus camera module, a user terminal 200 and an external server 300). Image data may be collected, and a quality score for fundus image data may be calculated by analyzing the collected fundus image data.

In various embodiments, the computing device 100 may extract one or more feature values from the fundus image data using an artificial intelligence model, and each of a plurality of preset quality levels for the fundus image data using the extracted one or more feature values. It is possible to calculate the probability of belonging to, and by using the probability of belonging to each of the plurality of quality classes calculated, a quality score for the fundus image data may be calculated. However, it is not limited thereto.

In various embodiments, the computing device 100 may analyze the image data and determine a quality score threshold for filtering the image data. For example, the computing device 100 classifies a plurality of image data into a plurality of quality classes using quality scores for each of the plurality of image data, and determines disease classification performance for image data included in each class. A plurality of quality grades may be clustered into two groups, and a quality score threshold value of image data may be calculated using a quality score boundary value between the clustered two groups. However, it is not limited thereto.

In various embodiments, the computing device 100 may filter the fundus image data based on the quality score of the fundus image data. For example, when the quality score of the fundus image data is less than a preset reference score (eg, a quality score threshold), the computing device 100 may filter the fundus image data and exclude it from the subject of diagnosis reading. However, it is not limited thereto.

In various embodiments, the computing device 100 may guide the re-taking of the fundus image data based on the quality score of the fundus image data. For example, when the quality score of the fundus image data is less than a preset reference score (eg, a quality score threshold), the computing device 100 reproduces the fundus image to the user terminal 200 of the user who uploaded the fundus image data. A guide message instructing you to shoot and upload can be output. However, it is not limited thereto. Here, when the computing device 100 obtains a user input from the user terminal 200 indicating that the fundus image data is not re-uploaded or re-upload is impossible within a preset time, the fundus image data is filtered and Can be excluded. However, it is not limited thereto.

In various embodiments, when the quality score of the fundus image data is equal to or higher than a preset reference score (eg, a quality score threshold), the computing device 100 analyzes the fundus image data to diagnose and predict a fundus-related disease. May be performed, and a result of performing diagnostic reading may be provided to the user terminal 200. For example, the computing device 100 diagnoses whether there is a fundus-related disease by analyzing fundus image data using a previously learned artificial intelligence model, and writes the diagnosis result in the form of a diagnosis result report to the user terminal 200 ) Can be provided. However, it is not limited thereto.

In one embodiment, the user terminal 200 may be connected to the computing device 100 through the network 400, provide fundus image data to the computing device 100, or provide various data from the computing device 100 (e.g. : You can receive a diagnosis result report through analysis of fundus image data).

In various embodiments, the user terminal 200 may include a smartphone, a tablet PC, a notebook, and a desktop having a display on at least a portion of the user terminal 200, and provided from the computing device 100 through the display. A user interface (UI) (eg, a fundus image data upload UI, a diagnostic result report UI, etc.) can be output.

In various embodiments, the user terminal 200 may separately include a camera module for generating image data therein, or may be connected to an external camera device to receive image data generated from an external camera device. However, it is not limited thereto.

In one embodiment, the external server 300 may be connected to the computing device 100 through the network 400, and various data necessary for the computing device 100 to perform a process of calculating the quality score of the fundus image data (Eg, fundus image data) may be provided, or various data generated from the computing device 100 (eg, a result of calculating a quality score of fundus image data) may be provided and stored. For example, the external server 300 may be a hospital server that stores fundus image data for each of a plurality of patients. However, it is not limited thereto. Hereinafter, with reference to FIGS. 2 and 3, a method of calculating a quality score of fundus image data performed by the computing device 100 will be described.

2 is a diagram showing an apparatus 100 for calculating a quality score of fundus image data according to another embodiment of the present invention, and FIG. 3 is a flowchart of a method of calculating a quality score of fundus image data according to another embodiment of the present invention to be.

2 and 3, in step S110, the computing device 100 may collect fundus image data from the outside. For example, the computing device 100 may receive fundus image data (eg, image data and video data) generated by photographing a user's fundus from a fundus camera module (not shown) in real time.

In various embodiments, the computing device 100 is connected to the user terminal 200 or the external server 300, and at least one fundus image from among a plurality of fundus image data previously stored in the user terminal 200 or the external server 300 Data can be provided. However, it is not limited thereto.

In step S120, the computing device 100 may extract one or more feature values from the fundus image data by analyzing the fundus image data collected in step S110. For example, the computing device 100 may extract one or more feature values by analyzing the fundus image data collected in step S110 using the feature extraction module 110.

Here, the feature extraction module 110 uses an artificial intelligence model that has previously learned a plurality of fundus image data as training data, the probability that the fundus image data belongs to the abnormal grade, the quality map of the fundus image data, and the sharpness ( Brightness), a focus value (Focus), an illumination histogram (Illumination histogram), and a roundness (Roundness) may be a module for extracting a feature value including at least one of. However, the present invention is not limited thereto, and any feature value required to calculate a quality score for fundus image data may be applied.

In various embodiments, the computing device 100 is operated by the feature extraction module 110, and two binary classification models (e.g., normal and abnormal) classify fundus image data according to different criteria. : An abnormal grade probability for fundus image data may be calculated using the first binary classification model 111 and the second binary classification model 112).

Also, the computing device 100 may generate a quality map visualizing an abnormal area of fundus image data using two binary classification models.

Here, the binary classification model includes a plurality of preset quality grades (e.g., a first grade (e.g. good grade), a second grade lower than the first grade (e.g., usable grade)), and a third grade lower than the second grade (e.g. : Reject grade)), a plurality of fundus image data (eg, Figs. 7 and 8) classified according to the training data may be pre-trained artificial intelligence models. For example, the binary classification model may use a fundus image data set (for example, FIG. 7), and 90% (11288 pieces) is obtained by dividing the training data set among a plurality of fundus image data included in the fundus image data set. And the remaining 10% (1255) can be used as verification data for verifying the binary classification model. In addition, taking into account that the data structure is different according to the type of the binary classification model (the first binary classification model 111 and the second binary classification model 112), sampling so that the number of data for each binary label is the same (for example, FIG. 8 )can do. However, it is not limited thereto.

In various embodiments, the computing device 100 uses the first binary classification model 111 to classify the first class and the second class as a normal class and classify the third class as an abnormal class, so that the fundus image data is an abnormal class. (Example: The probability of belonging to the third grade), which is the probability of a first abnormal grade, can be calculated.

In addition, the computing device 100 may generate a first quality map (eg, FIG. 4) visualizing an abnormal patch included in the fundus image data using the first binary classification model 111. For example, the computing device 100 may generate a first quality map for visualizing an abnormal patch included in the fundus image data in red and visualizing a normal patch in blue. However, it is not limited thereto.

In various embodiments, the computing device 100 uses the second binary classification model 112 that classifies the first class as a normal class and classifies the second and third classes as an abnormal class, so that the fundus image data is an abnormal class. The second abnormal grade probability, which is the probability of belonging to the second grade or the third grade, can be calculated.

In addition, the computing device 100 may generate a second quality map (eg, FIG. 4) visualizing an abnormal patch included in the fundus image data using the second binary classification model 112. However, it is not limited thereto.

In various embodiments, the computing device 100 may calculate a roundness for a fundus disc (eg, an outer rim of the fundus) included in the fundus image data using the semantic segmentation model 113. .

Here, the semantic segmentation model 113 may be an artificial intelligence model in which various open data related to the fundus (eg, FIG. 9) are previously learned as training data. For example, the semantic partitioning model 113 can use 400 iChallenge-AMD data, 187 PALM Challenge data, and 800 REFUGE Challenge data as training data, and some (about 80) of 800 REFUGE Challenge data. It can be used as verification data for verifying the semantic segmentation model 113. However, it is not limited thereto.

In various embodiments, the computing device 100 may detect a region (eg, FIG. 5) corresponding to the nipple of the fundus included in the fundus image data using the semantic segmentation model 113, and the ellipse matching method (ellipse fitting) can be used to extract the major axis length (Minor_axis(Disc)) and minor axis length (Major_axis(Disc)) of the region corresponding to the detected fundus nipple, and the ratio of the extracted major axis length and minor axis length (e.g.: The roundness of the fundus can be calculated using Minor_axis(Disc)/Major_axis(Disc)).

In this case, when the length of the major axis of the nipple of the fundus is 0, the computing device 100 may calculate the roundness as 0. However, it is not limited thereto.

In various embodiments, the computing device 100 may calculate the sharpness of the fundus image data using the feature extraction model 114.

Here, the feature extraction model 114 extracts samples for each feature (eg, sharpness, focus value, and illuminance histogram) from a plurality of fundus image data (eg, FIG. 7) classified according to a plurality of preset quality classes. And, it may be an artificial intelligence model that has previously learned data (eg, FIG. 10) obtained by feature engineering the extracted sample as training data. However, it is not limited thereto.

In various embodiments, the feature extraction model 114 is hand crafted by an administrator who calculates the quality score for the fundus image data using the device for calculating the quality score of the fundus image data according to various embodiments of the present invention. It can be a model. For example, the feature extraction model 114 may be a model set according to a result derived through an experiment. However, it is not limited thereto.

In various embodiments, the computing device 100 uses the feature extraction model 114 to detect fundus image data that is difficult to accurately identify the fundus because it is too bright or too dark as a whole, so that the sharpness of the fundus image data (for example, the first Sharpness and second sharpness) can be calculated.

In various embodiments, the computing device 100 may calculate the first sharpness by using Equation 2 below, and may calculate the second sharpness by using Equation 3 below. However, it is not limited thereto.

, grayscale(X)는

으로 안저 영상 데이터의 그레이 스케일 값, X_R는 안저 영상 데이터의 R채널 값, X_G는 안저 영상 데이터의 G채널 값, X_B는 안저 영상 데이터의 B채널 값 및 mean(X)는

일 수 있다.Where X is the fundus image data value, and std(X) is

, grayscale(X) is

As the gray scale value of the fundus image data, X _R is the R channel value of the fundus image data _{, X G is the G channel value of the fundus image data, X B} is the B channel value and mean(X) of the fundus image data.

Can be

In various embodiments, the computing device 100 may calculate the focus value of the fundus image data using the feature extraction model 114. For example, the computing device 100 may calculate the focal value of the fundus image data by using the feature extraction model 114 to detect fundus image data that is difficult to identify small bleeding due to the blurred focus.

In various embodiments, the computing device 100 may calculate a focus value for fundus image data using Equation 4 below. However, it is not limited thereto.

, grayscale(X)는

으로 안저 영상 데이터의 그레이 스케일 값, X_R는 안저 영상 데이터의 R채널 값, X_G는 안저 영상 데이터의 G채널 값, X_B는 안저 영상 데이터의 B채널 값, , kernel_x는

으로 x방향의 3x3 커널(예: kernel_x=

), kernel_y는

으로 y방향의 3x3 커널(예: kernel_y=

)일 수 있다.Where X is the fundus image data value, and std(X) is

, grayscale(X) is

As the gray scale value of the fundus image data, X _R is the R channel value of the fundus image data _{, X G is the G channel value of the fundus image data, X B} is the B channel value of the fundus image data, and kernel _x is

3x3 kernel in the x direction (e.g. kernel _x =

), kernel _y is

3x3 kernel in y direction (e.g. kernel _y =

) Can be.

In various embodiments, the computing device 100 may generate an illuminance histogram of fundus image data using the feature extraction model 114. For example, the computing device 100 may generate an illuminance histogram of fundus image data by using the feature extraction model 114 to detect fundus image data having uneven brightness. Hereinafter, it will be described with reference to FIG. 6.

6 is a flowchart of a method of generating an illuminance histogram performed by an apparatus for calculating a quality score of fundus image data in various embodiments.

Referring to FIG. 6, in step S210, the computing device 100 may pre-process the fundus image data. For example, the computing device 100 may resize the fundus image data to a preset size (eg, 256 x 256). However, it is not limited thereto.

In step S220, the computing device 100 may extract one or more patches from fundus image data whose size has been converted to a preset size through step S210. For example, the computing device 100 may extract one or more patches while moving the pre-processed fundus image data according to a preset condition (eg, window size 32, stride 16). However, it is not limited thereto.

In step S230, the computing device 100 may calculate a representative value for each of the one or more patches extracted through the step S220. For example, the computing device 100 may calculate a representative value for each of one or more patches using Equation 5 below. However, it is not limited thereto.

일 수 있다.Here, P is one or more patch values, P _R is the R channel value _{of one or more patches, P- G} is the G channel value of one or more patches, P _B is the B channel value of one or more channels, and mean(x) is

Can be

In step S240, the computing device 100 may generate an illuminance map (eg, FIG. 11) using representative values of each of one or more patches calculated through step S230. For example, the computing device 100 may generate an illuminance map in the form of a matrix having a preset size (eg, 15x15) using representative values of each of one or more patches. However, it is not limited thereto.

In step S250, the computing device 100 may generate an illuminance histogram using the illuminance map generated in step S240. For example, the computing device 100 calculates pixel values in the range of 0 to 80 in 17 of the total 18 bins (eg, storage 1 to 17), and calculates the pixel value in the range of 0 to 80, and In the storage), an illuminance histogram (eg, FIG. 12) may be generated by calculating pixel values in the range of 80 to 255. However, it is not limited thereto.

In various embodiments, the computing device 100 may calculate an illuminance feature value using the illuminance histogram generated in step S250. For example, the computing device 100 may calculate the illuminance feature value by using the difference between the illuminance histogram of the normal region (e.g., Fig. 12(A)) and the illuminance histogram of the abnormal region (e.g. Fig. 12(B)). I can. However, it is not limited thereto.

Referring back to FIGS. 2 and 3, in step S130, the computing device 100 may calculate a probability that the fundus image data belongs to each of a plurality of preset quality classes by using one or more feature values extracted in step S120. For example, the computing device 100 uses the quality classification module 120 to determine one or more feature values (eg, a first abnormal grade probability, a first quality map, a second abnormal grade probability, a second quality) extracted in step S120. By analyzing map, roundness, first sharpness, second sharpness, focus value, illuminance histogram and illuminance feature value), the probability that fundus image data belongs to the first class (prob(G ₁ )) and fundus image data are determined. The probability of belonging to the second grade (prob(G ₂ )) and the probability that the fundus image data belong to the third grade (prob(G ₃ )) can be calculated.

In various embodiments, the computing device 100 may calculate a probability that the fundus image data belongs to each of a plurality of preset quality classes using a classification model (eg, a classification model) operated by the quality classification module 120. have.

Here, the classification model may be an artificial intelligence model previously learned by using one or more feature values extracted from the feature extraction module 110 as training data. For example, the classification model may extract one or more feature values from a training data set and a verification data set of a fundus image data set (eg, FIG. 7 ), and use the extracted one or more feature values as training data. At this time, 90% of one or more feature values extracted from the training data set may be used for training and the remaining 10% may be used for verification of the classification model. However, it is not limited thereto.

Thereafter, when new fundus image data is acquired, the computing device 100 extracts one or more of the above-described feature values from the fundus image data in order to evaluate the quality of the fundus image data, and this is a pre-learned artificial intelligence model. By inputting to, the probability that the fundus image data belongs to each of a plurality of preset quality classes may be calculated.

In step S140, the computing device 100 may calculate a quality score for the fundus image data using a probability that the fundus image data calculated in step S130 belongs to each of a plurality of preset quality classes. For example, the computing device 100 may calculate a quality score for fundus image data using the quality score calculation module 130 (eg, FIG. 13 ).

In various embodiments, the computing device 100 may calculate a quality score for the fundus image data using Equation 1 below.

Here, prob(G ₁ ) is the probability that the fundus image data belongs to the first grade, prob(G ₂ ) is the probability that the fundus image data belongs to the second grade, and prob(G ₃ ) is the fundus image data belongs to the third grade. The probability of belonging, C ₁ , C _2, and C ₃ may be preset constant values (eg, 2 for _{C 1 , 1 for C 2} , and 0 for _{C 3 ).}

According to various embodiments of the present disclosure, by classifying fundus image data using a quality score calculated according to the method for calculating the quality score of the fundus image data, it has higher classification performance than a conventional fundus image data quality classification model. Has an advantage.

More specifically, the classification performance of the apparatus 100 for calculating the quality score of the fundus image data according to an embodiment of the present invention by classifying the fundus image data based on the verification data set of the fundus image data set (eg, FIG. 7 ). As a result of the measurement, as shown in Table 1 below, the classification performance was 2.31% higher than that of the conventional fundus image data quality classification model.

Based on this, it can be verified that the quality score of the fundus image data calculated by the apparatus 100 for calculating the quality score of the fundus image data according to an embodiment of the present invention is valid.

In various embodiments, the computing device 100 may analyze the image data and determine a quality score threshold for filtering the image data. Hereinafter, it will be described with reference to FIG. 14.

14 is a flowchart of a method of calculating a quality score threshold for filtering image data according to another embodiment of the present invention.

Referring to FIG. 14, in step S301, the computing device 100 may collect image data from the outside, and may calculate a quality score for the image data by analyzing the image data collected in step S302 (eg: Steps S110 to S140 of Figure 3).

In step S304, the computing device 100 may group the image data into the Q quintile based on the quality score of each of the plurality of image data calculated in step S302. For example, the computing device 100 sorts a plurality of image data based on the quality score (eg, sorts in ascending order), and arranges the sorted plurality of image data to have the same scale (eg, the same number of image data). ) Can be grouped into Q groups. However, it is not limited thereto.

In step S305, the computing device 100 may evaluate the disease classification performance for each of the Q groups grouped through the step S304 using the disease classification result extracted in step S303.

For example, the computing device 100 analyzes one or more image data included in each of a plurality of quality classes, and determines the precision of each of the plurality of quality classes (for example, the actual disease is determined as having a disease). A ratio of image data corresponding to a patient with a disease) and a recall (eg, a ratio of image data that is determined to have a disease through a diagnosis model among image data corresponding to a patient with an actual disease) may be calculated.

Thereafter, the computing device 100 may calculate an F1 score indicating disease classification performance for one or more image data using precision and recall. For example, the computing device 100 may calculate an f1 score by calculating a harmonic average of precision and recall (eg, 2*(precision*reproducibility)/(precision+reproducibility)). However, the present invention is not limited thereto, and FPR (False Positive Rate) (e.g., the ratio of the image data determined to have a disease through the diagnosis module among image data corresponding to a patient without actual disease), and a Receiver Operating Characteristic (ROC) curve Disease classification performance can be judged using various indicators such as (e.g., a graph showing the change between FPR and recall) and AUC (Area Under Curve) (e.g., the area value of the ROC curve).

In step S306, the computing device 100 may cluster the Q groups into two groups based on the disease classification performance evaluation result for each Q group calculated in step S305.

For example, the computing device 100 compares the F1 score with the reference F1 score, and includes a first group including one or more quality grades having good disease classification performance (eg, a group having an F1 score equal to or greater than the reference F1 score, a first class, and Second grade) and a second group including one or more quality grades with poor disease classification performance (eg, a group having an F1 score less than the reference F1 score, a third grade).

In various embodiments, the computing device 100 uses a machine learning algorithm such as K-means clustering or Gaussian Mixture Model (GMM), without setting a separate reference F1 score, and forming Q groups into two groups. Can be clustered with However, it is not limited thereto.

In step S307, the computing device 100 may perform Connected Component Labeling (CCL) on one or more image data included in two clustered groups (eg, a first group and a second group). . For example, the computing device 100 may perform connection element labeling using a result of clustering a plurality of quality classes into two groups as an input value.

In operation S308, the computing device 100 may determine the number of connection element labels generated as a result of labeling connection elements for two clustered groups (eg, a first group and a second group).

In step S309, when it is determined that the number of connection element labels generated as a result of the connection element labeling performed in step S307 is not two (for example, FIG. 15(b)), the connection element The labeling result of the connection element in which the number of labels is not two can be excluded from the quality score threshold extraction target.

In step S310, when the computing device 100 determines that the number of connection element labels generated as a result of the connection element labeling performed in step S307 is two (for example, FIG. 15(a)), the connection element label A connection element labeling result having two numbers of can be set as a quality score threshold extraction target.

In step S311, the computing device 100 may determine whether the Q quartile is less than a _{preset quartile maximum value Q MAX.} In this case, when it is determined that the _{Q quartile is less than the preset quartile maximum value Q MAX} , the computing device 100 may sequentially perform steps S304 to S310 again. At this time, the computing device 100 may change the Q quartile and repeatedly perform steps S304 to S310 again (eg, FIG. 15 ).

In step S312, when the computing device 100 determines that the Q quintile _{is equal to or greater than the preset quartile maximum value (Q MAX} ) through step S311, the result of labeling the connection element set as a target for extracting the quality score threshold in step S309. The quality score threshold value may be calculated using the connection element labeling result from which two connection element labels are extracted among the connection element labeling results of.

In various embodiments, the computing device 100 extracts a quality score boundary value between two clustered groups (eg, a first group and a second group) when two connection element labels are extracted, and the extracted quality The average of the score boundary values is calculated, and the average of the calculated quality score boundary values may be determined as a quality score threshold value of the image data. For example, the computing device 100 has 12, 16, 18, 19, 22, 24, 30, and 36 times determined that the number of connection element labels is 2 (for example, the times in which the connection element labeling is repeatedly performed). In the case of the round, the average of the quality score boundary values between the first group and the second group in the 12, 16, 18, 19, 22, 24, 30, and 36 rounds is calculated, and the calculated 12, 16, 18, 19, The average of the quality score boundary values at times 22, 24, 30, and 36 may be determined as the quality score threshold value of the image data. However, it is not limited thereto.

The above-described method for calculating the quality score of fundus image data has been described with reference to the flowchart shown in the drawings. For the sake of simplicity, the method of calculating the quality score of the fundus image data has been described by showing a series of blocks, but the present invention is not limited to the order of the blocks, and some blocks are performed in a different order from those shown and described herein. Or can be performed simultaneously. In addition, a new block not described in the specification and drawings may be added, or some blocks may be deleted or changed. Hereinafter, a hardware configuration of the apparatus 100 for calculating a quality score of fundus image data according to another embodiment of the present invention will be described with reference to FIG. 16.

16 is a hardware configuration diagram of an apparatus for calculating a quality score of fundus image data according to another embodiment of the present invention.

Referring to FIG. 16, an apparatus 500 for calculating a quality score of fundus image data (hereinafter, “computing apparatus 500”) according to another embodiment of the present invention includes a processor 510 and a memory 520. I can. In various embodiments, the computing device 500 may further include a network interface (or communication interface) (not shown), a storage (not shown), and a bus (not shown).

In one embodiment, the processor 510 may control the overall operation of each component of the computing device 500. The processor 510 may be configured to include a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), or any type of processor well known in the art.

In various embodiments, the processor 510 may perform an operation on at least one application or program for executing the method according to the embodiments of the present invention. In various embodiments, the processor 510 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. can do.

In various embodiments, the processor 510 temporarily and/or permanently stores a signal (or data) processed inside the processor 510, and a RAM (Random Access Memory, not shown) and a ROM (ROM). -Only Memory, not shown) may further be included. In addition, the processor 510 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, RAM, and ROM.

In one embodiment, the processor 510 executes one or more instructions stored in the memory 520 to perform the method described with respect to FIGS. 2 to 13 (eg, a method of calculating a quality score of fundus image data). can do.

For example, the processor 510 executes one or more instructions stored in the memory 520 to collect fundus image data, extract one or more feature values from the fundus image data, and extract one or more feature values. A fundus image comprising calculating a probability that the fundus image data belongs to each of a plurality of preset quality grades by using and calculating a quality score for the fundus image data using the probability of belonging to each of the plurality of quality grades. A method of calculating the quality score of the data can be performed. However, it is not limited thereto.

In addition, the processor 510 calculates a quality score for each of a plurality of image data by executing one or more instructions stored in the memory 520, and ranks the plurality of image data based on the calculated quality score. Classifying as, determining disease classification performance for one or more image data included in each of the plurality of quality grades, clustering the plurality of quality grades into two groups according to the determined disease classification performance, and clustering the two A method of calculating a quality score threshold value for filtering image data, including calculating a quality score threshold value of the image data by using the quality score boundary value between groups, may be performed.

In one embodiment, the memory 520 may store various types of data, commands, and/or information. The memory 520 may store programs (one or more instructions) for processing and controlling the processor 510. Programs stored in the memory 520 may be divided into a plurality of modules according to functions.

In various embodiments, steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. Software modules include Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

Components of the present invention may be implemented as a program (or application) and stored in a medium in order to be combined with a computer as hardware to be executed. Components of the present invention may be implemented as software programming or software elements, and similarly, embodiments include various algorithms implemented with a combination of data structures, processes, routines or other programming elements, including C, C++ , Java, assembler, or the like may be implemented in a programming or scripting language. Functional aspects can be implemented with an algorithm running on one or more processors.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You can understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

100: device for calculating the quality score of fundus image data
110: feature extraction module
120: quality classification module
130: quality score calculation module
200: user terminal
300: external server
400: network

Claims (10)

In the method performed by the computing device,
Collecting fundus image data;
Extracting one or more feature values from the fundus image data;
Calculating a probability that the fundus image data belongs to each of a plurality of preset quality classes using the extracted one or more feature values; And
Comprising the step of calculating a quality score for the fundus image data by using the calculated probability of belonging to each of the plurality of quality classes,
Method for calculating the quality score of fundus image data. The method of claim 1,
The plurality of quality grades,
A first grade, a second grade lower than the first grade, and a third grade lower than the second grade,
The step of calculating the quality score,
Comprising the step of calculating a quality score for the fundus image data using Equation 1 below,
Method for calculating the quality score of fundus image data.
[Equation 1]

Here, prob(G ₁ ) is the probability that the fundus image data belongs to the first grade, prob(G ₂ ) is the probability that the fundus image data belongs to the second grade, and prob(G ₃ ) is the fundus image data The probability of falling into the third class, C ₁ , C ₂ and C ₃ are preset constant values The method of claim 2,
The one or more feature values,
At least one of a probability that the fundus image data belongs to an abnormal grade, a quality map, brightness, a focus value, an illumination histogram, and a roundness of the fundus image data Includes,
The step of calculating the probability of belonging to each of the plurality of quality classes,
Comprising the step of calculating a probability that the fundus image data belongs to each of the plurality of quality classes from the one or more feature values using an artificial intelligence model that has previously learned a plurality of fundus image data as training data,
Method for calculating the quality score of fundus image data. The method of claim 3,
Extracting the one or more feature values,
A first abnormal grade probability that is a probability that the fundus image data belongs to the abnormal grade using a first binary classification model that classifies the first grade and the second grade as a normal grade and classifies the third grade as an abnormal grade And extracting a first quality map visualizing a region determined to be abnormal in the fundus image data; And
A second abnormal grade probability that is a probability that the fundus image data belongs to the abnormal grade using a second binary classification model that classifies the first grade as a normal grade and classifies the second grade and the third grade as abnormal grades And extracting a second quality map visualizing a region determined to be abnormal in the fundus image data,
Method for calculating the quality score of fundus image data. The method of claim 3,
Extracting the one or more feature values,
Detects an area corresponding to the disc from the fundus image data, extracts the long axis length and the short axis length of the area corresponding to the detected fundus nipple, and extracts the length of the extracted major axis and the extracted short axis Comprising the step of calculating the roundness of the fundus using the ratio of the length of,
Method for calculating the quality score of fundus image data. The method of claim 3,
The sharpness includes a first sharpness and a second sharpness,
Extracting the one or more feature values,
Comprising the step of calculating the first sharpness for the fundus image data using Equation 2 below, and calculating the second sharpness for the fundus image data using Equation 3 below,
Method for calculating the quality score of fundus image data.
[Equation 2]

Here, X is the fundus image data value, std(X) is

, grayscale(X) is

As a gray scale value of the fundus image data _{, X R} is an R channel value of the fundus image data _{, X G is a G channel value of the fundus image data, X B} is a B channel value and mean(X) of the fundus image data. Is

[Equation 3]

The method of claim 3,
Extracting the one or more feature values,
Comprising the step of calculating a focus value for the fundus image data using Equation 4 below,
Method for calculating the quality score of fundus image data.
[Equation 4]

Here, X is the fundus image data value, std(X) is

, grayscale(X) is

As a gray scale value of the fundus image data _{, X R} is an R channel value of the fundus image data _{, X G is a G channel value of the fundus image data, X B} is a B channel value of the fundus image data, and kernel _x is

As a 3x3 kernel in the x direction, kernel _y is

3x3 kernel in y direction The method of claim 3,
The step of extracting the one or more feature values,
Preprocessing the fundus image data;
Extracting one or more patches having a preset size from the preprocessed fundus image data;
Calculating a representative value for each of the one or more patches using Equation 5 below;
Generating an illuminance map using the calculated representative value; And
Including the step of generating the illuminance histogram using the illuminance map,
Method of calculating the quality score of fundus image data.
[Equation 5]

Here, P is the one or more patch values, P _R is the R channel value _{of the one or more patches, P- G} is the G channel value of the one or more patches, P _B is the B channel value and mean(x ) Is

A memory for storing one or more instructions; And
And a processor that executes the one or more instructions stored in the memory,
The processor executes the one or more instructions,
An apparatus for performing the method of claim 1. A computer program combined with a computer as hardware and stored in a recording medium readable by a computer to perform the method of claim 1.

Images