KR102471991B1 - System and Method for Extracting Visual Function Information Through Optical Coherence Tomography based on Artificial Intelligence - Google Patents

Abstract

The present invention relates to a system and method for extracting visual function information through optical coherence tomography based on artificial intelligence technology. More specifically, the system and method for extracting visual function information through optical coherence tomography based on artificial intelligence technology comprises: an OCT image extraction unit extracting an OCT image including at least one of a ganglion cell layer (GCL), a retinal nerve fiber layer (RNFL), and a plan view of the retinal nerve fiber layer (RNFL_ENFACE) for both eyes from an SS-OCT test sheet; a horizontal inversion application unit applying horizontal inversion to an arbitrary single eye in the OCT image; a preprocessing unit performing preprocessing using a histogram matching technique to improve consistency of the OCT image to which the horizontal inversion is applied, and preprocessing using a CLAHE technique to improve contrast of the OCT image to which the horizontal inversion is applied; and a visual function information output unit outputting visual function information of a subject by inputting a preprocessed OCT image to a pre-learned neural network model. Accordingly, consistency and accuracy in diagnosing eye diseases and determining progression thereof can be improved.

Description

System and Method for Extracting Visual Function Information Through Optical Coherence Tomography based on Artificial Intelligence}

The present invention relates to a system and method for extracting visual function information through optical coherence tomography based on artificial intelligence technology. It relates to a system and method for extracting visual function information through technology-based optical coherence tomography.

In general, optical coherence tomography (OCT) is a technique for non-invasively taking three-dimensional tomographic images of the eyeball and other human tissues by using the difference in arrival time of light reflected in tissue by passing light through it. . The optical coherence tomography apparatus may acquire a tomography image of the eyeball, acquire a 3D tomography image by successively stacking the tomographic images, and take a 3D tomographic image of the ocular tissue at micrometer scale resolution. That is, optical coherence tomography images are easy to identify diseases occurring in the retina of the eye and their progress.

However, even when an image is obtained from an optical coherence tomography device, a medical staff clinically determines whether or not an eye disease is present and checks its progress, resulting in poor diagnostic consistency and accuracy. In this regard, Related Document 1 relates to a functional OCT data processing method, and an OCT image can be acquired by an OCT imaging device that scans the retina of an object while the retina is repeatedly stimulated by light stimulation, but from the OCT image There is a technical limitation that cannot diagnose an eye disease or automatically determine its progress.

KR 10-2021-0012971

The present invention is to solve the above problems, and in order to improve the consistency and accuracy in diagnosing eye diseases and determining the progress of eye diseases from optical coherence tomography images, input preprocessed OCT images to a pre-learned neural network model and The purpose of this study is to obtain a system and method for extracting visual function information through artificial intelligence technology-based optical coherence tomography that outputs visual function information.

In order to achieve the above object, the system for extracting visual function information through optical coherence tomography based on artificial intelligence technology of the present invention is a ganglion cell layer (GCL), retinal nerve fiber layer (RNFL), and retinal nerve fiber layer for both eyes from the SS-OCT test strip. an OCT image extraction unit for extracting an OCT image including at least one of a plan view (RNFL_ENFACE) of; a horizontal inversion application unit for applying horizontal inversion to an arbitrary monocular in the OCT image; A preprocessing unit that performs preprocessing using a histogram matching technique to improve the consistency of the OCT image to which horizontal inversion is applied, and preprocessing using a CLAHE technique to improve the contrast of the OCT image to which horizontal inversion is applied. ; and a visual function information output unit configured to output visual function information of a subject by inputting the preprocessed OCT image to a pre-learned neural network model.

In order to achieve the above object, the method for extracting visual function information through optical coherence tomography based on artificial intelligence technology of the present invention is a ganglion cell layer (GCL) for both eyes, a retinal nerve fiber layer, from an SS-OCT test strip by an OCT image extractor. (RNFL), an OCT image extraction step of extracting an OCT image including at least one of a plan view (RNFL_ENFACE) of a retinal nerve fiber layer; a horizontal inversion application step of applying horizontal inversion to an arbitrary monocular in the OCT image by a horizontal inversion application unit; By the pre-processing unit, a histogram matching technique is applied and preprocessed to improve the consistency of the OCT image to which horizontal inversion is applied, and a CLAHE technique is used to improve the contrast of the OCT image to which horizontal inversion is applied. A pre-processing step that is pre-processed; and a visual function information outputting step of outputting the visual function information of the subject by inputting the preprocessed OCT image to the pre-learned neural network model by the visual function information output unit.

As described above, according to the present invention, by inputting the preprocessed OCT image to the pre-learned neural network model and outputting the test subject's visual function information, consistency and accuracy are improved in diagnosing eye diseases and determining the progress from optical coherence tomography images. There are remarkable effects that can be done.

1 is a block diagram of a visual function information extraction system through optical coherence tomography based on artificial intelligence technology of the present invention.
2 is a diagram showing an OCT image before preprocessing and a preprocessed OCT image according to an embodiment of the present invention.
3 is a diagram showing a neural network model according to an embodiment of the present invention.
4 is a diagram showing visual function information clustered into seven clusters according to an embodiment of the present invention.

The terms used in this specification have been selected from general terms that are currently widely used as much as possible while considering the functions in the present invention, but these may vary depending on the intention of a person skilled in the art, precedent, or the emergence of new technologies. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, not simply the name of the term.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, it should not be interpreted in an ideal or excessively formal meaning. don't

Visual function information extraction system through optical coherence tomography based on artificial intelligence technology

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. 1 is a block diagram of a visual function information extraction system through optical coherence tomography based on artificial intelligence technology of the present invention. 2 is a diagram showing an OCT image before preprocessing and a preprocessed OCT image according to an embodiment of the present invention. 3 is a diagram showing a neural network model according to an embodiment of the present invention. 4 is a diagram showing visual function information clustered into seven clusters according to an embodiment of the present invention.

Referring to FIG. 1, the system for extracting visual function information through optical coherence tomography based on artificial intelligence technology of the present invention includes an OCT image extractor 100, a horizontal inversion application unit 200, a preprocessor 300, and output of visual function information. Includes section 400.

More specifically, the OCT image extraction unit 100 is an OCT image including at least one of a ganglion cell layer (GCL), a retinal nerve fiber layer (RNFL), and a plan view of the retinal nerve fiber layer (RNFL_ENFACE) for both eyes from the SS-OCT test strip. extract

In general, Optical Coherence Tomography (OCT) is a technique for measuring the subsurface of an opaque or translucent object with micrometer-level resolution, among which SS-OCT (Swept Source Optical Coherence Tomography) has a wavelength It is a technique of imaging with variable optical coherence tomography.

That is, the SS-OCT test strip may include OCT images taken from various perspectives in three dimensions with an optical coherence tomography system in which the wavelength is variable among optical coherence tomography techniques. Here, the OCT image extraction unit 100 may extract the ganglion cell layer (GCL) image such as the color image shown on the left side of FIG. 2 using an automatic image extraction algorithm, and the black and white image shown on the right side of FIG. The same retinal nerve fiber layer (RNFL) and plan view (RNFL_ENFACE) images of the retinal nerve fiber layer can be extracted. At this time, the plan view (RNFL_ENFACE) image of the retinal nerve fiber layer refers to a view of the retinal nerve fiber layer photographed in three dimensions as viewed from the XY plane.

Meanwhile, the OCT image extraction unit 100 may store OCT images for each subject by matching the identification number of the subject to be examined. And most preferably, the size of the ganglion cell layer (GCL) image may be 200 in width and 200 in height, and the retinal nerve fiber layer (RNFL) and plan view (RNFL_ENFACE) image of the retinal nerve fiber layer may be 280 in width and 200 in height. .

Next, the horizontal inversion application unit 200 applies horizontal inversion to an arbitrary monocular in the OCT image. That is, since the OCT image is extracted for both eyes, a right eye image and a left eye image exist respectively. At this time, all left-eye images may be horizontally inverted into right-eye images or all right-eye images may be horizontally inverted into left-eye images in order to improve the accuracy of inspection.

Next, the preprocessing unit 300 performs preprocessing using a histogram matching technique to improve the consistency of the OCT image to which the horizontal inversion is applied, and CLAHE to improve the contrast of the OCT image to which the horizontal inversion is applied. ) technique is used for preprocessing.

Regarding the consistency of the OCT images, it cannot be determined that all OCT images extracted from the OCT image extraction unit 100 were captured under the same conditions. Differences in brightness and contrast may occur depending on numerous variables such as year and date of capture, photographer, device manufacturer, device damage, test subject's posture, etc. This is a neural network that performs matrix calculation for each pixel of the OCT image and derives an output value. can have a significant impact on the model. Therefore, the pre-processing unit 300 may use a histogram matching technique that matches the brightness and contrast between the OCT images to the pixel probability distribution form of the OCT image as a reference. In general, the Histogram Matching technique is a method of changing the main color of a target image to the main color of a reference image. , there is a remarkable effect of adjusting the overall contrast for images with the same domain.

In addition, contrast enhancement of the OCT image is very important in determining visual field damage information. The CLAHE method evenly flattens the histogram distribution level for the brightness of the image and consequently increases the contrast of the image, and accordingly, information such as blood vessels in the image can be more easily identified.

The CLAHE technique divides an image into small blocks having a certain size and performs histogram equalization for each block, and has a remarkable effect of attenuating noise in an image compared to the conventional equalization method. Here, histogram equalization is a method of increasing the contrast of an image by adjusting pixel values concentrated in a narrow range to be evenly distributed over the entire range of 0 to 255.

Meanwhile, the pre-processing unit 300 may horizontally connect the pre-processed OCT images so that they can be input to the neural network model in the same size. Referring to FIG. 2, the pre-processing unit 300 pre-processes with the histogram matching technique and the CLAHE technique and shows the thickness of the ganglion cell layer (GCL) image in the form of a heat map and the plan view of the retinal nerve fiber layer shown as the difference in brightness ( RNFL_ENFACE) Images can be horizontally concatenated and final pre-processed into an image with a width of 480 and a height of 200. FIG. 2(a) shows an OCT image before preprocessing, and FIG. 2(b) shows an OCT image after final preprocessing.

Next, the visual function information output unit 400 outputs the visual function information of the subject by inputting the pre-processed OCT image to the pre-learned neural network model.

The architecture of the neural network model is the same as the embodiment of FIG. 3 . First, the neural network model may apply weights pretrained on the ImageNet dataset using Inception ResnetV2 as a base model. After the basic model, a plurality of fully connected layers having a Relu activation function are generated by generating as many dense layers as each feature through Global Average Pooling (GAP). It is possible to compress the features by going through. In addition, the neural network model may finally extract as many visual function information as a predetermined number.

For example, if 2048 features are extracted through global average pooling (GAP), 2048 dense layers can be generated, and 1024 dense layers compressed by 1/2 in one fully connected layer. Features are extracted, 512 features compressed by 1/2 are extracted in the next fully connected layer, and 256 features compressed by 1/2 can be extracted in the next fully connected layer. . Finally, 68 visual function information can be extracted, and most preferably displayed in the shape of the cornea. Here, the visual function information is a numerical value in units of decibels (dB) in the corresponding area.

The final extraction of 68 visual function information from the 256 features compressed by 1/2 in the Fully Connected Layer activates neurons through the Relu activation function, It outputs the predicted value for each. Accordingly, the number of visual function information extracted from the fully connected layer may be changed.

In addition, the visual function information output unit 400 includes a clustering unit 410 that clusters the plurality of visual function information into n clusters according to a predetermined criterion and calculates an average value between the clusters, and the clustering unit ( 410) may linearly transform the m pieces of visual function information in an arbitrary cluster, calculate the average value, and log-transform the average value again. Here, n and m are positive integers.

At this time, the visual function information output unit 400 most preferably uses Latent Class Analysis to cluster the visual function information that is deteriorating among a plurality of the visual function information and the adjacent visual function information that is most related to the visual function information to obtain 68 pieces of visual function information. By clustering into 7 clusters, there is a remarkable effect of reflecting the deterioration pattern of the central visual field more appropriately than the general model and providing the damage pattern of the central visual field more intuitively.

For example, the clustering unit 410 may cluster into groups 1 to 7, and group 1 may include 15 visual function information. In addition, the clustering unit 410 may linearly transform (1/Lambert) 15 visual function information (dB) into [Equation 1] below.

The clustering unit 410 calculates the average value (1/Lambert) for the linearly transformed 15 pieces of visual function information (1/Lambert), and calculates the average value (1/Lambert) again as [Equation 1]. It is possible to obtain the log-transformed average value (dB) again by substituting in .

Scaling of logarithmic conversion from (1/Lambert) to (dB) can lead to an artificial curvilinear relationship between structure and function. There are downsides to minimize. In order to overcome this disadvantage, the clustering unit 410 calculates the average value (1/Lambert) for the linearly transformed 15 pieces of visual function information (1/Lambert), and then performs log-transformed scaling (Scaling). let it do

A method for extracting visual function information through optical coherence tomography based on artificial intelligence technology

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. The method for extracting visual function information through optical coherence tomography based on artificial intelligence technology of the present invention includes an OCT image extraction step (S100), a horizontal inversion application step (S200), a preprocessing step (S300), and a visual function information output step (S400). include

More specifically, the OCT image extraction step (S100) is a plan view of the ganglion cell layer (GCL), retinal nerve fiber layer (RNFL), and retinal nerve fiber layer for both eyes from the SS-OCT test strip by the OCT image extraction unit 100 ( An OCT image including at least one of RNFL_ENFACE) is extracted.

Most preferably, in the OCT image extraction step (S100), a ganglion cell layer (GCL) image in which the thickness is displayed in a heat map format and a plan view (RNFL_ENFACE) image of the retinal nerve fiber layer (RNFL_ENFACE) displayed in a contrast format may be extracted. In this case, the GCL image may have a size of 200 in width and 200 in height, and the RNFL_ENFACE image may have a size of 280 in width and 200 in height.

Meanwhile, in the OCT image extraction step (S100), the OCT image for each subject may be stored by matching the extracted OCT image to the unique number of the subject.

Next, in the horizontal inversion applying step (S200), horizontal inversion is applied to an arbitrary monocular in the OCT image by the horizontal inversion application unit 200. That is, since the OCT image is extracted for both eyes, a right eye image and a left eye image exist respectively. At this time, in the horizontal inversion application step (S200), all left-eye images may be horizontally inverted into right-eye images or all right-eye images may be horizontally inverted into left-eye images to improve the accuracy of inspection.

Next, in the preprocessing step (S300), a histogram matching technique is used to improve the consistency of the OCT image to which the horizontal inversion is applied by the preprocessor 300, and the OCT image to which the horizontal inversion is applied is contrasted. It is pretreated using the CLAHE technique to improve

First, in terms of consistency of the OCT images, it cannot be determined that all OCT images extracted from the OCT image extraction step (S100) were captured under the same conditions. Differences in brightness and contrast may occur depending on numerous variables, such as year and date of capture, photographer, device manufacturer, device damage, and test subject's posture. It can have a big impact on the neural network model it derives.

Therefore, in the preprocessing step (S300), a histogram matching technique that matches the brightness and contrast between the OCT images to the pixel probability distribution form of the OCT image as a reference is used first to improve the consistency of the OCT images. can In general, the Histogram Matching technique is a method of changing the main color of a target image to the main color of a reference image. , there is a remarkable effect of adjusting the overall contrast for images with the same domain.

In addition, in the preprocessing step (S300), a CLAHE technique may be used to easily grasp information on visual field damage through contrast enhancement. The CLAHE method is a technique in which the histogram distribution level for the brightness of an image is evenly flattened and consequently the contrast of the image is increased, and accordingly, information such as blood vessels in the image can be more easily identified.

More specifically, the CLAHE technique is a method of dividing an image into small blocks having a certain size and performing histogram equalization for each block. It works. Here, histogram equalization is a method of increasing the contrast of an image by adjusting pixel values concentrated in a narrow range to be evenly distributed over the entire range of 0 to 255.

Meanwhile, in the preprocessing step (S300), the preprocessed OCT images may be horizontally connected so that they can be input to the neural network model in the same size. That is, in the preprocessing step (S300), the ganglion cell layer (GCL) image preprocessed by the histogram matching technique and the CLAHE technique and the thickness is shown in the form of a heat map and the plan view (RNFL_ENFACE) image of the retinal nerve fiber layer shown by the difference in brightness may be connected horizontally and finally pre-processed into an image having a width of 480 and a height of 200.

Next, in the visual function information outputting step (S400), the visual function information of the subject is output by inputting the pre-processed OCT image to the pre-learned neural network model by the visual function information output unit 400.

The architecture of the neural network model is the same as the embodiment of FIG. 3 . First, the neural network model may be applied with pre-learned weights of the ImageNet dataset using Inception ResnetV2 as a base model. After the basic model, Dense Layers are generated as many as the number of features through Global Average Pooling (GAP), and a plurality of Fully Connected Layers having a Relu activation function The features may be compressed through. In addition, the neural network model may finally extract as many visual function information as a preset number.

For example, if 2048 features are extracted through global average pooling (GAP), 2048 dense layers can be generated, and 1024 dense layers compressed by 1/2 in one fully connected layer. Features are extracted, 512 features compressed by 1/2 are extracted in the next fully connected layer, and 256 features compressed by 1/2 can be extracted in the next fully connected layer. . Finally, 68 visual function information can be extracted, and most preferably displayed in the shape of the cornea. Here, the visual function information is a numerical value in units of decibels (dB) in the corresponding area.

Finally, 68 pieces of visual function information are extracted from the 256 features compressed by 1/2 in the Fully Connected Layer. The predicted values for each are output. Accordingly, the number of visual function information extracted from the fully connected layer may be changed.

On the other hand, the visual function information output step (S400) is a clustering step (S410) in which a plurality of visual function information is clustered into n clusters by the clustering unit 410 according to a predetermined criterion and then an average value between the clusters is calculated. In the clustering step (S410), after linear transformation of the m pieces of visual function information in a random cluster, the average value may be calculated, and the average value may be log-transformed again. Here, n and m are positive integers.

At this time, in the visual function information output step (S400), Latent Class Analysis is most preferably used, and among a plurality of the visual function information, the deteriorated visual function information and the adjacent visual function information most closely related are clustered to obtain 68 visual function information. By being clustered into 7 clusters, there is a remarkable effect that the deterioration pattern of the central visual field can be reflected more appropriately than the general model and the damage pattern of the central visual field can be provided more intuitively.

For example, in the clustering step (S410), groups 1 to 7 may be clustered, and group 1 may include 15 visual function information. In the clustering step (S410), 15 visual function information (dB) may be linearly converted (1/Lambert) into [Equation 1]. In the clustering step (S410), the average value (1/Lambert) for the linearly transformed 15 pieces of visual function information (1/Lambert) is calculated, and the average value (1/Lambert) is calculated again using [Equation 1]. The average value (dB) log-transformed again can be obtained by substituting in .

Scaling of logarithmic conversion from (1/Lambert) to (dB) can lead to an artificial curvilinear relationship between structure and function. There are downsides to minimize. In order to overcome this disadvantage, the clustering unit 410 calculates the average value (1/Lambert) for the linearly transformed 15 pieces of visual function information (1/Lambert), and then performs log-transformed scaling (Scaling). let it do

As described above, according to the present invention, by inputting the preprocessed OCT image to the pre-learned neural network model and outputting the test subject's visual function information, consistency and accuracy are improved in diagnosing eye diseases and determining the progress from optical coherence tomography images. There are remarkable effects that can be done.

Embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments that perform necessary tasks may be stored on a computer readable storage medium and executed by one or more processors.

And aspects of the subject matter described herein may be described in the general context of computer-executable instructions, such as a program module or component executed by a computer. Generally, program modules or components include routines, programs, objects, and data structures that perform particular tasks or implement particular data types. Aspects of the subject matter described herein may be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and/or the components of the system, structure, device, circuit, etc. described in are combined or combined in a different form than the described method, or in a different configuration. Appropriate results can be achieved even when substituted or substituted by elements or equivalents.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

100.. OCT image extraction unit
200.. Horizontal inversion application part
300.. pre-processing
400.. Visual function information output unit
410.. Colonization Department

Claims (5)

An OCT image extraction unit for extracting an OCT image including at least one of a ganglion cell layer (GCL), a retinal nerve fiber layer (RNFL), and a plan view (RNFL_ENFACE) of the retinal nerve fiber layer for both eyes from the SS-OCT test strip;
a horizontal inversion application unit for applying horizontal inversion to an arbitrary monocular in the OCT image;
A pre-processing unit that preprocesses using a histogram matching technique to improve the consistency of the OCT image to which horizontal inversion is applied and preprocesses it using a CLAHE technique to improve the contrast of the OCT image to which horizontal inversion is applied. ; and
A visual function information extraction system through an optical coherence tomography based on artificial intelligence technology comprising: a visual function information output unit for outputting visual function information of a subject by inputting the preprocessed OCT image to a pre-learned neural network model. According to claim 1,
The pre-processing unit,
A visual function information extraction system through artificial intelligence technology-based optical coherence tomography, characterized in that for horizontally connecting the preprocessed OCT images so that they can be input in the same size to the neural network model. According to claim 1,
The neural network model,
Using Inception ResnetV2 as a base model, weights pretrained on the ImageNet dataset are applied,
After the basic model, as many dense layers as the number of features are generated through Global Average Pooling (GAP),
Compressing the feature through a plurality of fully connected layers having a Relu activation function,
A visual function information extraction system through artificial intelligence technology-based optical coherence tomography, characterized in that the visual function information by a predetermined number is extracted. According to claim 1,
The visual function information output unit,
A clustering unit configured to cluster the plurality of visual function information into n clusters according to a predetermined criterion and then calculate an average value between the clusters;
The clustering unit,
A visual function information extraction system through artificial intelligence technology-based optical coherence tomography, characterized in that the average value is calculated after linear transformation of the m visual function information in any cluster, and the average value is log-transformed again.
Here, n and m are positive integers. The OCT image extraction unit extracts an OCT image in which an OCT image including at least one of a ganglion cell layer (GCL), a retinal nerve fiber layer (RNFL), and a plan view of the retinal nerve fiber layer (RNFL_ENFACE) for both eyes is extracted from the SS-OCT test strip. step;
a horizontal inversion application step of applying horizontal inversion to an arbitrary monocular in the OCT image by a horizontal inversion application unit;
By the pre-processing unit, a histogram matching technique is applied and preprocessed to improve the consistency of the OCT image to which horizontal inversion is applied, and a CLAHE technique is used to improve the contrast of the OCT image to which horizontal inversion is applied. A pre-processing step that is pre-processed; and
A visual function information output step of outputting the visual function information of the subject by inputting the preprocessed OCT image to the pre-learned neural network model by the visual function information output unit; through optical coherence tomography based on artificial intelligence technology, including A method for extracting visual function information.

Images