JP2021058285A - Image processing device, image processing method and program - Google Patents

Abstract

To provide an image processing device that can acquire tomographic information from which highly reliable analysis result can be acquired.SOLUTION: An image processing device includes: an acquisition unit for acquiring a first tomographic image of a subject's eye; and an arithmetic processing unit for generating a second tomographic image reduced with a fold back image in the first tomographic image from the first tomographic image using a learned model.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing apparatus, an image processing method, and a program.

As an ophthalmic apparatus capable of non-invasively obtaining a tomographic image of an eye to be inspected, an apparatus (OCT apparatus) using an optical coherence tomography (OCT), which photographs using low coherence light, is known. There is. The OCT device divides the low coherent light into reference light and measurement light, irradiates the object to be measured with the measurement light, interferes with the return light from the object to be inspected and the reference light, and uses the spectral information of the interference light to indicate the object to be inspected. Take a picture of the fault.

Here, it is known that the retina of the human eye is composed of a plurality of layers, and in ophthalmic diagnosis, the layer structure is interpreted based on the imaging data of the OCT device, and the state of the lesion is confirmed. To do. In addition, in order to confirm the condition of the lesion using the volume image data of the retina, it is possible to display a tomographic image, analyze the layer structure of the retina, and display a layer thickness graph, layer thickness map, etc. It is valid.

However, when trying to obtain a tomographic image of an object to be inspected using spectral information, a folded image with respect to a normal tomographic image is generated based on a position called the coherence gate position where the measurement optical path and the reference optical path are equal to each other. It is known. In particular, when an eye to be inspected having a large curvature such as high myopia is photographed, a folded image is likely to occur. Further, even if the eye to be inspected has a small curvature, a folded image may occur if the position of the coherence gate is insufficiently adjusted or if the eye to be inspected moves during imaging.

When a folded image occurs in a tomographic image, it becomes difficult to identify the layer structure by image analysis. Therefore, there is a possibility that the layer thickness analysis data of the region where the folded image of the tomographic image is generated cannot be properly presented to the user. Therefore, Patent Document 1 discloses a technique in which a region in which a folded image is generated in a tomographic image is clearly shown on a layer thickness map, and a region having low reliability is displayed in an easy-to-understand manner.

Japanese Unexamined Patent Publication No. 2014-217423

However, the technique disclosed in Patent Document 1 makes it easy to identify a region where the reliability of the image analysis result is lowered due to folding back. Therefore, the technique disclosed in Patent Document 1 still displays unreliable image analysis results in the region where the folded image is generated.

Therefore, one of the objects of one embodiment of the present invention is to provide an image processing apparatus capable of acquiring tomographic information capable of acquiring highly reliable analysis results.

An image processing apparatus according to an embodiment of the present invention uses an acquisition unit for acquiring a first tomographic image of an eye to be inspected and a trained model to obtain a first tomographic image from the first tomographic image. It is provided with an arithmetic processing unit that generates a second tomographic image in which the folded image is reduced.

According to one embodiment of the present invention, it is possible to acquire tomographic information capable of acquiring highly reliable analysis results.

It is a schematic diagram which shows the structural example of the OCT apparatus which concerns on Embodiment 1. FIG. It is a block diagram which shows the structural example of the control part which concerns on Embodiment 1. FIG. It is a schematic diagram which shows an example of the display screen which concerns on Embodiment 1. FIG. It is a schematic diagram which shows an example of the machine learning model which concerns on Embodiment 1. It is a flowchart of a series of processing by the control unit which concerns on Embodiment 1. It is a block diagram which shows the structural example of the learning part which concerns on Embodiment 1. FIG. It is a figure explaining the method of generating the learning data which concerns on Embodiment 1. FIG. It is a flowchart of a series of processing by a learning part which concerns on Embodiment 1. It is a flowchart of a series of processing by the control unit which concerns on Embodiment 2. It is a schematic diagram for demonstrating the folding back reduction process which concerns on Embodiment 2. It is a schematic diagram which shows an example of the display screen which concerns on Embodiment 2. 6 is a flowchart of a series of processes by the control unit according to the third embodiment. It is a schematic diagram for demonstrating the folding back reduction process which concerns on Embodiment 3. It is a flowchart of a series of processing by the control unit which concerns on Embodiment 4. It is a schematic diagram which shows an example of the display screen which concerns on Embodiment 5. It is a schematic diagram which shows an example of the machine learning model which concerns on modification 1.

Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of the components, etc. described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar. In the present specification, the folded image means an image in which the tomographic image to be imaged is folded with reference to the coherence gate position. Further, the normal image means an image in which the tomographic image to be imaged is not folded with respect to the coherence gate position, in other words, an image other than the folded image. Further, the image in which the folded image is reduced means an image in which the region where the folded image appears is reduced in the image. For example, the image in which the folded image is removed or the folded image is corrected to a normal image. Includes images.

Further, the machine learning model refers to a learning model based on a machine learning algorithm. Specific algorithms for machine learning include the nearest neighbor method, the naive Bayes method, a decision tree, and a support vector machine. In addition, deep learning (deep learning) in which features and coupling weighting coefficients for learning are generated by oneself using a multi-layer neural network can also be mentioned. Further, for at least a part of the multi-layer neural network, for example, a convolutional neural network (CNN) can be used as a machine learning model. Further, a technique related to an autoencoder (self-encoder) may be used for at least a part of a multi-layer neural network. Further, a technique related to backpropagation (backpropagation method) may be used for learning. However, the machine learning is not limited to deep learning, and any learning using a model capable of extracting (expressing) the features of learning data such as images by learning may be used. At this time, the machine learning model may be, for example, a capsule network (Capsule Network; CapsNet). Here, in a general neural network, each unit (each neuron) is configured to output a scalar value, so that, for example, spatial information regarding the spatial positional relationship (relative position) between features in an image can be obtained. It is configured to be reduced. Thereby, for example, learning can be performed so as to reduce the influence of local distortion and translation of the image. On the other hand, in the capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is retained. Thereby, for example, learning can be performed in which the spatial positional relationship between the features in the image is taken into consideration. It should be noted that, as appropriate, any of the above algorithms that can be used can be applied to the following embodiments. The teacher data refers to learning data and is composed of a pair of input data and output data. The correct answer data refers to the output data of the learning data (teacher data).

The trained model is a model obtained (trained) by training a machine learning model by an arbitrary machine learning algorithm using appropriate learning data in advance. However, although the trained model is obtained by using appropriate training data in advance, it does not mean that further training is not performed, and additional learning can be performed. Additional learning can be performed even after the device has been installed at the site of use.

(Embodiment 1)
<Outline configuration of ophthalmic imaging device>
Hereinafter, an optical coherence tomography apparatus (OCT) apparatus will be described as an example of the ophthalmic apparatus according to the first embodiment of the present invention with reference to FIGS. 1 to 6. The OCT apparatus according to the present embodiment generates a tomographic image in which the folded image is reduced by using the trained model, and provides a highly reliable analysis result.

A schematic configuration of the OCT apparatus according to the present embodiment will be described with reference to FIG. FIG. 1 shows a schematic configuration example of the OCT apparatus according to the present embodiment. The OCT apparatus of this embodiment is a spectrum domain (SD: Spectrum Domain) -OCT that disperses interference light and obtains information in the depth direction from spectral information. The OCT apparatus of this embodiment is provided with an optical head unit 100, a spectroscope 150, a control unit 200 (image processing device), and a display unit 250. The display unit 250 is composed of an arbitrary display, and can display information on the subject, various images, and the like under the control of the control unit 200. Hereinafter, the configurations of the optical head unit 100, the spectroscope 150, and the control unit 200 will be described in order.

<Structure of optical head unit 100 and spectroscope 150>
The optical head unit 100 is provided with a measurement optical system for capturing a two-dimensional image and a tomographic image of the anterior eye portion Ea and the fundus Ef of the eye E to be inspected. Hereinafter, the inside of the optical head unit 100 will be described. In the optical head unit 100, an objective lens 101-1 is installed facing the eye E to be inspected, and the first dichroic mirror 102 and the second dichroic mirror 103 provided on the optical axis of the optical head unit 100 and function as an optical path separation unit. The optical path is separated by. As a result, the optical path from the objective lens 101-1 is branched into the measurement optical path L1 of the OCT optical system, the fundus observation optical path and the fixation lamp optical path L2, and the anterior eye observation optical path L3 for each wavelength band.

In the present embodiment, the optical path L3 for front eye observation is provided in the transmission direction of the first dichroic mirror 102, the measurement optical path L1 of the OCT optical system, and the fundus observation optical path and the fixation light path L2 are provided in the reflection direction. Has been done. Further, the measurement optical path L1 of the OCT optical system is provided in the transmission direction of the second dichroic mirror 103, and the fundus observation optical path and the fixation lamp optical path L2 are provided in the reflection direction. However, the direction in which the optical path of each optical system is provided is not limited to this, and may be arbitrarily changed according to a desired configuration.

Further, the optical path L2 is branched by a third dichroic mirror 118 into an optical path to an APD (avalanche photodiode) 115 for fundus observation and an optical path to the fixation lamp 116 for each wavelength band. In the present embodiment, the APD 115 is provided in the transmission direction of the third dichroic mirror 118, and the fixation lamp 116 is provided in the reflection direction. However, the fixation lamp 116 may be provided in the transmission direction of the third dichroic mirror 118, and the APD 115 may be provided in the reflection direction.

The optical path L2 is provided with a lens 101-2, an X scanner 117-1, a Y scanner 117-2, lenses 111, 112, a third dichroic mirror 118, an APD 115, and a fixation lamp 116 in this order from the second dichroic mirror 103. Has been done. The lens 111 can be driven in the direction of the optical axis indicated by the arrow in the drawing by a motor (not shown) controlled by the control unit 200 for adjusting the focus of the light for observing the fixation lamp and the fundus.

The APD 115 has a sensitivity in the wavelength of the illumination light for fundus observation (not shown), specifically in the vicinity of 780 nm. The APD115 is a single detector and detects the light scattered / reflected from the fundus Ef and returned. The APD 115 generates an output signal based on the detected light and sends it to the control unit 200. The control unit 200 can generate a two-dimensional image of the fundus Ef (frontal image of the fundus) based on the output signal from the APD 115. The fixation lamp 116 can be used to generate visible light to promote fixation of a subject.

Further, the X scanner 117-1 (for the main scanning direction) and the Y scanner 117-2 (for the sub scanning direction intersecting the main scanning direction) emit light emitted from an illumination light source for observing the fundus of the eye (not shown). It can function as a scanning unit for scanning on the fundus Ef of the eye. The scanning portion of the optical path L2 may be configured by using an arbitrary deflection means such as a polygon mirror or a galvano mirror. The lens 101-2 is arranged with the vicinity of the center position of the X scanner 117-1 and the Y scanner 117-2 as the focal position. The third dichroic mirror 118 is a prism on which a perforated mirror or a hollow mirror is deposited, and separates the illumination light emitted to the eye E to be inspected and the return light from the fundus Ef.

A lens 141 and an infrared CCD 142 for frontal eye observation are arranged in the optical path L3. The infrared CCD 142 has a sensitivity in the wavelength of the irradiation light for front eye observation (not shown), specifically in the vicinity of 970 nm. The infrared CCD 142 generates an output signal based on the detected light and sends it to the control unit 200. The control unit 200 can generate an anterior segment image based on the output signal from the infrared CCD 142.

The measurement optical path L1 of the OCT optical system is provided with an OCT optical system for capturing a tomographic image of the eye E to be inspected. More specifically, the OCT optical system is used to obtain an interference signal for generating a tomographic image of the fundus Ef of the eye E to be inspected. The OCTA optical system may be used to generate a tomographic image of the anterior segment Ea of the eye E to be inspected.

A lens 101-3, a mirror 121, and an X scanner 122-1 and a Y scanner 122-2 that function as an OCT scanning unit for scanning light on the fundus Ef of the eye E to be inspected are arranged in the measurement optical path L1. Has been done. The X-scanner 122-1 and the Y-scanner 122-2 are arranged so that the vicinity of the center position of the X-scanner 122-1 and the Y-scanner 122-2 is the focal position of the lens 101-3. Further, the vicinity of the center position of the X scanner 122-1 and the Y scanner 122-2 and the position of the pupil of the eye E to be inspected have an optical conjugate relationship. The configuration of the OCT scanning unit is not limited to this, and may be arbitrarily changed according to a desired configuration. For example, the OCT scanning unit may be configured by a MEMS mirror or the like that can deflect light in a two-dimensional direction with one sheet.

Further, lenses 123 and 124 are arranged in the measurement optical path L1, and the lens 123 is indicated by an arrow in the drawing by a motor (not shown) controlled by the control unit 200 in order to adjust the focusing of the OCT optical system. It can be driven in the axial direction.

Next, the configuration of the optical path from the measurement light source 126, the reference optical system, and the spectroscope 150 will be described. The measurement light source 126 is a low coherent light source for incidenting the measurement light into the measurement optical path. The measurement light source 126 is connected to the optical coupler 125 via an optical fiber 125-1. Optical fibers 125-1 to 125-4 are connected to the optical coupler 125. Optical fibers 125-1 to 125-4 are single-mode optical fibers connected to and integrated with the optical coupler 125.

The fiber end of the optical fiber 125-2 is arranged on the measurement optical path L1, and the measurement light enters the measurement optical path L1 through the optical fiber 125-2. On the other hand, the fiber end of the optical fiber 125-3 is arranged in the optical path of the reference optical system, and the reference light described later enters the optical path of the reference optical system through the optical fiber 125-3. A lens 131, a dispersion compensation glass 132, and a reference mirror 133 are provided in the optical path of the reference optical system. Further, the optical fiber 125-4 is connected to the spectroscope 150.

The Michelson interferometer is composed of the measurement light source 126, the optical coupler 125, the optical fibers 125-1 to 4, the lens 131, the dispersion compensation glass 132, the reference mirror 133, and the spectroscope 150. In this embodiment, the Michelson interference system is used as the interference system, but a Mach-Zehnder interference system may be used. Depending on the difference in light intensity between the measurement light and the reference light, a Mach-Zehnder interference system can be used when the light intensity difference is large, and a Michelson interference system can be used when the light intensity difference is relatively small.

The light emitted from the measurement light source 126 is divided into the measurement light on the optical fiber 125-2 side and the reference light on the optical fiber 125-3 side via the optical coupler 125 through the optical fiber 125-1. The measurement light is irradiated to the fundus Ef of the eye E to be observed through the measurement optical path L1 of the OCT optical system described above, and reaches the optical coupler 125 through the same optical path by reflection or scattering by the retina.

On the other hand, the reference light reaches the reference mirror 133 and is reflected through the optical fiber 125-3, the lens 131, and the dispersion compensation glass 132. Then, the reference light returns to the optical coupler 125 through the same optical path. Here, the reference mirror 133 is held in an adjustable position in the optical axis direction indicated by an arrow in the drawing by a motor or the like (not shown) controlled by the control unit 200.

In the optical coupler 125, the measurement light and the reference light are combined to form interference light. Here, the measurement light and the reference light cause interference when the optical path length of the measurement light and the optical path length of the reference light are substantially the same. The control unit 200 controls a motor or the like (not shown) and moves the reference mirror 133 in the optical axis direction, so that the optical path length of the reference light can be adjusted to the optical path length of the measurement light that changes depending on the eye E to be inspected. The interference light generated by the optical coupler 125 is guided to the spectroscope 150 via the optical fiber 125-4.

The spectroscope 150 is provided with a lens 151, a diffraction grating 152, a lens 153, and a line sensor 154. The interference light emitted from the optical fiber 125-4 becomes substantially parallel light through the lens 151, is separated by the diffraction grating 152, and is imaged on the line sensor 154 by the lens 153. The line sensor 154 has a configuration in which a plurality of pixels, that is, light receiving elements are arranged in a row, and all the pixels can be read out at once by a predetermined clock. The control unit 200 can generate a tomographic image of the eye E to be inspected by using the interference signal based on the interference light output from the line sensor 154.

Next, the control unit 200 (image processing device) will be described with reference to FIG. FIG. 2 shows a configuration example of the control unit 200. The control unit 200 is provided with an acquisition unit 201, a device control unit 202, a display control unit 203, and an image processing unit 210.

The acquisition unit 201 acquires various information, images, and the like from the optical head unit 100 connected to the control unit 200, an external device (not shown), and an input unit (not shown). For example, the acquisition unit 201 can acquire information such as the output signals of the APD 115 and the infrared CCD 142 and the interference signal output from the line sensor 154 from the optical head unit 100. The acquisition unit 201 may acquire the information corresponding to these from an external device connected to the control unit 200. Here, the control unit 200 may be directly connected to an external device, or may be connected via an arbitrary network such as a LAN or the Internet. Further, the input unit may include a keyboard, a mouse, and the like. When the display unit 250 is composed of a touch panel display, the display unit 250 may function as at least a part of the input unit.

The acquisition unit 201 includes fault data acquired by the optical head unit 100, a signal obtained by subjecting the interference signal to Fourier transform, a signal obtained by subjecting the signal to arbitrary processing, a tomographic image based on these, and the like. Can be obtained. Further, the acquisition unit 201 may acquire similar tomographic data from an external device.

The device control unit 202 controls the optical components of the optical head unit 100, such as controlling the OCT device, more specifically, driving control of each scanner and lighting control of the light source. The display control unit 203 causes the display unit 250 to display patient information, an image processed by the image processing unit 210, information obtained by analyzing the image, and the like.

The image processing unit 210 generates an image based on information acquired from the optical head unit 100, an external device, or the like, and performs image processing on the image. The image processing unit 210 is provided with a front image generation unit 211, a tomographic image generation unit 212, an arithmetic processing unit 213, and an analysis unit 214.

The front image generation unit 211 generates a front image of the eye E to be inspected based on the signal of the APD 115 in the fundus observation optical system and the signal of the infrared CCD 142 of the anterior eye observation optical system. More specifically, the front image generation unit 211 uses the output signal acquired from the APD 115 when the device control unit 202 controls the X scanner 117-1 and the Y scanner 117-2 and scans the illumination light on the fundus Ef. Based on this, a frontal image can be generated. Further, the front image generation unit 211 is based on an output signal from the infrared CCD 142 that detects light emitted from a light source for front eye observation (not shown) and reflected by the front eye portion Ea of the eye E to be inspected. Images can be generated.

The tomographic image generation unit 212 generates a tomographic image of the fundus Ef of the eye E to be inspected based on the interference signal from the line sensor 154 acquired by the acquisition unit 201. The tomographic image generation unit 212 can obtain information in the depth direction at one point of the fundus Ef by wavelength-decomposing the interference signal acquired from the line sensor 154 by Fourier transform. Further, by controlling the X scanner 122-1 and the Y scanner 122-2 by the device control unit 202 and scanning the measurement light on the fundus Ef in an arbitrary main scanning direction, the above processing is repeated for the interference signal obtained. , The tomographic image generation unit 212 can generate a two-dimensional tomographic image of the fundus Ef. Further, by moving the measurement light in a direction intersecting (orthogonal) with the main scanning direction (secondary scanning direction) and scanning the measurement light again in the main scanning direction, the above processing is repeated for the interference signal to obtain a fault. The image generation unit 212 can generate three-dimensional volume data (three-dimensional tomographic image) of the fundus Ef. The tomographic image generation unit 212 can also generate a two-dimensional tomographic image or three-dimensional volume data of the anterior segment Ea based on an interference signal when the measurement light is applied to the anterior segment Ea of the eye E to be inspected. ..

At this time, due to the characteristics of the Fourier transform, a folded image is generated in the tomographic image near the optical path length where the optical path length difference between the reference light and the measurement light becomes zero. The conditions under which the folded image is generated may be a case where the curvature of the fundus Ef is large and the optical path length of the measurement light varies greatly depending on the location, or a case where the optical path length adjustment of the reference light by the reference mirror 133 is insufficient. When a folded image is generated, the structure of the retina or the like in the tomographic image becomes unclear, and it becomes difficult to analyze the layer thickness and generate a layer thickness map, which will be described later, so that the diagnostic accuracy is lowered.

The arithmetic processing unit 213 performs the folding reduction processing for reducing the folding image in the tomographic image by using the trained model by the method described later. The arithmetic processing unit 213 can generate a tomographic image in which the folded image is reduced by the folded reduction process.

The analysis unit 214 can perform image analysis processing on a tomographic image or the like with a reduced folded image generated by the arithmetic processing unit 213. Further, the analysis unit 214 performs image analysis processing on various front images generated by the front image generation unit 211, tomographic images generated by the tomographic image generation unit 212, and various medical images acquired from an external device or the like by the acquisition unit 201. Can also be done. More specifically, the analysis unit 214 performs segmentation processing on, for example, a tomographic image in which the folded image is reduced, and detects the boundary of each layer of the retina of the eye to be inspected. The analysis unit 214 can calculate the thickness of the layer at each position of the fundus Ef from the detected boundary data.

Any known method can be used for the segmentation process. In this embodiment, the analysis unit 214 performs rule-based segmentation processing. Here, the rule-based process refers to a process that utilizes a known regularity, for example, a known regularity such as the regularity of the shape of the retina.

Specifically, the analysis unit 214 applies a median filter and a Sobel filter to the average tomographic image to generate an image (hereinafter, also referred to as a median image and a Sobel image, respectively). Next, the analysis unit 214 generates a profile for each data corresponding to the A scan from the generated median image and the Sobel image. The generated profile is a brightness value profile in the median image and a gradient profile in the Sobel image. Then, the analysis unit 214 detects the peak in the profile generated from the Sobel image. The analysis unit 214 can detect the boundary of each region of the retinal layer by referring to the profile of the median image corresponding to before and after the detected peak or between the peaks. The method of segmentation processing is an example.

The control unit 200 may be configured by a general-purpose computer including an arithmetic unit and a memory, or may be configured by a dedicated computer of the OCT device. Further, each component of the control unit 200 may be composed of a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The processor may be, for example, a GPU (Graphical Processing Unit), an FPGA (Field-Programmable Gate Array), or the like. In addition, each component may be configured by a circuit or the like that performs a specific function such as an ASIC.

3A and 3B show an example of display screens 300 and 310 displayed on the display unit 250 by the display control unit 203 according to the present embodiment. The display screen 300 shown in FIG. 3A shows a tomographic image 301, a front image 302, and an analysis result 305 that have not been subjected to the folding reduction processing.

The tomographic image 301 is a tomographic image of the eye E to be inspected generated by the tomographic image generation unit 212. In the example shown in FIG. 3A, folded images are generated at both ends of the tomographic image 301, and the structure of the retina is unclear. The front image 302 is a front image of the eye E to be inspected generated by the front image generation unit 211. The tomographic image 301 and the frontal image 302 may be tomographic images acquired by the acquisition unit 201 from an external device as long as they correspond to each other. The analysis result 305 shows the result of image analysis of the tomographic image 301 or the front image 302 by the analysis unit 214.

The indicator 303 indicates the position of the tomographic image 301 on the fundus Ef. The examiner can change the tomographic image 301 displayed on the display screen 300 by moving the indicator 303 up and down by an input means such as a mouse (not shown). Further, the indicator 304 indicates the range in which the three-dimensional tomographic image is acquired.

The fold-back reduction button 306 is a button indicating whether or not to display an image that has undergone fold-back reduction processing, and is a button that indicates a tomographic image display mode. When the wrapping reduction button 306 is displayed as OFF, it indicates that an image in which the wrapping reduction processing has not been performed is displayed. When the examiner presses a button with an input unit (not shown), the display control unit 203 causes the display unit 250 to display the image for which the folding reduction process has been performed.

FIG. 3B shows a display screen 310 including a tomographic image 311 that has been subjected to turn-back reduction processing by the control unit 200 by a method described later. The tomographic image 311 is different from the tomographic image 301 in that the central portion of the fundus Ef is equivalent to the tomographic image 301, but the folded images are removed at both ends. As a result, the examiner can confirm the clear structure of the retina, and the diagnostic accuracy is improved. Further, by performing image analysis on the tomographic image 301, it is possible to obtain a highly reliable analysis result 315 for, for example, the layer thickness, and the diagnostic accuracy is improved.

The fold reduction button 316 is displayed as ON, which indicates that a tomographic image in which the fold reduction process has been performed is displayed. By pressing a button with an input unit (not shown), the examiner can instruct the display control unit 203 to display an image that has not been subjected to the folding reduction processing. The message 317 is a message (text) displayed only when the wrapping reduction button 316 is ON. Message 317 is displayed to explain to the examiner that the tomographic image 311 is an image generated using a trained model described later (for example, a trained model by deep learning).

Next, with reference to FIG. 4, an example in which the machine learning model used by the arithmetic processing unit 213 according to the present embodiment is configured by a convolutional neural network (CNN) will be described. FIG. 4 shows an example of the configuration 410 of the trained model used by the arithmetic processing unit 213 in order to reduce the folding back. As the machine learning model according to the present embodiment, for example, FCN (Full Convolutional Network), SegNet, or the like can be used.

The machine learning model shown in FIG. 4 is composed of a plurality of layers responsible for processing and outputting an input value group. The types of layers included in the configuration 410 of the machine learning model include a convolution layer, a Downsampling layer, an Upsampling layer, and a Merger layer.

The convolutional layer is a layer that performs convolutional processing on an input value group according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image.

The downsampling layer is a layer that performs processing to reduce the number of output value groups to be smaller than the number of input value groups by thinning out or synthesizing input value groups. Specifically, as such a process, for example, there is a Max Polling process.

The upsampling layer is a layer that performs processing to increase the number of output value groups to be larger than the number of input value groups by duplicating the input value group or adding the interpolated value from the input value group. Specifically, as such a process, for example, there is a linear interpolation process.

The composite layer is a layer in which a value group such as an output value group of a certain layer or a pixel value group constituting an image is input from a plurality of sources, and the processing is performed by concatenating or adding them.

Note that if the parameter settings for the layers and nodes that make up the neural network are different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may differ. That is, in many cases, the appropriate parameters differ depending on the embodiment, and therefore, the values can be changed to preferable values as needed.

In addition to the method of changing the parameters as described above, there are cases where the CNN can obtain better characteristics by changing the configuration of the CNN. Better characteristics include, for example, more accurate alignment position information output, shorter processing time, shorter training time for machine learning models, and the like.

The CNN configuration 410 used in the present embodiment has a function of an encoder composed of a plurality of layers including a plurality of downsampling layers and a function of a decoder composed of a plurality of layers including a plurality of upsampling layers. It is a net type machine learning model. In the U-net type machine learning model, position information (spatial information) that is ambiguous in a plurality of layers configured as encoders is displayed in layers of the same dimension (layers corresponding to each other) in a plurality of layers configured as a decoder. ) (For example, using a skip connection).

Although not shown, as an example of changing the configuration of the CNN, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolutional layer. Good. Through these steps of CNN, the features of the captured image can be extracted.

When input data is input to the trained model, output data according to the design of the trained model is output. The trained model outputs, for example, output data that is likely to correspond to the input data according to the tendency trained with the trained data. The trained model according to the present embodiment is configured to be able to output a tomographic image 402 in which the folded image is removed and the folded image is reduced when the tomographic image 401 is input according to the learning tendency. There is.

Here, the learning data regarding the trained model according to the present embodiment will be described. The training data is composed of a pair group of input data and output data. In the present embodiment, the tomographic image is used as input data, and the tomographic image with reduced folded images is used as output data. The method of creating the learning data will be described later. Further, the input data of the training data according to the present embodiment may include a tomographic image in which no folded image is generated, and the output data in this case may be a tomographic image used as the input data. As a result, the trained model according to the present embodiment appropriately obtains a tomographic image without a folded image without performing unnecessary folding reduction processing when a tomographic image without a folded image is input. Can be output to.

FIG. 5 is a flowchart showing a series of processes from shooting to display by the control unit 200. In step S501, the acquisition unit 201 acquires the data of the interference signal when the device control unit 202 controls the optical head unit 100 and scans the fundus Ef of the eye E to be inspected. After that, the tomographic image generation unit 212 generates a tomographic image of the eye E to be inspected based on the acquired interference signal data, and the acquisition unit 201 acquires the tomographic image. The acquisition unit 201 may acquire an interference signal or a tomographic image from an external device. Further, the acquisition unit 201 may acquire output signals from the APD 115 or the infrared CCD 142 from the optical head unit 100, and the front image generation unit 211 may generate and acquire a front image based on these output signals. The acquisition unit 201 may acquire these output signals or front images from an external device.

In step S502, the arithmetic processing unit 213 generates a tomographic image 402 in which the folded image is reduced from the tomographic image 401 acquired in step S601 using the trained model (folded back reduction process). Specifically, the arithmetic processing unit 213 inputs the tomographic image 401 into the trained model, and generates the tomographic image 402 based on the output of the trained model.

In step S503, the analysis unit 214 performs image analysis processing such as layer analysis of the tomographic image 402 in which the folded image is reduced, and calculates the analysis result such as the layer thickness. In addition, the analysis unit 214 can perform arbitrary image analysis on the tomographic image and the frontal image acquired in step S501.

In step S504, the display control unit 203 displays the tomographic image 402 generated in step S502, the analysis result calculated in step S503, and the front image on the display unit 250 as shown in FIG. 3 (b). The display control unit 203 sets the tomographic image in which the folded image is not reduced, the analysis result thereof, and the tomographic image in which the folded image is reduced in response to the operator's instruction such as the operation of the folded-back reduction buttons 306 and 316. The analysis result can be switched and displayed on the display unit 250. When the display process by the display control unit 203 is completed in step S504, a series of processes by the control unit 200 is completed.

According to the above embodiment, it is possible to acquire a tomographic image in which the folded image is reduced by using the trained model, and by providing such a tomographic image to the examiner, the diagnostic accuracy can be improved. it can. Moreover, based on such a tomographic image, highly reliable analysis results can be obtained. Therefore, the diagnostic accuracy can be further improved by providing the examiner with the tomographic image obtained by the folding reduction process according to the present embodiment and the analysis result thereof.

Next, learning when preparing a trained model will be described with reference to FIGS. 6 to 8. The machine learning model may be learned in advance, and the arithmetic processing unit 213 according to the present embodiment may perform turn-back reduction processing using the learned model that has already been learned. The learning data is not limited to the data acquired by using the optical head unit 100, and may be the data acquired by an apparatus or the like having the same type of measurement optical system.

Here, a learning device that performs learning to generate a trained model used for the turnaround reduction process will be described with reference to FIG. The learning device is provided with a learning unit 600. The learning device may be a device different from the control unit 200, but the control unit 200 may be provided with the learning unit 600. When the learning unit 600 is provided in the control unit 200, the control unit 200 can use the learned model generated by the learning unit 600 in advance for the folding reduction processing, and learns when performing the folding reduction processing. It is not necessary to generate a completed model one by one.

The learning device may be configured by a general-purpose computer including an arithmetic unit and a memory, or may be configured by a dedicated computer of the OCT device. Further, each component of the learning device may be composed of a software module executed by a processor such as a CPU, MPU, GPU, or FPGA. In addition, each component may be configured by a circuit or the like that performs a specific function such as an ASIC.

FIG. 6 shows a configuration example of the learning unit 600. The learning unit 600 processes the entire learning. The learning unit 600 is provided with an image storage unit 601, a learning data generation unit 602, and a learning processing unit 603. The image storage unit 601 is a storage for storing a plurality of tomographic images for generating learning data used for learning. The image storage unit 601 may be a memory, a portable medium, or the like as long as it is a medium capable of storing a tomographic image. The image storage unit 601 stores a tomographic image in which no folded image of the eye to be inspected is generated. It is desirable that many tomographic images are stored, but the number may be sufficient for learning.

The learning data generation unit 602 generates learning data of the machine learning model by a method described later using the image stored in the image storage unit 601. The learning processing unit 603 performs learning processing on the machine learning model using the learning data generated by the learning data generation unit 602. The machine learning model in which the learning process is performed has the same configuration as the configuration 410 shown in FIG. 4, but is a CNN in a state where the parameters have not been trained.

FIG. 7 is a diagram for explaining a method of generating learning data by the learning data generation unit 602. The tomographic image 701 is a tomographic image without folding back, which is stored in the image storage unit 601 in advance. The method for obtaining the tomographic image 701 may be any method as long as it is a method for obtaining a tomographic image without a folded image. In the present embodiment, the line sensor 154 is replaced with a line sensor having twice the number of pixels, and a special OCT device that doubles the imaging range in the depth direction is used to acquire a tomographic image without a folded image.

Further, for example, the reference mirror 133 is driven to change the imaging position in the depth direction to perform imaging, and a plurality of tomographic images having different imaging positions in the depth direction are acquired, and they are combined and processed to return. An imageless tomographic image 701 may be acquired. Further, for example, a tomographic image having a small curvature and no folded image may be selected and used as the tomographic image 701.

The learning data generation unit 602 generates a tomographic image 702 as input data of learning data and a tomographic image 703 as output data from the tomographic image 701. Specifically, the training data generation unit 602 divides the tomographic image 701 into upper and lower halves, flips the upper half image upside down, and superimposes it on the lower half image, so that the tomographic image becomes input data of the training data. Generate 702.

However, in the portion where the folded image of the actual tomographic image is generated, the optical path length difference between the measurement light and the reference light is opposite, so that an image in which the phase correction of signal processing is applied in reverse is generated. Therefore, the learning data generation unit 602 can flip the upper half of the tomographic image 701 upside down, apply the phase correction of signal processing in the opposite direction, and then superimpose the image 701.

The phase correction may be performed by a known filter process. Further, even if the tomographic image is inverse Fourier transformed and returned to the interference signal, a tomographic image to which the phase correction of signal processing is applied in the reverse direction is generated, and the tomographic image 702 is generated using the tomographic image and the tomographic image 701. Good. When the image storage unit 601 has an interference signal corresponding to the tomographic image 701, a tomographic image to which the dispersion compensation process corresponding to the phase correction is applied on the positive side and a tomographic image to which the tomographic image is applied on the negative side are generated. Then, the tomographic image 702 may be generated by using the upper half and the lower half of these tomographic images.

Further, the learning data generation unit 602 cuts out the lower half of the tomographic image 701 and generates a tomographic image 703 as output data of the learning data. The learning data generation unit 602 can also adjust the tomographic image without a folded image to an image size corresponding to the number of pixels of the line sensor 154 and use it as input data and output data of the learning data. As a result, the trained model trained using the training data can output an appropriate tomographic image without performing unnecessary fold reduction processing when a tomographic image without folds is input. it can.

The method of generating the learning data is an example, and may be changed according to a desired configuration. For example, when a tomographic image with small curvature and no folded image is selected and used as the tomographic image 701, the tomographic image 701 is divided into upper and lower parts at an arbitrary position, the upper image is inverted upside down, and then a signal is signaled. The phase correction of the processing can be applied in the opposite direction and superimposed. In this case, the generated tomographic image is padded or the like so that the image size in the depth direction of the generated tomographic image becomes the image size corresponding to the number of pixels of the line sensor 154 used in the optical head unit 100. Good. Further, the same processing can be performed when the tomographic image 701 obtained by synthesizing a plurality of tomographic images having different imaging positions in the depth direction is used. If the image size of the generated tomographic image in the depth direction is larger than the image size corresponding to the number of pixels of the line sensor 154 used in the optical head unit 100, the generated tomographic image is trimmed. Good.

The learning processing unit 603 performs learning processing of the machine learning model by using the tomographic image 702 and the tomographic image 703 generated by the learning data generation unit 602 as input data and output data of the learning data. The input data and the output data may not be the entire image but may be data in a rectangular region included in the image. In this case, it is assumed that the input data and the output data are cut out from regions corresponding to each other. Since such training data focuses on the local region, there is an advantage that folding back can be reduced without being affected by the global structure such as the state of curvature of the fundus. Further, in the original tomographic image, by creating a large number of pairs of rectangular area images while changing the positions of the areas to different coordinates, it is possible to enhance the pair group constituting the learning data.

When using a trained model trained using such training data, the arithmetic processing unit 213 corresponds the tomographic image as input data with the training data when performing the folding reduction processing. It is divided into area images for each image size and input to the trained model. After that, the arithmetic processing unit 213 arranges and combines each of the rectangular region image groups output from the trained model in the same positional relationship as each of the rectangular region image groups input to the trained model. As a result, the arithmetic processing unit 213 can generate a tomographic image with a reduced number of folded images corresponding to the tomographic image as input data.

FIG. 8 is a flowchart of a series of processes performed by the learning unit 600. First, in step S801, the learning unit 600 reads all the tomographic images without folding back from the image storage unit 601.

In step S802, the learning data generation unit 602 generates a tomographic image 702 which is input data of the learning data for all the tomographic images read by the learning unit 600 by the above-mentioned method. Further, in step S803, the learning data generation unit 602 generates a tomographic image 703 which is the output data of the learning data for all the tomographic images read by the learning unit 600 by the above-mentioned method.

In step S804, the learning processing unit 603 compares the tomographic image output by the machine learning model in which the tomographic image 702 is input with the tomographic image 703. Here, the learning processing unit 603 may use the loss function to calculate the difference between the tomographic image output from the machine learning model and the tomographic image 703 which is the output data of the learning data. After that, the learning processing unit 603 adjusts parameters such as the coupling weighting coefficient of each layer shown in FIG. 4 so that the difference between the tomographic image output by the machine learning model and the tomographic image 703 which is the output data of the learning data becomes small. ·Update.

In the present embodiment, a method generally used in deep learning such as an error backpropagation method is used as the parameter optimization method, but other known methods may be used as long as the parameter optimization method can reduce the turnaround. Good. Here, the error backpropagation method is a method of adjusting the coupling weighting coefficient and the like between the nodes of each neural network so that the above difference becomes small. The learning processing unit 603 repeatedly adjusts the parameters for all the input data and output data pairs of the learning data, and optimizes the parameters.

The configuration of the machine learning model may be different from that shown in FIG. 4, and for example, the type, number, and order of the layers may be different. For example, as described above, since the phase correction is applied to the folded portion in the opposite direction, the folded portion tends to have streaks in the vertical direction. Therefore, a special convolutional layer that captures the characteristics of this vertical streak may be added to the configuration of the machine learning model.

In addition, rule-based processing and layers focusing on the curved structure of the tomographic image may be added to the configuration of the machine learning model. For example, it may include a process of detecting the inclination of the tomographic image and determining the point where the inclination changes discontinuously as the start point of the turnaround.

In addition, the learning processing unit 603 may use a hostile generative network (GAN: Generative Adversarial Networks). When GAN is used, the learning processing unit 603 provides an authenticity evaluation unit using a learned model that determines whether the tomographic image output by the machine learning model being trained is an image generated by deep learning. Can have. The learning processing unit 603 learns from the machine learning model so as to generate a more natural tomographic image by mutually feeding back the machine learning model used by the authenticity evaluation unit and the machine learning model being trained. Can be done.

When the learning process by the learning process unit 603 is completed in step S804, a series of processes by the learning unit 600 is completed. According to such processing of the learning unit 600, when a tomographic image in which a folded image is generated is input, it is possible to generate a trained model trained to output a tomographic image in which the folded image is reduced. ..

In addition, additional learning may be performed in parallel with OCT imaging. In that case, the image processing unit 210 determines whether or not the tomographic image generated by the tomographic image generation unit 212 has a folded image, sends only the image without the folded image to the image storage unit 601, and adds it by the learning unit 600. It is possible to generate training data for. By repeating the additional learning with the continuation of OCT imaging, it is possible to improve the processing accuracy of the folding reduction processing.

As described above, the control unit 200 according to the present embodiment includes the acquisition unit 201 and the arithmetic processing unit 213. The acquisition unit 201 acquires a tomographic image 401 (first tomographic image) of the eye E to be inspected. The arithmetic processing unit 213 uses the trained model to generate a tomographic image (second tomographic image) in which the folded image in the tomographic image 401 is reduced from the tomographic image 401. In particular, in the present embodiment, the arithmetic processing unit 213 generates a tomographic image 402 in which the folded image is removed as a tomographic image in which the folded image is reduced.

According to this configuration, it is possible to acquire a tomographic image in which the folded image is reduced by using the trained model, and by providing such a tomographic image to the examiner, the diagnostic accuracy can be improved. Moreover, based on such a tomographic image, highly reliable analysis results can be obtained. Therefore, the diagnostic accuracy can be further improved by providing the examiner with the tomographic image obtained by the folding reduction process according to the present embodiment and the analysis result thereof.

In addition, the control unit 200 further includes an analysis unit 214 that analyzes a medical image of the eye to be inspected. The analysis unit 214 detects the layer boundary in at least one of the tomographic image 401 and the tomographic image in which the folded image is reduced, and outputs the image analysis result. The control unit 200 can obtain a highly reliable analysis result by obtaining the image analysis result of the tomographic image in which the folded image is reduced by the analysis unit 214.

Further, the control unit 200 further includes a display control unit 203 that switches between the tomographic image 401 and the tomographic image in which the folded image is reduced and displays them on the display unit 250 according to the display mode. The display control unit 203 switches between the analysis result of the tomographic image 401 and the analysis result of the tomographic image in which the folded image is reduced according to the display mode, and displays it on the display unit 250. Further, the display control unit 203 sends a sentence explaining the tomographic image together with the tomographic image in which the folded image is reduced, for example, a sentence explaining that the tomographic image is a tomographic image generated by using the trained model to the display unit 250. It can be displayed. In this case, the diagnostic efficiency can be improved by presenting the examiner with an explanation of the tomographic image.

Further, the trained model is a machine learning model in which training is performed using training data in which a tomographic image is used as input data and a tomographic image without a folded image corresponding to the tomographic image is used as output data. The learning data is data generated using a tomographic image that does not include a folded image. Therefore, the arithmetic processing unit 213 according to the present embodiment can obtain the tomographic image 402 from which the folded image is removed according to the learning tendency by using the trained model.

The input data and output data of the training data of the trained model are not limited to tomographic images. For example, the data of the interference signal acquired from the line sensor may be used as the input data, and the data of the interference signal with the reduced folded image may be used as the output data. When such a trained model is used, the control unit 200 so that the arithmetic processing unit 213 performs the turnaround reduction processing before the signal processing for generating the tomographic image from the interference signal data by the tomographic image generation unit 212. Can be configured.

Further, in step S501, three-dimensional volume data including a plurality of two-dimensional tomographic images may be acquired. In this case, in step S502, the arithmetic processing unit 213 can perform the folding reduction processing on each of the two-dimensional tomographic images to generate a plurality of two-dimensional tomographic images in which the folding images are reduced. As a result, the arithmetic processing unit 213 can generate three-dimensional volume data in which the folded image is reduced.

Since the GPU can perform efficient calculations by processing more data in parallel with respect to the learning unit 600, the GPU is used when learning is performed a plurality of times using a learning model such as deep learning. It is effective to process with. Therefore, the GPU may be used in addition to the CPU for the processing by the learning unit 600. In this case, when the learning program including the learning model is executed, the CPU and the GPU cooperate to perform the learning to perform the learning. The processing of the learning unit 600 may be performed only by the CPU or GPU. Further, the arithmetic processing unit 213 (estimation unit) that executes the processing using the trained model may also use the GPU in the same manner as the learning unit 600.

Further, the learning unit 600 may include an error detecting unit and an updating unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error backpropagation method is a method of adjusting the coupling weighting coefficient between the nodes of each neural network so that the above error becomes small.

Further, the tomographic image 701 used for generating the learning data may include tomographic images taken in various situations. For example, the tomographic image 701 may include a tomographic image group in which the inclination of the image in the tomographic image is various angles. Further, the tomographic image 701 may include a tomographic image group to which various noise patterns are added.

(Embodiment 2)
In the first embodiment, the arithmetic processing unit 213 generated a tomographic image from which the fold image is removed as a tomographic image in which the fold image is reduced from the tomographic image in which the fold image is generated in the fold reduction process. On the other hand, in the folding reduction process according to the second embodiment, as a tomographic image in which the folding image is reduced, a tomographic image in which the folding image in the tomographic image is corrected to a normal image is generated.

Hereinafter, the OCT apparatus and the learning apparatus according to the present embodiment will be described with reference to FIGS. 9 to 11, focusing on the differences from the OCT apparatus and the learning apparatus according to the first embodiment. Since the configuration of the OCT apparatus according to the present embodiment is the same as the configuration of the OCT apparatus according to the first embodiment, the description will be omitted using the same reference number. Since the configuration of the learning device according to the present embodiment is the same as the configuration of the learning device according to the first embodiment, the description will be omitted using the same reference number.

FIG. 9 shows a flowchart of a series of processes by the control unit 200 according to the present embodiment. In the process by the control unit 200 according to the present embodiment, since steps S501 and S502 are the same steps as steps S501 and S502 according to the first embodiment, the description thereof will be omitted. In step S502, when the arithmetic processing unit 213 uses the trained model to generate a tomographic image from which the folded image is removed, the process proceeds to step S901.

In step S901, the arithmetic processing unit 213 performs image composition processing using the tomographic image acquired in step S501 and the tomographic image generated in step S502, and corrects the folded image in the tomographic image into a normal image. Generate. Hereinafter, the turn-back reduction process according to the present embodiment, including the processes of steps S501 and S901, will be described with reference to FIG. FIG. 10 is a diagram for explaining the folding back reduction process performed by the arithmetic processing unit 213.

The tomographic image 401 and the tomographic image 402 correspond to the input data and the output data related to the arithmetic processing using the trained model, and correspond to the tomographic image acquired in step S501 and the tomographic image generated in step S502, respectively. To do. Since the tomographic image 401 and the tomographic image 402 are the same as those shown in FIG. 4, the description thereof will be omitted.

When the image composition process in step S901 is started, the arithmetic processing unit 213 performs the subtraction process of the tomographic image 402 from the tomographic image 401 in step S911. As a result, the arithmetic processing unit 213 can generate a folded image 1001 composed of only the folded image.

Next, in step S912, the arithmetic processing unit 213 flips the folded image 1001 upside down to generate a tomographic image 1002. After that, in step S913, the arithmetic processing unit 213 connects the tomographic image 1002 to the upper part of the tomographic image 402 to correct the folded image to a normal image and generate the tomographic image 1003 in which the folded image is reduced. Can be done. In the tomographic image 1003, since the folded image is corrected to a normal image, it is possible to provide the examiner with tomographic information in a wider range than the tomographic image 402 according to the first embodiment. The arithmetic processing unit 213 may reapply the reverse phase correction to the tomographic image 1002. The process may be performed in the same manner as the process by the learning data generation unit 602 in the first embodiment.

In the OCT apparatus, the signal strength of the DC component of the Fourier transform is remarkably increased. Therefore, in the tomographic image 1003, white lines may be generated in the upper and lower central portions corresponding to the DC components. Therefore, the arithmetic processing unit 213 provides a mask region 1004 that does not display the tomography of the fundus by image processing in the upper and lower central portions of the tomographic image 1003. The position of the tomographic image 1003 provided with the mask region 1004 is a position corresponding to the coherence gate position where the optical path lengths of the measurement light and the reference light are equal.

When the tomographic image 1003 is generated by the folding reduction process of the arithmetic processing unit 213, the process proceeds to step S902. In step S902, the analysis unit 214 performs image analysis processing such as layer analysis on the tomographic image 1003, and calculates the analysis result such as the layer thickness. In addition, the analysis unit 214 may also perform image analysis on the tomographic image and the frontal image acquired in step S501. Here, in the tomographic image 1003, since the folded image is corrected to a normal image, an appropriate image analysis process can be performed on a wider range than the tomographic image 402 according to the first embodiment. The analysis unit 214 can be configured to ignore the mask region 1004 in the tomographic image 1003 in the image analysis.

In step S903, the display control unit 203 displays the tomographic image 1003 generated in step S901, the analysis result calculated in step S902, and the front image on the display unit 250. Further, the display control unit 203 can also display the tomographic image 401 acquired in step S501, the analysis result thereof, and the front image on the display unit 250. Here, FIGS. 11A and 11B show display screens 1100 and 1110 to be displayed on the display unit 250 by the display control unit 203 according to the present embodiment.

FIG. 11A shows a display screen 1100 including a tomographic image 1101 that has not been subjected to the folding reduction processing. The display screen 1100 shows a tomographic image 1101, a front image 1102, and an analysis result 1105 that have not been subjected to the folding reduction processing. Since the indicators 1103 and 1104 and the folding reduction button 1106 are the same as the indicators 303 and 304 and the folding reduction button 306 shown on the display screen 300 according to the first embodiment, the description thereof will be omitted.

FIG. 11B shows a display screen 1110 including a tomographic image 1111 that has undergone folding reduction processing. The display screen 1110 shows a tomographic image 1111, a front image 1102, and an analysis result 1115 that have undergone folding reduction processing. Here, the tomographic image 1111 and the analysis result 1115 correspond to the tomographic image 1003 generated in step S901 and the analysis result calculated in step S902. Further, the mask region 1118 in the tomographic image 1111 corresponds to the mask region 1004 in the tomographic image 1003. Since the wrap-around reduction button 1116 and the message 1117 are the same as the wrap-around reduction button 316 and the message 317 shown on the display screen 310 according to the first embodiment, the description thereof will be omitted.

In addition, the display control unit 203 responds to the operator's instructions such as the operation of the folding reduction buttons 1106 and 1116 to obtain a tomographic image in which the folding image is not reduced, an analysis result thereof, and a tomographic image in which the folding image is reduced. The analysis result can be switched and displayed on the display unit 250. When the display process by the display control unit 203 is completed in step S903, a series of processes by the control unit 200 is completed.

As described above, the arithmetic processing unit 213 according to the present embodiment inputs the tomographic image 401 into the trained model, and the tomographic image 402 (third tomographic image 402) from which the folded image in the tomographic image 401 is removed from the trained model. Image) is acquired. Further, the arithmetic processing unit 213 flips the difference between the tomographic image 401 and the tomographic image 402 (folded image 1001) upside down to generate the tomographic image 1002 (fourth tomographic image). Further, the arithmetic processing unit 213 generates a tomographic image 1003 (second tomographic image) by connecting the tomographic image 402 and the tomographic image 1002. Here, the tomographic image 1003 is a tomographic image in which the folded image in the tomographic image 401 appears as a normal image.

According to the folding reduction processing according to the present embodiment, the folding image is reduced by using the trained model, and a tomographic image obtained by modifying the folding image into a normal image can be acquired. Therefore, it is possible to provide the examiner with appropriate tomographic information for a wide range of the tomographic image, and it is possible to improve the diagnostic accuracy. In addition, based on such a tomographic image, highly reliable analysis results can be obtained even in the region where the folded image was generated. Therefore, the diagnostic accuracy can be further improved by providing the examiner with the tomographic image obtained by the folding reduction process according to the present embodiment and the analysis result thereof.

Further, in the present embodiment, the tomographic image 1003 generated by the arithmetic processing unit 213 is provided with a mask region 1004 at a position corresponding to the coherence gate position. This makes it possible to prevent white lines from being generated in the upper and lower central portions corresponding to the DC components in the tomographic image 1003. Further, by the folding reduction processing using the trained model, it is possible to prevent an image showing a signal that is not originally acquired from appearing in the same region.

The method of obtaining the tomographic image 1003 in which the folded image is corrected may be different from the folded reduction process described with reference to FIG. For example, the trained model may be trained using the training data using both the tomographic image 402 and the folded image 1001 as output data. In this case, the learning data generation unit 602 generates tomographic images 702 and 703 from the tomographic image 701 as in the first embodiment. In addition, the learning data generation unit 602 can generate a folded tomographic image in which the upper half of the tomographic image used when generating the tomographic image 702 is inverted upside down. Here, the tomographic image 703 and the tomographic image generated by the learning data generation unit 602 correspond to the tomographic image 402 and the tomographic image 1001, respectively. Further, the learning processing unit 603 can perform the learning processing of the machine learning model by using the tomographic image 703 and the folded tomographic image generated by the learning data generation unit 602 as the output data of the learning data. By using the trained model that has been trained in this way, the arithmetic processing unit 213 can generate the tomographic image 402 and the folded image 1001 from the tomographic image 401.

As in the first embodiment, the rectangular area image may be used as the input data and the output data of the learning data. In this case, the learning data generation unit 602 may use the rectangular region image of the tomographic image 703 and the rectangular region of the folded tomographic image at the positions corresponding to the rectangular region image as the input data as output data.

Similarly, for example, the trained model may be trained using the training data using both the tomographic image 402 and the tomographic image 1002 as output data. In this case, the learning data generation unit 602 generates tomographic images 702 and 703 from the tomographic image 701 as in the first embodiment. Further, the learning data generation unit 602 also uses the tomographic image of the upper half used when generating the tomographic image 702 as the output data of the learning data. Here, the tomographic image 703 and the tomographic image of the upper half generated by the learning data generation unit 602 correspond to the tomographic image 402 and the tomographic image 1002, respectively. Further, the learning processing unit 603 can perform the learning processing of the machine learning model by using the tomographic image 703 generated by the learning data generation unit 602 and the tomographic image of the upper half as the output data of the learning data. By using the trained model that has been trained in this way, the arithmetic processing unit 213 can generate the tomographic image 402 and the tomographic image 1002 from the tomographic image 401.

As in the first embodiment, the rectangular area image may be used as the input data and the output data of the learning data. In this case, the learning data generation unit 602 may use the rectangular region image of the tomographic image 703 at the position corresponding to the rectangular region image as the input data and the rectangular region of the upper half of the tomographic image as output data.

Further, the trained model may be trained using the training data in which the tomographic image 1003 in which the folded image is corrected is used as the output data of the training data. More specifically, the trained model uses the training data using the tomographic image 401 as the input data and the tomographic image 1003 corresponding to the tomographic image 401 in which the folded image appears as a normal image as the output data. You may study. In this case, the learning data generation unit 602 generates the tomographic image 702 as in the first embodiment, but the tomographic image 703 does not have to be generated. The learning processing unit 603 uses the tomographic image 702 generated by the learning data generation unit 602 as input data for training data, and uses the tomographic image 701 stored in the image storage unit 601 as output data for learning data to perform machine learning. The model can be trained. By using the trained model that has been trained in this way, the arithmetic processing unit 213 can directly generate the tomographic image 1003 in which the folded image is corrected from the tomographic image 401. In this case, step S901 may be omitted.

Even when the arithmetic processing unit 213 directly generates the tomographic image 1003 from the tomographic image 401 using the trained model, the rectangular region images of the tomographic image 702 and the tomographic image 701 are used as input data and output data of the training data. Can be used. In this case, first, a trained model is prepared in which the image of the rectangular region in the upper half image of the tomographic image 701 corresponding to the rectangular region image of the tomographic image 702 used as the input data is trained as the output data. .. Similarly, a trained model is prepared in which the image of the rectangular region in the lower half image of the tomographic image 701 corresponding to the rectangular region image of the tomographic image 702 used as the input data is trained as the output data.

The arithmetic processing unit 213 divides the tomographic image to be the input data into each rectangular region and inputs it to each of these trained models. After that, the arithmetic processing unit 213 sets each of the rectangular region image groups output from the trained model in the same positional relationship as each of the rectangular region image groups input to the trained model for each trained model. Place and combine. As a result, the arithmetic processing unit 213 can generate an image corresponding to the upper half image and the lower half image of the tomographic image 1003 in which the folded image is corrected. Therefore, by combining these, the tomographic image 1003 Can be generated.

It is also possible to generate the tomographic image 1003 directly from the tomographic image 401 using one trained model. In this case, as the output data of the training data, the rectangular region image in the upper half image and the lower half image of the tomographic image 701 corresponding to the rectangular region image which is the input data may be used.

Further, in the present embodiment, the arithmetic processing unit 213 provides a mask region 1004 that does not display the tomographic fault of the fundus by image processing in the upper and lower central portions (positions corresponding to the coherence gate positions) of the tomographic image 1003. On the other hand, the learning data generation unit 602 may use an image in which a mask region is provided at the upper end of the tomographic image 702 to generate output data of the learning data. In this case, the arithmetic processing unit 213 may provide a mask area at the lower end of the tomographic image 1002. Further, when directly obtaining the tomographic image 1003 in which the folded image is corrected by using the trained model, the training data generation unit 602 uses an image in which mask regions are provided in the upper and lower central portions of the tomographic image 701 as the training data. It may be used to generate output data.

(Embodiment 3)
In the second embodiment, the arithmetic processing unit 213 performs the folding reduction processing on the entire tomographic image as the input data. On the other hand, when there is a lesion in the macula in the affected eye, changes such as unevenness may occur in the central portion of the tomographic image in which the image of the macula is depicted. Therefore, it may not be necessary to perform image correction processing on the central portion of the tomographic image. Further, in the trained model, it is conceivable to modify only the vertical features of the tomographic image without modifying the horizontal features. In such a case, if the correction process is applied to the entire tomographic image, the image of the lesion in the central portion of the tomographic image may be changed.

Therefore, in the third embodiment, the folding reduction processing by the arithmetic processing unit 213 is applied to a region other than the central portion of the tomographic image as input data, particularly a region where a folding image is generated. Hereinafter, the OCT apparatus and the learning apparatus according to the present embodiment will be described with reference to FIGS. 12 and 13, focusing on the differences from the OCT apparatus and the learning apparatus according to the second embodiment. Since the configuration of the OCT apparatus according to the present embodiment is the same as the configuration of the OCT apparatus according to the second embodiment, the description will be omitted using the same reference number. Since the configuration of the learning device according to the present embodiment is the same as the configuration of the learning device according to the second embodiment, the description will be omitted using the same reference number.

FIG. 12 shows a flowchart of a series of processes by the control unit 200 according to the present embodiment. In the process by the control unit 200 according to the present embodiment, in step S501, the same process as that of the same step according to the first embodiment is performed, and thus the description thereof will be omitted. Further, FIG. 13 is a diagram for explaining a turn-back reduction process performed by the arithmetic processing unit 213, including the processes of steps S1201 to S1203. In FIG. 13, the tomographic image 401 corresponds to the input data related to the arithmetic processing using the trained model, and corresponds to the tomographic image acquired in step S501.

When the acquisition unit 201 acquires the tomographic image in step S501, the process proceeds to step S1201. In step S1201, the arithmetic processing unit 213 extracts images 1301, 1302 (folded area images) of the region where the folded image is generated from the tomographic image 401 acquired in step S501. The arithmetic processing unit 213 can extract an image of a predetermined region of the tomographic image as an image of a region where a folded image is generated. In the present embodiment, for example, two folded region images 1301 and 1302 having an image size of 1/4 of the lateral image size of the tomographic image are extracted from both ends in the lateral direction of the tomographic image. In addition, the arithmetic processing unit 213 can extract the image of the remaining region of the tomographic image as the image 1305 of the central region. The predetermined region is not limited to a region having an image size of 1/4 of the lateral image size of the tomographic image from both ends in the lateral direction of the tomographic image, and an arbitrary image size can be obtained according to a desired configuration. It may be a region having.

In step S1202, the arithmetic processing unit 213 uses the trained model to correct the folded image from each folded area image 1301, 1302 to a normal image, and generates region images 1303 and 1304 in which the folded image is reduced, respectively. To do. As the training data, the image of the region where the folded image occurs in the tomographic image 702 shown in FIG. 7 can be used as the input data, and the image of the corresponding region of the tomographic image 701 can be used as the output data. Regarding the region where the folded image related to the training data is generated, a predetermined region of the tomographic image, for example, a region having an image size of 1/4 of the lateral image size of the tomographic image from both ends in the lateral direction of the tomographic image. Can be.

In step S1203, image composition processing is performed using each region image 1303, 1304 in which the folded image is reduced and the image 1305 in the central region, and the entire tomographic image 1306 in which the folded image is corrected to a normal image is generated. .. More specifically, in step S1231 included in step S1203, the arithmetic processing unit 213 connects the area image 1303 to the left end of the image 1305 in the central region and connects the region image 1304 to the right end. The arithmetic processing unit 213 may perform padding processing for adding an arbitrary pixel value between the area images 1303 and 1304 in the upper part of the image 1305 in the central region. Further, the arithmetic processing unit 213 can provide the mask region 1307 in the tomographic image 1306 as in the third embodiment.

When the tomographic image 1306 is generated in step S1203, the process proceeds to step S902. Since steps S902 and S903 are the same steps as steps S902 and S903 according to the second embodiment, description thereof will be omitted. In step S903, when the display process is completed, a series of processes is completed.

As described above, the arithmetic processing unit 213 according to the present embodiment extracts the folded region images 1301, 1302 (images of the first region) of the region where the folded image is generated from the tomographic image 401. Further, the arithmetic processing unit 213 inputs the folded area images 1301 and 1302 into the trained model to obtain the area images 1303 and 1304 (second area image). The arithmetic processing unit 213 generates the tomographic image 1306 by connecting the area images 1303 and 1304 to the image 1305 in the area other than the area where the folded image is generated in the tomographic image 401. In particular, in the present embodiment, the arithmetic processing unit 213 extracts an image of a predetermined region in the tomographic image 401 as the folded region images 1301 and 1302.

According to the fold-back reduction process according to the present embodiment, the same effect as that of the fold-back reduction process according to the second embodiment can be obtained. Furthermore, even if there is a lesion in the macula in the affected eye, it is possible to suppress alteration of the lesion information by not performing unnecessary correction processing on the central part of the tomographic image in which the image of the macula is depicted. be able to.

As a method of generating training data, as described above, the training data generation unit 602 uses the image of the region where the folded image occurs in the tomographic image 702 shown in FIG. 7 as input data, and the corresponding region of the tomographic image 701. The image of may be generated as output data.

Further, in the present embodiment, the arithmetic processing unit 213 generated the area images 1303 and 1304 from the folded area images 1301 and 1302 by using the trained model. On the other hand, for the trained model, training may be performed using both the folded region images 1301 and 1302 as input data and the image of the region other than the image 1305 of the central region of the tomographic image 1306 as the output data. In this case, the learning data generation unit 602 may generate both the images of the region where the folded image occurs in the tomographic image 702 shown in FIG. 7 as input data. Further, the learning data generation unit 602 may generate an image of the region of the tomographic image 701 as output data other than the region corresponding to the image of the remaining region of the tomographic image 702 (the image of the region of the central portion).

In this case, the arithmetic processing unit 213 uses the trained model to generate an image of a region other than the region corresponding to the image 1305 of the central portion of the tomographic image 1306 from the folded region images 1301 and 1302. be able to. As a result, the arithmetic processing unit 213 can generate the tomographic image 1306 by connecting the generated image and the image 1305 in the central region. In this case, the padding process in the connection process (step S1231) can be omitted.

Further, the arithmetic processing unit 213 may perform image analysis processing such as segmentation processing on the tomographic image as input data, specify a region where a folded image is generated, and extract an image of the region. Any known method may be used for the segmentation process for identifying the region where the folded image is generated. In this case, the lateral size of the region extracted by the arithmetic processing unit 213 is indefinite because it is based on the folded image in the tomographic image.

Therefore, in such a case, the arithmetic processing unit 213 can divide the image of the extracted region into a plurality of rectangular region images having a predetermined image size and input the image to the trained model. Here, the rectangular area image may have an area overlapping with other rectangular area images. The arithmetic processing unit 213 arranges and combines each of the rectangular region image groups output from the trained model in the same positional relationship as each of the rectangular region image groups input to the trained model. As a result, the arithmetic processing unit 213 can correct the folded image to a normal image and generate a region image in which the folded image is reduced. Subsequent processing may be the same as in this embodiment. When combining each of the rectangular region image groups output from the trained model, the rectangular region image groups may be combined so that the overlapping regions in the rectangular region image overlap each other. In this case, in the regions that are superimposed on each other, the pixel value may be determined using only the pixel value of any one rectangular region image among the superimposed rectangular region images, or the superimposed rectangular region image may be determined. The average value, the median value, or the like of the pixel values may be determined as the pixel values.

In this case, as described in the second embodiment, learning is performed using the rectangular region image in the upper half image of the tomographic image 701 corresponding to the rectangular region image of the tomographic image 702 used as the input data as the output data. Prepare a trained model. Similarly, a trained model is prepared in which the image of the rectangular region in the lower half image of the tomographic image 701 corresponding to the rectangular region image of the tomographic image 702 used as the input data is trained as the output data.

The arithmetic processing unit 213 divides the tomographic image to be the input data into each rectangular region and inputs it to each of these trained models. After that, the arithmetic processing unit 213 sets each of the rectangular region image groups output from the trained model in the same positional relationship as each of the rectangular region image groups input to the trained model for each trained model. Place and combine. As a result, the arithmetic processing unit 213 can generate an image corresponding to the upper half image and the lower half image of the region image 1303, 1304 in which the folded image is corrected. Therefore, by combining these, the region can be generated. Images 1303 and 1304 can be generated.

It should be noted that the region images 1303 and 1304 can be directly generated from the folded region images 1301 and 1302 using one trained model. In this case, as the output data of the training data, the rectangular region image in the upper half image and the lower half image of the tomographic image 701 corresponding to the rectangular region image of the tomographic image 702 which is the input data may be used.

Further, in this case, the learning data generation unit 602 does not have to use the image of the portion of the tomographic images 701 and 702 in which the folded image does not occur, for example, the image of the central portion, for generating the learning data. As a result, the number of learning data can be reduced and the time required for the learning process can be shortened.

Further, when the arithmetic processing unit 213 performs segmentation processing on the tomographic image, identifies a region where a folded image occurs, and extracts an image of the region, the extracted image corresponds to the image size of the learning data. The padding process may be performed as described above. In this case, the image size of the folded area image input to the trained model by the arithmetic processing unit 213 can be made constant so as to correspond to the image size of the training data. Similar processing can be performed. The arithmetic processing unit 213 can perform trimming processing on the region image output from the trained model so as to have the image size before the padding processing, and connect the region image with the image in the central region.

The learning data in this case may be the same learning data as the learning data according to the present embodiment. The trained model may be trained so as to output the value as it is for the padded portion. In this case, for example, an image including a region having a pixel value similar to the pixel value used for the padding process may be used as the input data and the output data of the training data.

Even in these cases, the same effect as that of the present embodiment can be obtained. Therefore, even if there is a lesion in the macula in the affected eye, it is possible to suppress alteration of the lesion information by not performing unnecessary correction processing on the central part of the tomographic image in which the image of the macula is depicted. be able to.

Further, in the present embodiment, the arithmetic processing unit 213 provides a mask region 1307 in the upper and lower central portions of the tomographic image 1306 so that the tomography of the fundus is not displayed by image processing. On the other hand, the learning data generation unit 602 may use an image in which mask regions are provided in the upper and lower central portions of the tomographic image 701 to generate output data of the learning data.

In the present embodiment, the arithmetic processing unit 213 generated a tomographic image in which the folded image is corrected to a normal image. On the other hand, the arithmetic processing unit 213 may generate a tomographic image from which the folded image is removed, as in the first embodiment. In this case as well, it is possible to suppress alteration of the lesion information by not performing unnecessary processing on the central portion of the tomographic image in which the image of the macula is depicted. As the training data in this case, the image of the region where the folded image of the tomographic image 702 is generated may be used as the input data, and the image of the corresponding region of the tomographic image 703 may be used as the output data.

(Embodiment 4)
In the first to third embodiments, the control unit 200 automatically generates a tomographic image in which the folded image is reduced, and produces a tomographic image in which the folded image is not reduced and a tomographic image in which the folded image is reduced at the time of display. It was switched and displayed according to the instruction of the operator. On the other hand, in the fourth embodiment, the control unit determines whether or not to perform the turn-back reduction process according to the setting or the instruction of the operator.

Hereinafter, the OCT apparatus and the learning apparatus according to the present embodiment will be described with reference to FIG. 14, focusing on the differences from the OCT apparatus and the learning apparatus according to the second embodiment. Since the configuration of the OCT apparatus according to the present embodiment is the same as the configuration of the OCT apparatus according to the second embodiment, the description will be omitted using the same reference number. Since the configuration of the learning device according to the present embodiment is the same as the configuration of the learning device according to the second embodiment, the description will be omitted using the same reference number.

FIG. 14 shows a flowchart of a series of processes by the control unit 200 according to the present embodiment. In the process by the control unit 200 according to the present embodiment, steps S501 and S502 are the same as steps S501 and S502 according to the first embodiment, and thus description thereof will be omitted. Further, since steps S901 to S903 are the same as steps S901 to S903 according to the second embodiment, the description thereof will be omitted. In the present embodiment, when the tomographic image is acquired in step S501, the process proceeds to step S1401.

In step S1401, the image processing unit 210 determines whether or not the display mode of the control unit 200 is the folding reduction mode. Specifically, when the fold reduction button is ON in the previous shooting, it is determined that the fold reduction mode is set, and when the fold reduction button is OFF, it is determined that the mode is not the fold reduction mode. Here, the fold-back reduction button may be the same as the fold-back reduction buttons 1106 and 1116 shown in FIG.

Further, the image processing unit 210 may determine whether or not the folding reduction mode is performed based on the state of the tomographic image instead of the state of the folding reduction button. Specifically, the image processing unit 210 determines whether or not a folded image is generated in the tomographic image by image analysis such as segmentation processing, and if a folded image is generated, it is in the folded reduction mode, and the folded image is displayed. If it does not occur, it may be determined that the mode is not the return reduction mode. Any known method may be used for the segmentation process for determining whether or not a folded image is generated in the tomographic image.

Further, whether or not the wrapping reduction mode is set can be changed later by the examiner operating the wrapping reduction button after the display screen is displayed. Further, in step S1401, a folding reduction mode is selected in which the display unit 250 displays whether or not the display control unit 203 performs the folding reduction processing, and the image processing unit 210 performs the folding reduction processing according to the instruction of the operator. It may be determined whether or not it is.

If the image processing unit 210 determines in step S1401 that the control unit 200 is in the turn-back reduction mode, the process proceeds to step S502. On the other hand, if the image processing unit 210 determines in step S1401 that the control unit 200 is not in the folding reduction mode, the process proceeds to step S902. Since the processing after step S502 is the same as the processing according to the second embodiment, the description thereof will be omitted. When the processing shifts from step S1401 to step S902, in step S902, the analysis unit 214 may perform image analysis on a tomographic image, a frontal image, or the like that has not been subjected to the folding reduction processing. When the display process is completed in step S903, a series of processes is completed.

As described above, the arithmetic processing unit 213 according to the present embodiment generates the tomographic image 1003 in response to the setting or the instruction of the operator. Further, the arithmetic processing unit 213 may determine whether or not the tomographic image 401 includes the folded image, and may generate the tomographic image 1003 when it is determined that the tomographic image 401 includes the folded image.

As a result, the control unit 200 according to the present embodiment can execute the turn-back reduction process as needed, and can reduce the calculation load. Further, in the first to third embodiments, since the folding reduction process is automatically executed, the learning data is configured to include a tomographic image in which no folding image is generated. On the other hand, since it is assumed that the control unit 200 according to the present embodiment applies the folding reduction processing to the tomographic image in which the folding image is generated, it is not necessary to use the image in which the folding image is not generated in the learning data. May be good. Therefore, the number of learning data to be prepared can be reduced, and the learning time can be shortened.

The series of processes according to the present embodiment has been described mainly on the difference from the series of processes according to the second embodiment, but in the series of processes according to the first and third embodiments, the same return as that of the present embodiment has been described. It may be determined whether or not the reduction process is performed.

(Embodiment 5)
In the first to fourth embodiments, the configuration in which the analysis unit 214 performs image analysis and calculates the analysis result such as the layer thickness has been described. On the other hand, the analysis unit 214 may generate a map image such as a layer thickness map from a tomographic image or the like in which the folded image is reduced.

Here, the technique disclosed in Patent Document 1 described above makes it easy to identify a region where the reliability is lowered by the folded image in the layer thickness map, and is reliable with respect to the region where the folded image is generated. It did not provide high layer thickness data. Therefore, the technique disclosed in Patent Document 1 has a problem that the layer thickness information of the region where the folded image is generated cannot be used for diagnosis.

On the other hand, the control unit according to the first to fourth embodiments can acquire a highly reliable analysis result even in the region where the folded image is generated. Therefore, by using such an analysis result, a wide range can be obtained. It is possible to generate a map image showing a highly reliable analysis result. Hereinafter, the OCT apparatus according to the fifth embodiment, which generates a map image showing highly reliable analysis results over a wide range, will be described with reference to FIG. 15, focusing on the difference from the OCT apparatus according to the second embodiment.

Since the configuration of the OCT apparatus according to the present embodiment is the same as the configuration of the OCT apparatus according to the second embodiment, the description will be omitted using the same reference number. Since the configuration of the learning device according to the present embodiment is the same as the configuration of the learning device according to the second embodiment, the description will be omitted using the same reference number. Further, since the flow of a series of processes by the control unit is the same as the flow of a series of processes by the control unit according to the second embodiment, the description thereof will be omitted. However, in the present embodiment, in step S902, the analysis unit 214 generates a map image of the eye E to be inspected as an image analysis result based on the tomographic image subjected to the folding reduction processing. Further, the analysis unit 214 may perform image analysis on a tomographic image or a frontal image that has not been subjected to the folding reduction processing.

The analysis unit 214 according to the present embodiment generates a layer thickness map of the eye E to be inspected, which is a kind of map image, as an image analysis result. The layer thickness map is a map in which the thickness distribution of the fundus Ef is shaded or colored. The analysis unit 214 detects the boundary of each layer of the retina of the eye E to be examined by the segmentation process from each tomographic image of the three-dimensional volume data generated by the tomographic image generation unit 212. Since the segmentation process is the same process as that of the first embodiment, the description thereof will be omitted.

The analysis unit 214 calculates the thickness of each layer at each location of the fundus Ef from the detected boundary data. Then, the analysis unit 214 generates a layer thickness map in which shades and colors are added according to the thickness of each location of the fundus Ef.

Here, FIGS. 15A and 15B show an example of a display screen according to the present embodiment. FIG. 15A shows a display screen 1500 including a tomographic image 1501 that has not been subjected to the folding reduction processing. The display screen 1500 shows a tomographic image 1501, a front image 1502, and a layer thickness map 1504 that have not been subjected to the folding reduction processing. Since the indicator 1503 and the folding reduction button 1505 are the same as the indicator 1103 and the folding reduction button 1106 shown on the display screen 1100 according to the second embodiment, the description thereof will be omitted.

Here, the layer thickness map 1504 is a layer thickness map of the eye E to be inspected. The examiner can switch the layer to be displayed on the layer thickness map 1504 by a selection means such as a pull-down (not shown). The shade in the layer thickness map 1504 means that the darker the layer, the thicker the layer. In the layer thickness map 1504, the dark portion is circular, and the outside of the circle is white. The white part means that the analysis unit 214 failed to detect the thickness of the layer. This failure is due to the obscured layer structure due to the folded images seen at both ends of the tomographic image 1501.

On the other hand, FIG. 15B shows a display screen 1510 including a tomographic image 1511 that has undergone folding reduction processing. The display screen 1510 shows a tomographic image 1511, a front image 1502, and a layer thickness map 1514 that have undergone folding reduction processing. Here, the tomographic image 1511 is similar to the tomographic image 1111 shown in FIG. 11 (b). Further, the mask region 1517 in the tomographic image 1511 is the same as the mask region 1118 in the tomographic image 1111. Since the indicator 1513, the wrapping reduction button 1515, and the message 1516 are the same as the indicator 1113, the wrapping reduction button 1116, and the message 1117 shown on the display screen 1110 according to the second embodiment, the description thereof will be omitted.

In the layer thickness map 1514, the central portion is equivalent to the layer thickness map 1504, but the outer region also has a density, which is different from the layer thickness map 1504. This means that the analysis unit 214 succeeded in detecting the thickness of the layer up to the outer region. By generating a layer thickness map based on the tomographic image that has undergone the folding reduction process in this way, highly reliable layer thickness information can be obtained even in the region where the folding image is generated, and a wide range of layer thickness maps can be obtained. Can be generated. Therefore, by providing the layer thickness map 1514 to the examiner, the examiner can confirm the layer thickness of the retina in a wider range than the layer thickness map 1504, and the diagnostic efficiency can be improved.

As a diagnosis using a layer thickness map, for example, a diagnosis of glaucoma is known. It is known that the thickness distribution of a layer called ganglion cell complex (GCC) is important for the diagnosis of glaucoma. Therefore, as a layer thickness map showing the thickness of the layer of the ganglion cell complex, a layer thickness map based on the tomographic image subjected to the folding reduction treatment according to the present embodiment is used, and a layer with high reliability in a wide range. It is possible to obtain thickness information and improve the diagnostic efficiency of glaucoma.

As described above, the analysis unit 214 according to the present embodiment generates a map image using at least one of the tomographic image 401 and the tomographic image 1003 based on the detected layer boundary. Here, the map image includes a layer thickness map.

According to the folding reduction processing according to the present embodiment similar to the second embodiment, the folding image is reduced by using the trained model, and a tomographic image in which the folding image is corrected can be acquired. Therefore, it is possible to provide the examiner with appropriate tomographic information for a wide range of the tomographic image, and it is possible to improve the diagnostic accuracy. In addition, based on such a tomographic image, highly reliable analysis results can be obtained even in the region where the folded image was generated. Further, in the present embodiment, since a map image such as a layer thickness map can be provided as such an analysis result, the diagnostic accuracy can be improved by providing the examiner with highly reliable layer thickness information in a wide range. It can be improved further.

In this embodiment, a layer thickness map is taken as an example as a map image. However, the map image is not limited to this, and may be an image showing the analysis result using a tomographic image or the like for each position on the front image. For example, it may be a blood vessel density map showing the analysis result regarding the blood vessel density. As an analysis method for analyzing the blood vessel density from a tomographic image or the like, any known method may be used.

Further, the control unit according to the present embodiment has been described mainly on the difference from the control unit according to the second embodiment, but the control unit according to the first, third and fourth embodiments is also an analysis unit as in the present embodiment. 214 may be configured to generate a map image.

(Modification example 1)
In the above-described first to fifth embodiments, the machine learning model used as the trained model has a CNN configuration as shown in FIG. However, the configuration of the machine learning model is not limited to this. FIG. 16 shows another configuration example of the machine learning model used as the learned model for the folding reduction processing in the first to fifth embodiments.

The machine learning model shown in FIG. 16 is also composed of CNN. The configuration of the CNN includes a plurality of convolution processing blocks 1610 groups. The convolution processing block 1610 includes a Convolution layer 1611, a Batch Normalization layer 1612, and an activation layer 1613 using a rectifier liner unit.

The CNN configuration also includes a synthetic (Merge) layer 1620 and a final convolutional layer 1630. The composite layer 1620 synthesizes by connecting or adding the output value group of the convolution processing block 1610 and the pixel value group constituting the image. The final convolution layer 1630 outputs a group of pixel values constituting the tomographic image 402 synthesized by the composite layer 1620. In such a configuration, the pixel value group constituting the input tomographic image 401 is output via the convolution processing block 1610 group, and the pixel value group constituting the input tomographic image 1601 is a composite layer. Synthesized at 1620. After that, the combined pixel value group is formed into a tomographic image 402 in which the folded image is reduced in the final convolution layer 1630.

Even if the machine learning model having such a configuration is used, it is possible to generate a tomographic image in which the folded image is reduced, as in the first to fifth embodiments, and it is possible to improve the diagnostic accuracy.

(Modification 2)
In the fifth embodiment, the analysis unit 214 generated a layer thickness map using a tomographic image with reduced folds. On the other hand, the analysis unit 214 can also generate an En-Face image of the eye E to be inspected and an OCTA (OCT Angiography) front image by using each tomographic image of the three-dimensional volume data with reduced folding. In this case, the display control unit 203 can display the generated En-Face image and the OCTA front image on the display unit 250. In addition, the analysis unit 214 can also analyze the generated En-Face image and OCTA front image. At this time, the analysis unit 214 may obtain a blood vessel density map by analyzing the OCTA front image, for example.

Here, the En-Face image and the OCTA front image will be described. The En-Face image is a frontal image generated by projecting data in an arbitrary depth range in a three-dimensional tomographic image obtained by using optical interference in the XY directions. The front image is at least a part of the depth range of the volume data (three-dimensional tomographic image) obtained by using optical interference, and the data corresponding to the depth range determined based on the two reference planes is displayed on the two-dimensional plane. It is generated by projecting or integrating on.

For example, an En-Face image is generated by projecting data corresponding to a depth range determined based on a retinal layer detected by a segmentation process on a two-dimensional tomographic image of volume data onto a two-dimensional plane. be able to. As a method of projecting data corresponding to a depth range determined based on two reference planes on a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on the two-dimensional plane. Techniques can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range (depth range) in the depth direction of a region surrounded by two reference planes.

The depth range related to the En-Face image may be specified, for example, with reference to two layer boundaries relating to the retinal layer detected by the segmentation processing method described above or the segmentation processing using the learned model described later. Further, the depth range may be a range including a predetermined number of pixels in a deeper direction or a shallower direction with reference to one of the two layer boundaries relating to the retinal layer detected by these segmentation processes. Further, the depth range related to the En-Face image may be, for example, a range changed (offset) according to the instruction of the operator from the range between the two layer boundaries regarding the detected retinal layer. Good. At this time, the operator can change the depth range by, for example, moving an index indicating the upper limit or the lower limit of the depth range superimposed on the tomographic image.

The generated front image is not limited to the En-Face image (en-Face image of brightness) based on the brightness value as described above. The generated front image may be, for example, a motion contrast front image generated by projecting or integrating the data corresponding to the above-mentioned depth range on the motion contrast data between a plurality of volume data on a two-dimensional plane. Here, the motion contrast data is data indicating a change between a plurality of volume data obtained by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data showing changes between a plurality of tomographic images obtained at substantially the same position at different positions. The motion contrast front image is also referred to as an OCTA front image (OCTA En-Face image) relating to OCTA for measuring the movement of blood flow, and the motion contrast data is also referred to as OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals. , It may be obtained by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected.

Further, the three-dimensional OCTA data (OCT volume data) used when generating the OCTA front image is generated by using at least a part of the tomographic image or the interference signal common to the volume data including the tomographic image used for segmentation. It's okay. As a result, the volume data (three-dimensional tomographic image) and the three-dimensional OCTA data can correspond to each other. Therefore, using the three-dimensional motion contrast data corresponding to the volume data, for example, a motion contrast front image corresponding to a depth range determined based on the retinal layer detected by segmentation can be generated.

(Modification example 3)
In the above embodiment and the modified example, the analysis unit 214 is configured to perform rule-based segmentation. On the other hand, the analysis unit 214 may perform the segmentation process using the trained model. At this time, the analysis unit 214 may generate a label image from the tomographic image using the trained model. Here, the label image refers to a label image in which a region is labeled for each pixel of a tomographic image. Specifically, it is an image in which an arbitrary region is divided by a identifiable pixel value (hereinafter, label value) group among the region groups drawn in the acquired image. Here, the specified arbitrary region includes a region of interest (ROI: Region Of Interest), a volume of interest (VOI: Volume Of Interest), and the like.

By specifying the coordinate group of the pixel having an arbitrary label value from the image, the coordinate group of the pixel that depicts the corresponding region such as the retinal layer in the image can be specified. Specifically, for example, when the label value indicating the ganglion cell layer constituting the retina is 1, the coordinate group having the pixel value of 1 among the pixel groups of the image is specified, and the coordinate group corresponds to the coordinate group from the image. Extract the pixel group to be used. Thereby, the region of the ganglion cell layer in the image can be identified.

The segmentation process may include a process of reducing or enlarging the label image. At this time, the image complement processing method used for reducing or enlarging the label image shall use the nearest neighbor method or the like so as not to erroneously generate an undefined label value or a label value that should not exist at the corresponding coordinates. ..

Here, the segmentation process will be described in more detail. The segmentation process is a process for identifying an area called ROI or VOI, such as an organ or a lesion depicted in an image, for use in image diagnosis or image analysis. For example, according to the segmentation process, it is possible to identify the region group of the layer group constituting the retina from the image acquired by the OCT imaging of the back eye portion as the imaging target. If the area to be specified is not drawn in the image, the number of specified areas is 0. Further, as long as a plurality of region groups to be specified are depicted in the image, the number of the specified regions may be a plurality, or even one region surrounding the region groups so as to include the region groups. Good.

The specified area group is output as information that can be used in other processing. Specifically, for example, the coordinate group of the pixel group constituting each of the specified region groups can be output as a numerical data group. Further, for example, a coordinate group indicating a rectangular region, an ellipsoid region, a rectangular parallelepiped region, an ellipsoid region, or the like including each of the specified region groups can be output as a numerical data group. Further, for example, a coordinate group indicating a straight line, a curve, a plane, a curved surface, or the like corresponding to the boundary of the specified region group can be output as a numerical data group. Further, for example, a label image showing the specified region group can be output.

Here, as a machine learning model for segmentation, for example, a convolutional neural network (CNN) can be used. As the machine learning model according to this modification, for example, a CNN (U-net type machine learning model) as shown in FIG. 4 or a model combining CNN and LSTM (Long Short-Term Memory) can be used. Can be used. Further, FCN (Full Convolutional Network), SegNet, or the like can also be used as the machine learning model. Further, a machine learning model or the like that recognizes an object in a region unit can be used according to a desired configuration. As a machine learning model that recognizes an object in a region unit, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Further, as a machine learning model for recognizing an object in a region unit, YOLO (You Only Look Object) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used.

Further, as the training data of the machine learning model for segmentation, the tomographic image is used as the input data, and the label image in which the area is labeled for each pixel of the tomographic image is used as the output data. Label images include, for example, the inner limiting membrane (ILM), the nerve fiber layer (NFL), the ganglion cell layer (GCL), the photoreceptor inner segment outer segment junction (ISOS), the retinal pigment epithelial layer (RPE), and Bruch. Labeled images with labels such as membrane (BM) and choroid can be used. Other regions include, for example, the vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner nuclear layer (IPL), inner nuclear layer (INL), cornea, anterior chamber, and iris. And an image with a label such as a crystalline lens may be used.

Moreover, the input data of the machine learning model for segmentation is not limited to the tomographic image. It may be an anterior segment image, a frontal image obtained by using a fundus observation optical system, a fundus anterior image obtained by using a fundus camera or the like, or the En-Face image or OCTA frontal image described above. In this case, as the learning data, various images can be used as input data, and label images in which region names and the like are labeled for each pixel of various images can be used as output data. For example, when the input data of the training data is a frontal image of the fundus, the output data may be an image labeled with the peripheral portion of the optic disc, Disc, Cup, or the like.

The label image used as the output data may be an image in which each region is labeled in the tomographic image by a doctor or the like, or an image in which each region is labeled by the rule-based region detection process. There may be. However, if machine learning is performed using a label image that is not properly labeled as the output data of the training data, the image obtained by using the trained model trained using the training data is also properly labeled. There is a possibility that the label image will not be used. Therefore, by removing the pair containing such a label image from the training data, it is possible to reduce the possibility that an inappropriate label image is generated by using the trained model.

The analysis unit 214 can be expected to detect a specific region of various images at high speed and with high accuracy by performing the segmentation process using such a trained model for segmentation.

(Modification example 4)
In the above embodiment and the modified example, the analysis unit 214 performs image segmentation processing on the tomographic image in which the folded image is reduced. On the other hand, when the arithmetic processing unit 213 uses the trained model to generate a tomographic image in which the folded image is reduced, a label image with a label value attached to each pixel of the tomographic image may be generated. ..

In this case, as the output data of the training data of the trained model used by the arithmetic processing unit 213, a label image in which a label value is attached to each pixel of the tomographic image 703 and the tomographic image 703 in which the folded image does not occur is used. The learning data generation unit 602 may generate a label image of the tomographic image 703 by the segmentation process using the above-mentioned rule base or the trained model. As a result, the learning data generation unit 602 can generate the tomographic image 703 and the label image as output data of the learning data. The label image may be a label image generated by the learning data generation unit 602 modified by an examiner or the like. Further, since the trained model outputs output data that is likely to correspond to the input data according to the learning tendency, the label image may be excluded from the training data if the examiner is not appropriate.

In such a case, the analysis unit 214 can perform image analysis processing based on the label image output by the arithmetic processing unit 213. The arithmetic processing unit 213 may generate only a label image in which a label value is attached to each pixel for the tomographic image in which the folded image is reduced, as a tomographic image in which the folded image is not generated. In this case, the learning data generation unit 602 may use only the label image in which the label value is attached to each pixel of the tomographic image 703 as the output data of the learning data.

In these cases as well, the control unit 200 can acquire tomographic information capable of acquiring a highly reliable analysis result by generating a label image by the arithmetic processing unit 213 using the trained model. As the trained model, a trained model for generating a tomographic image with a reduced folding image and a trained model for generating a label image may be separately prepared, or both images are generated. One trained model for this may be prepared. As for the trained model for generating the label image, only the label image in which the label value is attached to each pixel of the tomographic image 703 in which the folded image does not occur may be trained as the output data of the training data.

(Modification 5)
In the modification 4, the arithmetic processing unit 213 outputs a label image of the tomographic image to which the folding reduction processing has been performed. On the other hand, the arithmetic processing unit 213 uses the trained model to generate a label image with a label value that can identify the region where the folded image is generated and the region where the folded image is not generated. May be good. In such a case, the label image relating to the training data of the trained model may be a label image to which a label value is given by a doctor or the like. Specifically, with respect to the tomographic image 702 shown in FIG. 7, an image in which a doctor or the like labels each pixel can be used.

In this case, the analysis unit 214 identifies a region where the folded image does not occur based on the label value in the generated label image, and selectively analyzes the layer structure in the region other than the region. It can be expected that highly reliable analysis results can be obtained. In this case, the control unit 200 does not generate the folding reduction image. However, the arithmetic processing unit 213 uses the trained model to generate a label image that can distinguish between the area where the folded image is generated and the area where the folded image is not generated, so that the control unit 200 is reliable. It is possible to obtain fault information that can obtain high analysis results.

(Modification 6)
Further, the analysis unit 214 may perform an image analysis process on the label image generated by the segmentation process using the trained model. At this time, the trained model for image analysis includes training data including a medical image including a tomographic image and an analysis result of the medical image, and an analysis result of the medical image and a medical image of a type different from the medical image. It may be obtained by learning using the included learning data or the like. The medical image may include a tomographic image, a fundus anterior image, an anterior segment image, an En-Face image, an OCTA anterior image, and the like.

Further, the training data of the trained model for performing image analysis includes a label image generated by using the trained model for image segmentation processing and an analysis result of a medical image using the label image. It may be. In this case, the analysis unit 214 can generate the analysis result of the tomographic image from the result of the image segmentation processing by using, for example, the trained model for generating the analysis result. Further, the trained model is obtained by learning using training data including input data including a set of a plurality of medical images of different types of predetermined parts, such as an En-Face image and an OCTA front image. May be good.

Further, the training data includes, for example, at least the analysis value (for example, average value, median value, etc.) obtained by analyzing the analysis area, the table including the analysis value, the analysis map, the position of the analysis area such as the sector in the image, and the like. The information including one may be the data labeled with the input data as the correct answer data (for supervised learning). The display control unit 203 may be configured to display the analysis result obtained by using the learned model for generating the analysis result on the display unit 250 in response to an instruction from the operator.

(Modification 7)
In addition, the analysis unit 214 in the various embodiments and modifications described above may generate an analysis result, a diagnosis result, or the like by using a tomographic image in which the folded image is reduced. At this time, the display control unit 203 may display the analysis results such as the layer thickness of the desired layer and various blood vessel densities on the display screen. In addition, the optic nerve head, macular region, vascular region, nerve fiber bundle, vitreous region, macular region, choroidal region, scleral region, lamina cribrosa region, retinal layer boundary, retinal layer boundary edge, photoreceptor cells, blood cells, The value (distribution) of the parameter relating to the site of interest including at least one such as the vascular wall, the vascular inner wall boundary, the vascular lateral boundary, the ganglion cell, the corneal region, the corner region, and Schlemm's canal may be displayed as the analysis result. At this time, for example, by analyzing a medical image to which various artifact reduction processes are applied, it is possible to display an accurate analysis result. The artifact is, for example, a false image region generated by light absorption by a blood vessel region or the like, a projection artifact, a band-shaped artifact in a front image generated in the main scanning direction of the measured light depending on the state of the eye to be inspected (movement, blinking, etc.), or the like. There may be. Further, the artifact may be any image loss region as long as it is randomly generated for each image taken on a medical image of a predetermined portion of the subject, for example. Further, the display control unit 203 may display the value (distribution) of the parameter relating to the region including at least one of the various artifacts (copy loss region) as described above on the display unit 250 as the analysis result. Further, the value (distribution) of the parameter relating to the region including at least one such as drusen, new blood vessel, vitiligo (hard vitiligo), and abnormal site such as pseudo-drusen may be displayed as the analysis result. The image analysis process may be performed by the analysis unit 214, or may be performed by an analysis unit different from the analysis unit 214. Further, the image for which the image analysis is performed may be an image in which the folded image is reduced, or an image in which the folded image is not reduced.

Further, the analysis result may be displayed in an analysis map, a sector showing statistical values corresponding to each divided area, or the like. The analysis result is a trained model (analysis result generation engine, trained for analysis result generation) obtained by the analysis unit 214 or another analysis unit learning the analysis result of the medical image including the tomographic image as training data. It may be generated using a model). At this time, the trained model is trained using training data including a medical image and an analysis result of the medical image, training data including a medical image and an analysis result of a medical image of a type different from the medical image, and the like. It may be obtained by.

Further, the learning data may include the area label image generated by the segmentation process and the analysis result of the medical image using them. In this case, the image processing unit 210 generates an analysis result of a tomographic image from a result obtained by executing a segmentation process (for example, a detection result of the retinal layer) using, for example, a trained model for generating an analysis result. It can function as an example of the analysis result generation unit. In other words, the image processing unit 210 uses a trained model (second trained model) for generating analysis results, which is different from the trained model for generating a tomographic image with reduced folded images, and the folded image. Image analysis results can be generated for each of the different regions identified by the segmentation process in the tomographic image with reduced. Further, the image processing unit 210 performs image analysis processing on a medical image such as an En-Face image or an OCTA front image obtained by using an image in which the folded image is reduced, using a trained model for generating an analysis result. You may.

Further, the trained model is obtained by learning using training data including input data in which a plurality of medical images of different types of predetermined parts are set, such as a luminance front image and a motion contrast front image. May be good. Here, the brightness front image corresponds to the brightness En-Face image, and the motion contrast front image corresponds to the OCTA En-Face image.

Further, the training data includes, for example, at least the analysis value (for example, average value, median value, etc.) obtained by analyzing the analysis area, the table including the analysis value, the analysis map, the position of the analysis area such as the sector in the image, and the like. The information including one may be the data labeled (annotated) with the input data as the correct answer data (for supervised learning). In addition, the analysis result obtained by using the trained model for generating the analysis result may be displayed according to the instruction from the operator.

In addition, the display control unit 203 in the above-described examples and modifications may display various diagnostic results such as glaucoma and age-related macular degeneration on the display screen. At this time, for example, by analyzing a medical image to which various artifact reduction processes as described above are applied, it is possible to display an accurate diagnostic result. Further, as the diagnosis result, the position of the specified abnormal portion or the like may be displayed on the image, or the state or the like of the abnormal portion may be displayed by characters or the like. Further, the classification result of the abnormal part or the like (for example, Curtin classification) may be displayed as the diagnosis result. Further, as the classification result, for example, information indicating the certainty of each abnormal part (for example, a numerical value indicating the ratio) may be displayed. In addition, information necessary for the doctor to confirm the diagnosis may be displayed as a diagnosis result. As the necessary information, for example, advice such as additional shooting can be considered. For example, when an abnormal site is detected in the blood vessel region in the OCTA front image, it may be displayed that fluorescence imaging using a contrast medium capable of observing the blood vessel in more detail than OCTA is performed.

The diagnosis result is generated by using a trained model (diagnosis result generation engine, trained model for generating diagnosis result) obtained by the control unit 200 learning the diagnosis result of the medical image as learning data. It may be. In addition, the trained model is based on training using training data including a medical image and a diagnosis result of the medical image, and training data including a medical image and a diagnosis result of a medical image of a type different from the medical image. It may be obtained.

Further, the learning data may include the area label image generated by the segmentation process and the diagnostic result of the medical image using them. In this case, the image processing unit 210 generates a diagnosis result of a tomographic image from a result obtained by executing a segmentation process (for example, a detection result of the retinal layer) using, for example, a trained model for generating a diagnosis result. It can function as an example of a diagnosis result generation unit. In other words, the image processing unit 210 uses a trained model (third trained model) for generating diagnostic results, which is different from the trained model for generating a tomographic image with reduced folded images, and the folded image. Diagnostic results can be generated for each of the different regions identified by the segmentation process in the tomographic image with reduced. Further, the image processing unit 210 generates a diagnostic result using a trained model for generating a diagnostic result for a medical image such as an En-Face image or an OCTA front image obtained by using a tomographic image with a reduced folded image. You may.

In addition, the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Information including at least one such as (general medical support information, etc.) and grounds for denying the diagnosis name (negative medical support information), etc. are labeled (annotated) in the input data as correct answer data (for supervised learning). It may be data. In addition, according to the instruction from the examiner, the diagnosis result obtained by using the trained model for generating the diagnosis result may be displayed.

Further, when detecting an abnormal portion, the image processing unit 210 may use a hostile generation network (GAN: Generative Adversarial Networks) or a variational autoencoder (VAE: Variational Auto-Encoder). For example, a DCGAN (Deep Convolutional GAN) consisting of a generator obtained by learning the generation of a tomographic image and a discriminator obtained by learning the discrimination between a new tomographic image generated by the generator and a real frontal image of the fundus of the eye. ) Can be used as a machine learning model.

When DCGAN is used, for example, the discriminator encodes the input tomographic image into a latent variable, and the generator generates a new tomographic image based on the latent variable. After that, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part. When VAE is used, for example, the input tomographic image is encoded by an encoder to be a latent variable, and the latent variable is decoded by a decoder to generate a new tomographic image. After that, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part. Although a tomographic image has been described as an example of the input data, a fundus image, a frontal image of the anterior eye, or the like may be used.

Further, the image processing unit 210 may detect an abnormal portion by using a convolutional autoencoder (CAE). When CAE is used, the same image is learned as input data and output data at the time of learning. As a result, when an image with an abnormal part is input to CAE at the time of estimation, an image without an abnormal part is output according to the learning tendency. After that, the difference between the image input to the CAE and the image output from the CAE can be extracted as an abnormal portion. In this case as well, not only the tomographic image but also the fundus image, the frontal image of the anterior eye, and the like may be used as input data.

In these cases, the image processing unit 210 provides information on the difference between the medical image obtained by using the hostile generation network or the autoencoder and the medical image input to the hostile generation network or the autoencoder as information on the abnormal portion. Can be generated as. As a result, the image processing unit 210 can be expected to detect the abnormal portion at high speed and with high accuracy. Here, the autoencoder includes VAE, CAE, and the like.

Further, a similar case image search using an external database stored in a server or the like may be performed using the analysis result, the diagnosis result, etc. obtained by the processing of the learned model as described above as a search key. If a plurality of images stored in the database are already managed by machine learning or the like with the feature amount of each of the plurality of images attached as incidental information, the image itself is used as a search key. A similar case image search engine (similar case image search model, trained model for similar case image search) may be used. For example, the image processing unit 210 uses a trained model (fourth trained model) for searching for similar case images, which is different from the trained model for generating a tomographic image with reduced folded images, and uses a folded image. From the reduced tomographic image of, it is possible to search for similar case images related to the tomographic image. In addition, the image processing unit 210 uses a trained model for similar case image search to obtain similar case images for medical images such as En-Face images and OCTA front images obtained by using tomographic images with reduced folded images. You may perform a search.

(Modification 8)
The folding reduction processing in the above-described embodiment and modification is not limited to the configuration performed based on the pixel value of the tomographic image. The various processes are performed on the interference signal acquired by the optical head unit 100, the signal obtained by subjecting the interference signal to Fourier transform, the signal obtained by subjecting the signal to arbitrary processing, and the tomographic data including the tomographic image based on these. May be applied. In these cases as well, the same effect as the above configuration can be obtained.

In the trained model for the folding reduction process according to the above-described embodiment and modification, the feature quantities include the magnitude of the brightness value of the tomographic image, the amount of blur, the order and inclination of the bright and dark areas, the position, the distribution, and the continuity. It can be considered that it is extracted as a part and used for the estimation process. In particular, in the trained model for the folding reduction process, for example, the feature amount of the folding is learned from the image with the folding and the image without the folding included in the training data, and the feature amount disappears in the estimation result. It is thought that learning will be done. Further, since the traveling direction of the retinal layer is clearly different between the folded region and the other regions, the trained model for the folded reduction process selectively reduces the folded region by learning such a feature amount, for example. There is a possibility that it can be done. Furthermore, in a trained model for image segmentation processing, etc., the magnitude of the brightness value of the tomographic image, the order and inclination of the bright and dark areas, the position, distribution, continuity, etc. are extracted as part of the feature amount and estimated. It can be considered that it is used for.

Further, the training data of various trained models is not limited to the data obtained by using the ophthalmic apparatus itself that actually performs the imaging, and the data obtained by using the same type of ophthalmic apparatus or the same type according to the desired configuration. It may be data obtained by using an ophthalmic apparatus or the like.

In addition, various trained models according to the above-described embodiment and modification can be provided in the control unit 200. The trained model may be composed of, for example, a CPU, a software module executed by a processor such as an MPU, GPU, or FPGA, or a circuit or the like that performs a specific function such as an ASIC. Further, these learned models may be provided in a device of another server connected to the control unit 200 or the like. In this case, the control unit 200 can use the trained model by connecting to a server or the like provided with the trained model via an arbitrary network such as the Internet. Here, the server provided with the trained model may be, for example, a cloud server, a fog server, an edge server, or the like. In this case, the trained model may be shared by a plurality of OCT devices.

Further, in the above-described embodiments and modifications, a spectrum domain OCT (SD-OCT) device using an SLD as a light source has been described as the OCT device, but the configuration of the OCT device according to the present invention is not limited to this. For example, the present invention may be applied to any other type of OCT device such as a wavelength sweep type OCT (SS-OCT: Swept Source OCT) device using a wavelength sweep light source capable of sweeping the wavelength of emitted light. Can be done. The present invention can also be applied to a Line-OCT device (or SS-Line-OCT device) using line light. The present invention can also be applied to a Full Field-OCT device (or SS-Full Field-OCT device) using area light. Further, the SLO device and the OCT device may include a wave surface compensation SLO (AO-SLO) device using a wave surface compensation optical system, a wave surface compensation OCT (AO-OCT) device, and the like. Further, the SLO device and the OCT device may include a polarized SLO (PS-SLO) device, a polarized OCT (PS-OCT) device, and the like for visualizing information on polarization phase difference and polarization elimination.

In the SS-OCT apparatus, when acquiring a tomographic image for generating learning data without using the line sensor 154, for example, the imaging range can be expanded by increasing the number of interference signal samples. .. More specifically, for example, the imaging range can be expanded by increasing the frequency of a clock signal called the k clock when sampling the interference signal to speed up the processing of the A / D converter. Therefore, in the case of the SS-OCT apparatus, the learning apparatus may use a tomographic image in which the processing of the A / D converter is accelerated to widen the imaging range to generate the learning data.

(Other Examples)
The present invention is also a process in which a program that realizes one or more functions of the above-described embodiments and modifications is supplied to a system or device via a network or storage medium, and a computer of the system or device reads and executes the program. It is feasible. A computer may have one or more processors or circuits and may include multiple separate computers or a network of separate processors or circuits to read and execute computer executable instructions.

The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Although the present invention has been described above with reference to the embodiments and modifications, the present invention is not limited to the above embodiments and modifications. The present invention also includes inventions modified to the extent not contrary to the gist of the present invention, and inventions equivalent to the present invention. In addition, the above-described embodiments and modifications can be appropriately combined as long as they do not contradict the gist of the present invention.

200: Control unit (image processing device), 201: Acquisition unit, 213: Arithmetic processing unit

Claims (28)

The acquisition unit that acquires the first tomographic image of the eye to be inspected,
An arithmetic processing unit that generates a second tomographic image in which the folded image in the first tomographic image is reduced from the first tomographic image using the trained model.
An image processing device. The image processing apparatus according to claim 1, wherein the second tomographic image is a tomographic image in which the folded image in the first tomographic image is removed or appears as a normal image. The arithmetic processing unit
The first tomographic image is input to the trained model,
From the trained model, a third tomographic image in which the folded image in the first tomographic image is removed is acquired.
The image processing apparatus according to claim 1 or 2, which generates the second tomographic image based on the third tomographic image. The arithmetic processing unit
The difference between the first tomographic image and the third tomographic image is flipped upside down to generate a fourth tomographic image.
The image processing apparatus according to claim 3, wherein the second tomographic image is generated by connecting the third tomographic image and the fourth tomographic image. The arithmetic processing unit
From the first tomographic image, an image of the first region where the folded image is generated is extracted.
By connecting the image of the second region obtained by inputting the image of the first region into the trained model with the image of the region other than the first region in the first tomographic image, the said The image processing apparatus according to claim 1 or 2, which generates a second tomographic image. The image processing apparatus according to claim 5, wherein the arithmetic processing unit extracts an image of a predetermined region in the first tomographic image as an image of the first region. The image processing apparatus according to claim 5, wherein the arithmetic processing unit analyzes an image of the first tomographic image and extracts an image of the first region. It also has an analysis unit that analyzes the medical image of the eye to be inspected.
The image processing according to any one of claims 1 to 7, wherein the analysis unit detects a layer boundary in at least one of the first tomographic image and the second tomographic image by the image analysis. apparatus. Based on the layer boundary, the analysis unit uses at least one of a map image, an En-Face image, and an OCTA front image using at least one of the first tomographic image and the second tomographic image. The image processing apparatus according to claim 8, wherein the image processing apparatus is generated. The image processing apparatus according to claim 9, wherein the map image includes at least one of a layer thickness map and a blood vessel density map. The image according to any one of claims 1 to 10, further comprising a display control unit that switches between the first tomographic image and the second tomographic image and displays them on the display unit according to the display mode. Processing equipment. A display control unit that switches between the first tomographic image and the second tomographic image and displays them on the display unit according to the display mode is further provided.
Any of claims 8 to 10, wherein the display control unit switches between the analysis result of the first tomographic image and the analysis result of the second tomographic image and displays them on the display unit according to the display mode. The image processing apparatus according to one item. The image processing device according to claim 11 or 12, wherein the display control unit displays a sentence explaining the second tomographic image together with the second tomographic image on the display unit. The image processing apparatus according to any one of claims 1 to 13, wherein a mask region is provided at a position corresponding to a coherence gate position in the second tomographic image. The arithmetic processing unit
It is determined whether or not the first tomographic image includes the folded image, and the image is determined.
The image processing apparatus according to any one of claims 1 to 14, which generates the second tomographic image when it is determined that the first tomographic image includes the folded image. The image processing apparatus according to any one of claims 1 to 14, wherein the arithmetic processing unit generates the second tomographic image in response to a setting or an instruction of an operator. The trained model is a machine learning model in which training is performed using training data using a tomographic image as input data and a tomographic image without a folded image as output data corresponding to the tomographic image. The image processing apparatus according to any one of 16. The trained model is a machine learning model in which training is performed using training data in which a tomographic image is used as input data and a tomographic image in which a folded image appears as a normal image corresponding to the tomographic image is used as output data. The image processing apparatus according to any one of claims 1, 2, and 5 to 16. The image processing apparatus according to claim 17 or 18, wherein the learning data is data generated using a tomographic image that does not include a folded image. The image according to any one of claims 1 to 19, wherein the second tomographic image is at least one of a tomographic image having brightness as a pixel value and a tomographic image having a label value for each pixel. Processing equipment. An acquisition unit that acquires tomographic data corresponding to the first tomographic image of the eye to be inspected,
An arithmetic processing unit that generates a second tomographic image in which the folded image in the first tomographic image is reduced from the tomographic data using the trained model.
An image processing device. The acquisition unit that acquires the first tomographic image of the eye to be inspected,
Using the trained model, a second tomographic image is labeled from the first tomographic image so as to be able to distinguish between a region in which the folded image is generated and a region in which the folded image is not generated in the first tomographic image. The arithmetic processing unit to be generated and
An image processing device. Using a trained model different from the trained model for generating the second tomographic image, the image analysis result is obtained from the second tomographic image or the medical image obtained by using the second tomographic image. The image processing apparatus according to any one of claims 1 to 22, further comprising an image processing unit for generating. A diagnostic result is generated from the second tomographic image or a medical image obtained by using the second tomographic image using a trained model different from the trained model for generating the second tomographic image. The image processing apparatus according to any one of claims 1 to 23, further comprising an image processing unit. From the second tomographic image or the medical image obtained by using the second tomographic image, an image obtained by using a hostile generation network or an auto encoder and an image obtained by the hostile generation network or the auto encoder were input to the hostile generation network or the auto encoder. The image processing apparatus according to any one of claims 1 to 24, further comprising an image processing unit that generates information regarding a difference from an image as information regarding an abnormal portion. Using a trained model different from the trained model for generating the second tomographic image, a similar case image is obtained from the second tomographic image or the medical image obtained by using the second tomographic image. The image processing apparatus according to any one of claims 1 to 25, further comprising an image processing unit for performing a search. Acquiring the first tomographic image of the eye to be inspected
Using the trained model, a second tomographic image in which the folded image in the first tomographic image is reduced is generated from the first tomographic image.
Image processing methods, including. A program that, when executed by a processor, causes the processor to perform each step of the image processing method according to claim 27.

Images