JP2021079013A - Medical image processing device, processing method for medical image processing device and program - Google Patents

Abstract

To properly output an image in which image quality of an input image is improved.SOLUTION: A medical image processing device includes: multiple image processing means (404a, 404b); and image processing choosing means (402) for choosing any one of the multiple image processing means. The chosen image processing means sets an image of subject's eye as an input image and outputs an image in which image quality of the input image is improved.SELECTED DRAWING: Figure 2

Description

The present invention relates to a medical image processing apparatus, a processing method and a program of the medical image processing apparatus.

In recent years, an optical coherence tomography (hereinafter referred to as OCT) using interference due to low coherence light has been put into practical use as an ophthalmic examination device. Further, scanning laser ophthalmoscope (hereinafter referred to as SLO) has been put into practical use as an ophthalmologic examination device. OCT and SLO are widely used as an ophthalmic apparatus for acquiring an image for ophthalmic diagnosis.

The OCT obtains a tomographic image of the measurement target by interfering the light reflected from the measurement target with the light reflected from the reference mirror and analyzing the interference light intensity. As such an OCT, the interference light is separated by using low coherence light, and the wavelength is first separated by using a spectral domain OCT (SD-OCT) obtained by replacing the depth information with frequency information and a wavelength sweeping light source. Wavelength sweep OCT (SS-OCT) is known. Further, SLO obtains a two-dimensional plane image of the fundus by irradiating and scanning the fundus with spot light and measuring the intensity of the reflected light.

In recent years, AO-OCT and AO-SLO, which are further equipped with adaptive optics (hereinafter referred to as AO) and depict the fine structure of the retina with high resolution, have been researched and developed (see Non-Patent Document 1). In fundus photography, it is generally difficult to draw the retina with high resolution because the optical system (cornea, crystalline lens, vitreous body) of the human eye is non-uniform and aberration occurs in the wavefront of the light that photographs the fundus. In AO-OCT and AO-SLO, the aberration of the light wave surface is measured by using the wave surface sensor, and the aberration is canceled by using the wave surface correction device, so that it is possible to acquire a high-resolution fundus image.

In addition, an angiography method using OCT and not using a contrast medium has been proposed. This angiography method is called OCT angiography (hereinafter referred to as OCTA). In OCTA, a plane blood vessel image (hereinafter, also referred to as OCTA image) is generated by integrating the acquired three-dimensional motion contrast data in the depth direction and projecting it on a two-dimensional plane. Here, the motion contrast data is data obtained by repeatedly photographing the same cross section and detecting a temporal change of the subject during the photographing. This data can be obtained, for example, by calculating the phase difference, vector difference, or temporal change in intensity of the complex OCT signal (see Patent Document 1). On the other hand, when the number of repetitions of the same cross section is small, there is a problem that the image quality of the obtained OCTA image is low.

Further, in recent years, a method of improving the image quality of medical images by using artificial intelligence such as machine learning has also been disclosed. For example, a method for increasing the resolution of a low-resolution image using an artificial intelligence engine is disclosed (see Patent Document 2). The artificial intelligence engine trains a plurality of combinations (learning samples) of a low-resolution image as an input and a high-resolution image as an output, and performs inference based on the combination (learning sample) to perform resolution conversion.

JP 2015-131107 JP-A-2018-5841

"Adaptive optics optical coherence tomography for high-resolution and high-speed 3D retinal in vivo imaging", R. et al. Zawadzki et. al, Optics Express, Vol. 13, pp. 8532-8546

However, image processing using machine learning requires a trained model learned with learning data corresponding to the subject. If the morphology of structures such as blood vessels depicted in the image and the shooting conditions are different, it may be difficult for one trained model to handle all subjects. Each time the subject, shooting conditions, or purpose changed, the user had to select a trained model. For example, in AO-OCT, microstructures of the retina, such as capillaries, photoreceptor cells, blood vessel walls, and drusen, which have significantly different morphologies, may be the subject, and it was necessary to switch to the corresponding trained model. Further, the subject may be not only a still image but also a moving image such as a blood flow or a pulsation of a blood vessel. Further, for example, it is necessary to switch the trained model according to the angle of view and resolution of the image and the part to be imaged. Even for the same subject, the suitable trained model may change depending on the fluctuation of the SN ratio and the contrast due to the shift of the focus position and the excess or deficiency of the amount of light. Also, even for the same image, there are cases where it is desirable to have a gradation of brightness for observation and cases where it is desirable to binarize it for analysis, depending on the purpose, and different trained models are required for each. It was. As described above, it has been complicated for the user to select an appropriate trained model.

An object of the present invention is to enable an image having improved image quality of an input image to be appropriately output.

The medical image processing apparatus of the present invention has a plurality of image processing means and an image processing selection means for selecting one of the plurality of image processing means, and the selected image processing means is the eye to be inspected. Is used as an input image, and an image with improved image quality of the input image is output.

According to the present invention, it is possible to appropriately output an image in which the image quality of the input image is improved.

It is a figure which shows the structural example of the ophthalmologic imaging apparatus. It is a figure which shows the structural example of the image generation part of the ophthalmologic imaging apparatus. It is a flow chart which shows an example of signal processing. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of the training data. It is a figure which shows the structural example of the ophthalmologic imaging apparatus. It is a flow chart which shows an example of signal processing. It is a figure which shows the example of the display screen. It is a flow chart which shows an example of a shooting procedure. It is a figure which shows the example of an image. It is a flow chart which shows the modification of the signal processing. It is a figure which shows the example of an image. It is a figure which shows the example of image processing. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a flow chart which shows the modification of the signal processing. It is a figure which shows the example of an image. It is a figure which shows the example of an image. It is a figure which shows the example of the display screen.

[First Embodiment]
[Overall configuration of ophthalmic imaging equipment]
FIG. 1 is a diagram showing a configuration example of the ophthalmologic imaging apparatus 200 according to the first embodiment. The ophthalmologic imaging apparatus 200 is a medical image processing apparatus and includes an optical system 100, a control unit 300, an image generation unit 400, a display control unit 500, and a storage unit 600. The optical system 100 is controlled by the control unit 300, irradiates the eye to be inspected 118 with the measurement light, and detects the return light from the eye to be inspected 118. The image generation unit 400 processes a signal to generate an image, and outputs the generated image to the display control unit 500. The image to be generated differs depending on the imaging means and the display content. For example, in the case of OCT, the image generation unit 400 _converts a two-dimensional tomographic image T _OCT , a three-dimensional OCT image V _OCT , and a three-dimensional OCT image into two dimensions based on the signal S OCT output by the optical system 100. Generates a projected two-dimensional frontal image of the fundus. In the case of SLO, the image generation unit 400 generates a two-dimensional frontal fundus image M _{SLO based on the SLO signal S SLO} output by the optical system 100. The display control unit 500 includes a display device such as a liquid crystal display, and displays an image generated by the image generation unit 400. Further, the signal and the generated image are stored in the storage unit 600 together with the information for identifying the eye to be inspected 118.

Further, the ophthalmologic imaging apparatus 200 can be realized by a PC (personal computer) connected to hardware having a specific function. For example, the optical system 100 can be realized by hardware, and the control unit 300, the image generation unit 400, and the display control unit 500 can be realized by a program (software module) that can be mounted on a PC. Hereinafter, the function is realized by the CPU, which is the arithmetic processing unit of the PC, executing the program, but the method is not limited to such a method. The image generation unit 400 may be realized by dedicated hardware such as ASIC. The display control unit 500 may be realized by a dedicated processor such as a GPU different from the CPU. Alternatively, the image generation unit 400 may be realized by a program that performs post processing after image acquisition, and may be in a form in which image data is exchanged from the storage unit 600 separately from the optical system 100. Further, the connection between the optical system 100 and the PC can also be realized by a configuration via a network.

<Image generation unit 400>
FIG. 2 is a diagram showing the configuration of the image generation unit 400. The image generation unit 400 includes a reconstruction unit 401, an image processing selection unit 402, and an image processing unit 403. The image processing unit 403 has a plurality of trained models 404a, 404b, and the like. Hereinafter, the trained models 404a and 404b and the like are referred to as trained models 404. The plurality of trained models 404 are a plurality of image processing units, and the image of the eye to be inspected 118 is used as an input image, and an image in which the image quality of the input image is improved is output.

FIG. 3 is a flowchart showing a processing method of the image generation unit 400. In step S401, the reconstruction unit 401 performs various processes on the signal output from the optical system 100 to generate an image relating to the eye to be inspected 118. Specifically, the reconstruction unit 401 reconstructs a predetermined OCT image, SLO image, OCTA image, or the like. The details will be described later. Examples of the generated image include a two-dimensional tomographic image, a frontal image, a three-dimensional volume data, a three-dimensional OCT image, and an image cut out for each layer of the retina. Further, the reconstruction unit 401 may generate an image from the signal stored in the storage unit 600 instead of the optical system 100.

In step S402, the image processing selection unit 402 selects one of the plurality of trained models 404 of the image processing unit 403 based on the image of the eye to be inspected 118 generated by the reconstruction unit 401 and the information about the image. To do. The image processing selection unit 402 may read the image stored in the storage unit 600 instead of the image generated by the reconstruction unit 401.

In step S403, the selected trained model 404 uses the image of the eye to be inspected 118 generated by the reconstruction unit 401 as an input image, and outputs an image with improved image quality of the input image by inference processing. The image of the eye to be inspected 118 is an image of the anterior segment of the eye to be inspected, the posterior segment of the eye to be inspected, or the microstructure of the eye to be inspected. The ultrastructure is blood vessels, capillaries, photoreceptor cells, or blood vessel walls, or soft vitiligo, hard vitiligo (drusen).

The trained model 404 uses a machine learning algorithm including deep learning and the like. The trained model 404 is preliminarily trained to output an image equivalent to the teacher image with improved image quality from the input image of a specific subject by inference processing. Training data is prepared for each subject, and a plurality of trained models 404 are prepared. Each trained model 404 is associated with a specific subject to be trained.

The image processing selection unit 402 may use the second trained model in order to select the trained model 404. The second trained model may also use a machine learning algorithm including deep learning. In the second trained model, the image generated by the reconstruction unit 401 is used as an input, and learning to select a trained model 404 suitable for the subject by inference processing is completed in advance. The second trained model may output the subject by inference processing instead of selecting the trained model 404. The image processing selection unit 402 can select the trained model 404 associated with the subject by inferring the subject using the second trained model.

The input to the second trained model may be input to information accompanying the image in addition to the image. Examples of accompanying information include patient information, examination information, and imaging data. Examples of patient information include the age and race of the patient, illness and pre-existing illness. In addition, examples of inspection information include inspection mode, scan pattern, measurement position / angle of view, previous measurement result, selection result of trained model 404 in previous measurement, and so on. .. Examples of shooting data include signal strength at the time of shooting, noise strength, SN ratio, contrast, focus position, OCT coherence gate position, or information that can be acquired other than the subject signal at the time of shooting, such as the amount of light from the light source and sensor gain. Can be mentioned. By learning by adding the information accompanying the image, it is expected that the appropriate trained model 404 or the subject can be easily identified.

Instead of using the second trained model, the image processing selection unit 402 may select the trained model 404 by conditional branching or the like based on the information accompanying the image. For example, the image processing selection unit 402 may select the trained model 404 in conjunction with the shooting mode. In the case of a mode for photographing a specific subject such as observation of photoreceptor cells and observation of capillaries, the image processing selection unit 402 may select a corresponding trained model based on the information. Further, in the case of the moving image mode for observing the state of blood flow and pulsation, the image processing selection unit 402 may select a trained model for moving images. The trained model for a moving image may process the moving image as a continuous still image, or may process the moving image including the relationship with the images before and after each frame of the moving image. May be good.

The image processing selection unit 402 selects the trained model 404 that uses the still image as the input image when the image of the eye to be inspected 118 is a still image, and inputs the moving image when the image of the eye to be inspected 118 is a moving image. Select the trained model 404 to be the image.

In addition, a general-purpose trained model may be prepared in case there is no corresponding trained model. The general-purpose trained model may be a trained model trained with learning data that is considered to be less dependent on the shape of the subject, such as focus position shift, brightness and contrast correction, and noise removal. In addition, when there is no trained model corresponding to the subject, in addition to the result of the inference processing, the certainty is also output, and when the certainty is equal to or less than a predetermined value, there may be no trained model. The image processing selection unit 402 selects a general-purpose trained model when the certainty of selecting a plurality of trained models 404 is less than or equal to the threshold value.

Further, if there is no corresponding trained model, the trained model may be added after shooting. For example, the image generation unit 400 may additionally read the corresponding trained model from an external storage means such as the storage unit 600 or the server. Further, as an extension of the function of the program, a trained model may be added from the outside. When adding the first trained model, the second trained model that selects the first trained model is also updated.

In addition, the trained model may be additionally created by the user. When the trained model corresponding to the image cannot be inferred, the image processing selection unit 402 acquires a pair of the original data and the teacher image corresponding to a specific subject and accumulates them as training data. The user may identify the subject that could not be inferred, or grouping may be performed for each similar subject using machine learning. Learning may be performed using the accumulated learning data, and a trained model corresponding to a new subject may be additionally created. The trained model to be created may target the first and second trained models. The learning data may be stored in the storage unit 600 or in an external storage means such as a server. In addition, it may be combined with learning data acquired by another device or another facility. Instead of creating learning data by the user, subjects that could not be inferred may be accumulated. A subject similar to the stored subject may be searched for from an external storage means such as a server, a corresponding trained model may be searched, and added.

<Example of image>
Next, examples of the images to be acquired are shown in FIGS. 4 to 7. An example of a two-dimensional front image is shown in FIG. The two-dimensional front image can be acquired by an OCT two-dimensional projected image, an SLO image, or a fundus camera. Similarly, an En-Face image cut out from a specific retinal layer and an OCTA image by motion contrast can be mentioned as examples of a two-dimensional front image. An example of a two-dimensional tomographic image that can be acquired by OCT is shown in FIG. 5, and an example of a three-dimensional volume is shown in FIG. The three-dimensional volume is obtained by acquiring a plurality of OCT tomographic images, for example, by extracting and displaying a portion corresponding to the retinal layer. An example of a three-dimensional tomographic image is shown in FIG. A three-dimensional tomographic image can be obtained by cutting a three-dimensional volume at an arbitrary cross section and displaying a two-dimensional tomographic image in the three-dimensional volume.

8 to 11 show examples of images that depict the fine structure of the retina with high resolution using AO-OCT and AO-SLO. An example of a two-dimensional front image of photoreceptor cells is shown in FIG. 8, an example of a blood vessel is shown in FIG. 9, an example of a blood vessel wall is shown in FIG. 10, and an example of soft vitiligo and hard vitiligo (drusen) is shown in FIG. It should be noted that the examples shown in FIGS. 4 to 11 are examples, and can be applied to other subjects.

An example of training data for training the trained model 404 will be described. Learning data is prepared for each specific subject described above. The learning data is composed of a plurality of pairs in which a low-quality input image of the same portion and a high-quality teacher image are one pair. Schematic diagrams of the input image and the teacher image of the two-dimensional front image are shown in FIGS. 12 (a) and 12 (b), respectively. The input image may be an image having the same image quality as the image generated by the reconstruction unit 401. The teacher image may be a high-quality image with respect to the input image. Examples of the high-quality image include a high-resolution image, noise such as random speckle noise and artifacts, and an image in which stigma due to dark lines and bright lines is reduced. Examples of high-quality images include images having appropriate brightness, contrast, and gradation of the subject, images with clear boundaries, images with less distortion, and images with less color and brightness unevenness. Examples of the method of acquiring the teacher image include an image in which an image corresponding to the input image is acquired a plurality of times and averaged by superimposition. Further, the teacher image may be an image processed by image processing of an image scanned at a high density with the same angle of view as the input image. Alternatively, the teacher image may use another photographing means having a higher image quality. Examples of other imaging means include devices that use adaptive optics such as AO-OCT and AO-SLO, and devices that use smaller beam diameters and different wavelengths to improve vertical and horizontal resolution and depth of penetration. Can be mentioned.

Further, a low image quality with poor shooting conditions may be used as an input image, an image taken under appropriate conditions may be used as a teacher image, and learning data of a pair of the input image and the teacher image may be used. As for the image actually taken, there are cases where only an image whose image quality is inferior to that taken under appropriate conditions can be obtained due to some restrictions. Examples of restrictions include cases where the condition of the eye to be inspected and adjustment at the time of imaging are insufficient. Examples of eye conditions to be inspected include cataracts and other illnesses, bleeding in the fundus, abnormal shape of the fundus, unstable fixation and inadequate averaging. is there. As an example of adjustment at the time of shooting, there are cases where the signal strength at the time of shooting is weak for some reason, the noise strength is high, the SN ratio is low, and the contrast is inappropriate. Alternatively, the focus position and the OCT coherence gate position may not be optimal. Depending on the configuration of the optical system 100, the amount of light of the light source, the optical filter, the set value of the sensor gain, and the adjustment of the polarization state of the interferometer may not be optimal. For these learning data, a pair of the input image acquired by imposing the corresponding constraint and the teacher image taken appropriately may be prepared. Alternatively, a pseudo input image may be generated by performing a process of degrading the image quality on the properly acquired image.

As an example of adjustment at the time of shooting, the case of the optical system 100 using adaptive optics will be described. When a high-resolution image is acquired using adaptive optics, the depth of field becomes shallow at the cost of high resolution. Therefore, the image quality becomes sensitive to the shift of the focus position. By preparing the trained model 404 corresponding to the focus position shift in advance, it is possible to improve the image quality (appropriate focus image) even for an image having the focus position shift.

<Specific configuration of the ophthalmologic imaging device 200>
Next, an example of a specific configuration of the ophthalmologic imaging apparatus 200 will be described.

<Structure of optical system 100>
FIG. 13 is a diagram showing a configuration example of the optical system 100. The main components of the optical system 100 are a light source 101, an AO unit 140, a reference optical system 150, a light receiving optical system 160, an anterior segment observation unit 170, a fixation lamp optical system 180, and an SLO unit 190.

The light source 101 is a low coherence light source, and for example, a Super Luminate Diode (hereinafter referred to as SLD) having a center wavelength of 840 nm and a wavelength width of 100 nm is used. The light source 101 uses an SLD light source, but a titanium sapphire laser or the like may also be used.

The light emitted from the light source 101 is guided to the fiber coupler 103 via a single mode fiber (hereinafter referred to as SM fiber) 102-1 and branched into measurement light and reference light. The branching ratio of the fiber coupler 103 is 90 (reference light): 10 (measurement light).

The branched measurement light is guided to the AO section 140 via the SM fiber 102-4, is converted into parallel light by the collimator 105-1, and is incident on the beam diameter variable section 141.

The beam diameter variable unit 141 is a unit that changes the beam diameter of the measurement light and changes the NA that irradiates the fundus Er of the eye 118 to be inspected. The beam diameter variable portion 141 is configured so that a plurality of lenses can be inserted and removed at adjustable positions. The beam diameter variable unit 141 is configured to be able to communicate with the control unit 300, and the beam diameter can be changed by an instruction from the control unit 300. This makes it possible to switch between an OCT mode in which an image is taken with the same resolution as a normal OCT and an AO-OCT mode in which an image is taken with a high resolution. In the OCT mode, the beam diameter is narrow and the image is taken with a low NA, and in the AO-OCT mode, the beam diameter is wide and the image is taken with a high NA. In the OCT mode, it is desirable to set the beam diameter so that it is 2 mm or less on the pupil of the eye to be inspected 118, and in the AO-OCT mode, it is desirable to operate the beam diameter variable portion 141 so that the beam diameter is 6 mm or more. However, if the pupil of the eye 118 to be examined does not open more than 6 mm or the entrance pupil of the eye 118 to be examined is narrow due to a cataract disease or the like, AO-OCT can be imaged with a beam diameter between 2 mm and 6 mm. It may have various shooting modes.

Next, the measurement light passes through the optical dividing unit 107 and is guided by the relay optical elements 130-1 to 130-6 to the wave surface correction device 104, the scanning optical system 108, and the eye to be inspected 118. The relay optical elements 130-1 to 130-6 are composed of a lens. The relay optical elements 130-1 to 130-6 are adjusted so that the entrance pupils of the wavefront correction device 104, the scan optical system 108, and the eye to be inspected 118 are substantially phase-conjugated.

The scanning optical system 108 is capable of performing two-dimensional scanning of x and y, and may perform two-dimensional scanning with a single mirror or may be composed of a plurality of scanners. For example, the scanning optical system 108 includes a galvano scanner in both the y (vertical) direction and the x (horizontal) direction. Further, the scanning optical system 108 can change (steering) the shooting position based on the instruction of the control unit 300.

The relay optical elements 130-5 and 130-6 have a function of adjusting the focus of the measurement light on the fundus Er. The relay optical element 130-6 is fixed on the stage 109 and can adjust the focus by moving in the optical axis direction. Although the relay optical element 130-6 adjusts the focus by moving the lens, a Badal Optimeter having a mechanism for fixing the lens and adjusting the optical path length may be used.

The relay optical elements 130-1 to 130-6 may be configured to use a mirror or the like in order to prevent the reflected light on the lens surface from becoming stray light.

The scanning optical system 108 and the stage 109 are controlled by the control unit 300, and the measurement light can be scanned in a desired range of the fundus Er of the eye to be inspected 118. The measurement light is incident on the eye to be inspected 118 by the focus lens mounted on the stage 109 and is focused on the fundus Er.

The measurement light that irradiates the fundus Er is reflected and scattered, follows the relay optical elements 130-1 to 130-6 in the reverse order, and returns to the light dividing unit 107. The measurement light is divided by the optical dividing unit 107 and guided to the wavefront sensor 106 via the relay optical elements 130-7, 130-8 and the aperture 132. The relay optical elements 130-7 and 130-8 are adjusted so as to be substantially phase-conjugated to the entrance pupil of the wave surface correction device 104, the scanning optical system 108, and the eye to be inspected 118. The aperture 132 is inserted to prevent stray light from entering the wavefront sensor 106. The branching ratio of the optical dividing unit 107 is determined by the amount of light incident on the eye to be inspected 118, the light utilization efficiency (throughput) of the AO unit 140, the sensitivity of the wavefront sensor 106, and the throughput of the light receiving optical system 160. Transmission): 10 (reflection).

The return light of the measurement light transmitted through the optical dividing unit 107 enters the fiber coupler 103 via the beam diameter variable unit 141, the collimator 105-1, and the SM fiber 102-4.

On the other hand, the reference light branched by the fiber coupler 103 is emitted via the SM fiber 102-3, and is converted into parallel light by the collimator 151. The reference light is reflected by the mirror 153 via the dispersion compensating glass 152. The mirror 153 is arranged on the stage 154, is driven in the optical axis direction in response to a difference in the axial length of the subject, and is controlled by the control unit 300 so as to adjust the coherent gate. For example, the mirror 153 changes the optical path length of the reference light, but it is sufficient if the optical path length difference between the optical path of the measurement light and the optical path of the reference light can be changed.

The reference light reflected by the mirror 153 follows the optical path in the reverse order, and enters the fiber coupler 103 via the dispersion compensation glass 152, the collimator 151, and the SM fiber 102-3. In the fiber coupler 103, the return light of the measurement light and the reference light are combined to form interference light, which is incident on the light receiving optical system 160.

The interference light incident on the light receiving optical system 160 enters the detector 164 via the collimator 161, the grating 162, and the imaging lens 163. The interference light is decomposed into a spectrum by the grating 162, received on the detector 164 for each spectrum component of the interference light _{, converted into a signal S OCT} , and input to the image generation unit 400.

<AO operation>
The operation of AO in the AO-OCT mode will be described below. In this embodiment, a Shack-Hartmann sensor is used as the wavefront sensor 106. The Shack-Hartmann sensor divides the measurement light by a microlens array and collects each divided measurement light on the two-dimensional sensor. When the signal of the two-dimensional sensor is imaged, it becomes an image in which the focusing points are lined up, and this is called a Hartmann image. The wavefront sensor 106 is connected to the control unit 300, and the control unit 300 reads the Hartmann image and obtains the aberration of the measured light from the amount of movement of the focusing point due to the aberration.

The control unit 300 calculates an input signal to the wave surface correction device 104 based on the aberration of the measurement light, and drives the wave surface correction device 104 so that the aberration of the measurement light is reduced. A series of operations of acquiring the Hartmann image by the wavefront sensor 106, calculating the aberration by the control unit 300, and driving the wavefront correction device 104 are repeatedly performed, and the movement is constantly continued so as to reduce the aberration of the measurement light. Further, the input signal to the wavefront correction device 104 is superposed with a signal for deforming the light wavefront of the measurement light into a desired shape, such as a concave shape (defocus) for changing the focus on the fundus Er. It may be a thing.

<Observation of the anterior segment>
The eye to be inspected 118 is irradiated by the anterior segment illumination light source 171 and photographed by the anterior segment observation unit 170. In this embodiment, an LED having a center wavelength of 740 nm is used as the anterior segment illumination light source 171. The light of the anterior segment illumination light source 171 is reflected by the dichroic mirrors 120-1 and 120-2, respectively, and is incident on the anterior segment observation unit 170. The anterior segment observation unit 170 is composed of an imaging lens 172 and a two-dimensional imaging device 173, and is adjusted so that the anterior segment of the eye to be inspected 118 is captured. The signal from the two-dimensional imaging device 173 is input to the display control unit 500 via the control unit 300 and displayed on the display device. A split prism may be introduced into the anterior segment observation unit 170 in order to facilitate the alignment of the eye to be inspected 118 in the direction along the optical axis.

The user turns on the anterior segment illumination light source 171 and aligns the eye to be inspected 118 before starting the OCT imaging. Further, the user may interrupt the photographing as necessary, turn on the anterior eye portion illumination light source 171, and confirm the position of the eye to be inspected 118. When the alignment is completed, the user turns off the anterior segment illumination light source 171 and completes the shooting with the anterior segment observation unit 170.

<Focus light optical system>
The fixation lamp optical system 180 includes an optical element 181 and a light emitting organic EL display module 182. As the display module 182, a liquid crystal, an LED array, or the like can be used.

The pattern lit and displayed on the display module 182 by the instruction from the control unit 300 is projected onto the fundus Er at an appropriate magnification via the optical element 181 and the dichroic mirrors 120-3, 120-2, 120-1. By changing the lighting position of the pattern, it is possible to induce the fixation of the eye 118 to be inspected and adjust so that the desired position of the fundus Er is captured. As the pattern, a cloth, a circle, a square, or the like can be used, and it is desirable to use a pattern that is most easily perceived by the eye 118 to be examined. Further, the size of the pattern may be changed or a color filter may be inserted to make adjustments so that the eye 118 to be inspected can be easily perceived.

The optical element 181 can adjust the focus on the fundus Er according to the instruction from the control unit 300. The optical element 181 is placed on the stage 183 and moved in conjunction with the stage 109 of the AO unit.

<SLO section>
The SLO unit 190 captures a wide range of fundus Er (angle of view 40 degrees × 40 degrees). The SLO unit 191 is composed of a light source, a two-dimensional scanner, a detector, and the like, and acquires about 15 SLO images per second. _{The SLO signal S SLO} acquired by the SLO unit 191 _{is converted into an SLO image M SLO} by the image generation unit 400, input to the display control unit 500, and displayed on a display device (not shown). The acquired SLO image can be used for confirming the imaging position of the diseased part or the like and designating the imaging position by an input device such as a keyboard or mouse (not shown).

<Tracking>
The optical system 100 detects the deviation of the imaging position due to the fixed vision fine movement of the eye 118 to be inspected, and performs tracking for adjusting the imaging position in real time. The procedure will be described below.

The image generation unit 400 selects one of the SLO images of the fundus Er acquired by the SLO unit 191 as a reference image and stores it in the storage unit 600. The reference image preferably has a high signal-to-noise ratio (hereinafter, S / N ratio) and less distortion of the image, and an averaged image of a plurality of SLO images may be used.

When the SLO unit 191 starts shooting, the acquired SLO image is sent to the control unit 300, and the amount of misalignment is obtained based on the calculation result of the correlation value with the reference image. The correlation value may be calculated by shifting the relative pixels of the reference image and the SLO image, or the Phase Only Correlation (POC) method using the Fourier transform may be used. Further, since the correlation value is calculated at high speed, an FPGA unit or a GPU unit may be used.

Based on the amount of misalignment obtained from the calculation, the control unit 300 sends an instruction to the scanning optical system 108, drives the galvano scanner in the x direction and the galvano scanner in the y direction, and adjusts the imaging position of the fundus Er. Since the tracking is based on the SLO image in which a wide area of the fundus Er is captured, the control unit 300 can obtain the amount of the displacement even if the shooting position is significantly deviated, and the same position can be stably obtained without frame out. It is possible to shoot.

<Control unit 300>
Next, the control unit 300 will be described. As described above, the control unit 300 is a software module executed by the CPU, controls each part of the optical system 100, and controls the overall operation of the ophthalmologic imaging apparatus 200. The control unit 300 also accepts input from a user who operates the ophthalmologic imaging apparatus 200. Specifically, the control unit 300 inputs information such as a patient ID that identifies the eye to be inspected 118, parameters necessary for imaging, selection of a pattern for scanning the fundus Er, and the like from a device such as a keyboard or mouse (not shown). .. Based on this, the control unit 300 has a function of controlling each unit and storing the obtained data such as signals and images in the storage unit 600.

<Image generation unit 400>
As described above, the image generation unit 400 generates and outputs an image related to the eye to be inspected 118 by performing various processes on the signal output from the optical system 100.

FIG. 14 shows a method of generating an OCT image in the reconstruction unit 401. In step S301, the reconstruction unit 401 _{remaps the} signal S OCT for the wavelength to the wave number. In step S302, the reconstruction unit 401 removes the DC component. In step S303, the reconstruction unit 401 acquires the _{OCT image TOCT by performing a Fourier transform.} In addition, the reconstruction unit 401 performs processing such as positioning and averaging of a plurality of OCT images, synthesizing images taken at a plurality of different fundus positions, _{and generating a three-dimensional OCT image V OCT.} In the generation of the SLO image, the reconstruction unit 401 performs processing such as _{synchronizing the signal S SLO} with the signal of the scanner, converting it into two-dimensional data, and converting it into the SLO image M _SLO.

<Display control unit 500>
Next, the display control unit 500 will be described. As described above, the display control unit 500 includes a display device such as a liquid crystal display, and displays an image input from the image generation unit 400. FIG. 15 shows an example of a screen displayed by the display control unit 500. In addition to the screen shown in the figure, an input screen for specific information of the eye to be inspected 118 such as a patient ID input by the control unit 300 is required.

In FIG. 15, the display device 501 displays a screen composed of an image display area 502 and an eye examination information display area 503. Information such as a patient ID, a name, and an age is displayed in the display area 503 of the eye examination information. In the image display area 502, image display areas 504, 505, 506, 507, 508 and buttons 520, 521, and 522, which are user interfaces that can be operated by the user, are arranged. Further, sliders 512-1, 512-2, 531 and 532, pull-down menus 513, and changeover switches 540 and 541 are arranged in the image display area 502. The images displayed in each display area and the functions of the user interface will be described in the flow of shooting operation described later.

[Imaging operation]
Next, the operation of the ophthalmologic imaging apparatus 200 will be described with reference to the flowchart shown in FIG.

<Step S101> (Start of front eye alignment operation)
First, in step S101, the user aligns the eye to be inspected 118 by the anterior eye observation unit 170.

<Step S102> (SLO, OCT operation start)
Next, in step S102, the control unit 300 starts the operation of SLO and OCT. The control unit 300 starts the operation of the light source of the SLO unit 191, the scanner unit, and the like, and irradiates the fundus Er of the eye to be inspected 118 with light in two dimensions. A preview of the SLO image is displayed in the image display area 504, and the user can confirm the shooting position and image quality of the fundus Er.

Further, the control unit 300 starts the operation of the light source 101 and the scanning optical system 108. Further, the wavefront correction device 104 initializes so that the reflecting surface becomes flat. A preview of the OCT image is displayed in the image display area 505. The pre-scan and preview display may be images that can determine whether OCT focus and coherent gate adjustment are appropriate. For example, an OCT image of a horizontal cross section (broken line 511-2) at the center of the imaging range of SLO or OCT may be used.

<Step S103> (Focus adjustment)
Next, in step S103, the control unit 300 adjusts the focus and coherent gate of SLO and OCT. Focus adjustment can be done automatically or manually, and the user can choose either. In the focus adjustment, the OCT focus is linked to the SLO focus, and the control unit 300 drives the stage 109 in accordance with the focus adjustment. When the focus is automatically adjusted, the autofocus controls the focus so that the control unit 300 maximizes the signal strength and the contrast based on the signal value of the image generation unit 400. Maximization is performed by evaluating the signal value while moving the focus. When manually adjusting the focus, the user operates the slider 531 while viewing the preview image of the SLO image shown in the image display area 504 and the preview image of the OCT image shown in the image display area 505. The control unit 300 drives the focus by operating the slider 531. In parallel with the focus adjustment, the user adjusts the coherent gate so that the retinal tomography of the fundus Er is within the image.

<Step S104> (SLO, fixation lamp adjustment)
Next, in step S104, the control unit 300 adjusts the SLO unit 191 and the fixation lamp optical system 180 so that a desired imaging position of the fundus Er is acquired. The lighting position of the fixed-view lamp optical system 180 is fixed, and the control unit 300 shoots a desired shooting position by steering the SLO unit 191.

<Step S105> (XY Fine Alignment)
Next, in step S105, the control unit 300 performs XY fine alignment of the OCT imaging region. The mark 510 indicating the OCT imaging position is superimposed and displayed on the image display area 504, and the user specifies the imaging position by moving the mark 510 with an input device such as a mouse. The control unit 300 drives the scan optical system 108 based on the position of the mark 510 and performs a pre-scan within the shooting range specified by the mark 510. The control unit 300 preview-displays the image obtained by the prescan in the image display area 505. Further, the broken lines 511-1 and 511-2 superimposed on the image of the image display area 504 are drawn along the x and y directions of the three-dimensional OCT image, and their respective positions are drawn by the sliders 512-1 and 512-2. Is adjustable. When the values indicated by the sliders 512-1 and 512-2 are updated, the tomographic images at the positions corresponding to the broken lines 511-1 and 511-2 are displayed in the display areas 506 (horizontal direction H) and 507 (vertical direction V). ) Is automatically drawn. The user adjusts the sliders 512-1, 512-2 to include the imaging position of the OCT. In response to the user's input, the control unit 300 operates the scan optical system 108 to perform XY fine alignment.

<Step S106> (Reference optical path length adjustment)
Next, in step S106, the control unit 300 fine-tunes the reference optical path length of the OCT. The control unit 300 adjusts the optical path length of the reference light in response to the user moving the reference optical path length adjustment slider 532 displayed on the monitor. Here, the user adjusts the display position of the retinal layer in the fundus tomographic image so as to match a desired position in the tomographic image display area.

<Step S107> (acquisition of reference SLO image)
Next, in step S107, the control unit 300 acquires a tracking reference image. The control unit 300 uses the SLO image acquired after the completion of the focus adjustment as a reference SLO image, and stores it in the storage unit 600 together with the root mean square (rot mean square, hereinafter referred to as rms) value of the SLO image. The control unit 300 may use an SLO image acquired after it is determined that there is no input from the user at a preset time as a reference SLO image and store it in the storage unit 600 together with the rms value. When the reference image is already stored in the storage unit 600, the control unit 300 compares the newly acquired rms value with the stored rms value, and rms the reference SLO image having the higher rms value. It is stored in the storage unit 600 together with the value.

<Step S108> (Start tracking)
Next, in step S108, the control unit 300 starts tracking using the reference SLO image acquired in step S107. The control unit 300 calculates the amount of misalignment from the feature points of the fundus plane image acquired by the SLO optical system. Then, the control unit 300 controls the scan optical system 108 based on the calculated deviation amount to perform tracking, that is, correct the irradiation position of the OCT measurement light on the fundus of the eye. Further, instead of the user starting the tracking instruction, the control unit 300 may start the tracking with the start of shooting.

<Step S109> (OCT image acquisition)
Next, in step S109, the control unit 300 acquires an OCT image. When the user presses the button 520, the control unit 300 starts acquiring the OCT three-dimensional data. The control unit 300 acquires data in the shooting range specified by the mark 510 based on preset parameters such as pixel resolution, speed, average number of times, and scan pattern.

<Step S110> (Execution of segmentation)
Next, in step S110, the control unit 300 executes segmentation of the OCT image. The data acquired in step S109 is sent to the image generation unit 400, imaged, and displayed in the image display areas 505, 506, and 507. In addition, segmentation (acquisition of site information) is performed, and the superposition of segmentation lines can be switched between display and non-display in the pull-down menu 513. The segmentation of the retina will be specifically described below.

The image generation unit 400 creates an image by applying a median filter and a Sobel filter to a tomographic image to be processed extracted from the OCT image (hereinafter, referred to as a median image and a Sobel image, respectively). Next, the image generation unit 400 creates a profile for each A scan from the created median image and Sobel image. The median image has a brightness value profile, and the Sobel image has a gradient profile. Then, the image generation unit 400 detects the peak in the profile created from the Sobel image. The image generation unit 400 extracts the boundary of each region of the retinal layer by referring to the profile of the median image corresponding to before and after the detected peak and between the peaks.

The segmentation result will be described with reference to FIG. FIG. 17 is a tomographic image with brightness averaged, and segmentation lines are superimposed with broken lines. Segmentation has detected, for example, 6 layers.

The breakdown of the 6 layers is as follows. (1) Nerve fiber layer (NFL). (2) A layer in which the ganglion cell layer (GCL) + the inner plexiform layer (IPL) are combined. (3) A layer in which the inner nuclear layer (INL) + the outer plexiform layer (OPL) are combined. (4) A layer in which the outer nuclear layer (ONL) + the external limiting membrane (ELM) are combined. (5) A layer in which Ellipoid Zone (EZ) + Interdigtion Zone (IZ) + Retinal pigment epithelium (RPE) are combined. (6) Choroid.

The above segmentation is an example, and other methods such as segmentation using Dijkstra's algorithm may be used. Moreover, the number of layers to be detected can be arbitrarily selected. Segmentation may be performed by a third trained model described later.

<Step S111> (AO-OCT mode start)
Next, in step S111, the control unit 300 starts wavefront correction. The user operates the changeover switch 541 from the OCT mode to the AO-OCT mode, so that the control unit 300 shifts to the AO-OCT mode. The control unit 300 drives the beam diameter variable unit 141 to change the beam diameter and start the operation of the AO. When the mode shifts to the AO-OCT mode, the control unit 300 drives the beam diameter variable unit 141 to change the beam diameter, and the wavefront correction device 104 starts wavefront correction. Here, the control unit 300 deforms the shape of the wavefront correction device 104 based on the aberration measured by the wavefront sensor 106, and corrects the aberration of the eye to be inspected other than the defocus component.

<Step S112> (AO-OCT imaging position designation)
Next, in step S112, the control unit 300 specifies the imaging position of the AO-OCT. When the wavefront correction is started in step S111, the control unit 300 performs preview shooting in the AO-OCT mode based on preset parameters such as scan amplitude and scan speed. For example, the control unit 300 takes a picture of the shooting position at the center of the area designated by the mark 514, and preview-displays the AO-OCT image in the display area 508.

In the AO-OCT mode, the broken lines 511-1 and 511-2 superimposed on the image in the image display area 504 or the image display area 505 are drawn along the x and y directions of the three-dimensional OCT image, and the slider 512- Each position can be adjusted by 1, 512-2. When the value indicated by the slider 512 is updated, the tomographic image at the position corresponding to the broken line 511 is automatically drawn in the display areas 506 (horizontal direction H) and 507 (vertical direction V). The user adjusts the slider 512 to include the imaging position of the AO-OCT. The OCT image displayed in the image display area 505 may be a three-dimensional volume image or a tomographic image as shown in FIGS. 6 and 7. The broken lines 511-1 and 511-2 may be configured so that an arbitrary direction can be specified by an input means such as a mouse, and a tomographic image generated from a three-dimensional OCT image by interpolation or the like is displayed in a display area to display a disease. It may be configured so that a part or the like can be easily specified.

In addition, the layers separated by segmentation can be selected individually or in combination from the pull-down menu 513, and the desired retinal layer is emphasized and displayed, such as making the unselected layer transparent or translucent. It is configured so that the shooting position can be easily adjusted. The AO-OCT imaging position is specified by the user moving the mark 514 superimposed on the display area 505, the display area 506, or the display area 507. The size and position of the mark 514 can be selected and changed by the user.

<Step S113> (AO-OCT image acquisition)
Next, in step S113, the control unit 300 starts photographing the AO-OCT. When the user presses the button 522, the control unit 300 starts AO-OCT imaging. Based on the shooting range specified by the mark 514, the control unit 300 determines parameters such as the scan amplitude and scan speed of the scan optical system 108, the number of shots, and the shooting order.

The obtained AO-OCT signal is converted into an AO-OCT image by the image generation unit 400. The control unit 300 stores the obtained data such as signals and images in the storage unit 600. Further, the control unit 300 displays the obtained AO-OCT image in the display area 508.

<Step S114> (End of shooting)
After acquiring the AO-OCT image, in step S114, the control unit 300 completes the imaging. The user looks at the AO-OCT image in the display area 508 and determines whether or not the desired imaging is completed. If the shooting fails or a different place is shot, the user returns to step S112 again in step S114, specifies the shooting position, and continues shooting. When the desired shooting can be obtained, the user completes the shooting and turns off the light source of the SLO or OCT. To end the shooting, for example, the button 520 or the end button (not shown) may be pressed.

After the shooting is completed, the user selects whether or not to perform image processing for improving the image quality by the image generation unit 400 on the image with the changeover switch 540. For example, when the user selects ON, the image generation unit 400 performs the processing as shown in FIG. 3 and displays the high-quality image processed in the display area 508. The high image quality processing may be performed on the display images displayed in the display areas 504 to 507.

[Second Embodiment] (Semantic Segmentation)
A second embodiment of the present invention will be described with reference to the flowchart shown in FIG. In FIG. 18, the same steps as in FIG. 3 have the same reference numerals.

In step S401, the reconstruction unit 401 performs various processes on the signal output from the optical system 100 to generate an image relating to the eye to be inspected 118.

Next, in step S411, the reconstruction unit 401 functions as a division unit and divides the image of the eye to be inspected 118 into a plurality of regions for each type of subject. An example of region division will be described with reference to FIGS. 9 and 19. FIG. 9 represents images of photoreceptor cells and capillaries obtained by AO-SLO. FIG. 19 shows a schematic diagram in which FIG. 9 is divided into regions. For simplicity, only the boundaries are shown in FIG. The reconstruction unit 401 is divided into a photoreceptor region 10 and a capillary region 11 by region division. Using the third trained model, the reconstruction unit 401 divides the image of the eye to be inspected 118 into a plurality of regions for each type of subject. The third trained model uses a machine learning algorithm including deep learning and the like. In the third trained model, learning to output the contour and type of the subject from the input image of the specific subject by inference processing is completed in advance (semantic segmentation is performed). As for the learning data, as in FIGS. 9 and 19, a plurality of pairs may be prepared in which an image showing a plurality of subjects is used as an input image and an image in which the subject is specified by being divided into regions is used as a teacher image.

Next, in step S412, the image processing selection unit 402 selects one of the plurality of trained models 404 for each region divided in step S411. The image processing selection unit 402 can select one of the plurality of trained models 404 for each region by using the fourth trained model. As the fourth trained model, the trained model 404 corresponding to the subject is selected in the same manner as the second trained model. The fourth trained model is different from the second trained model in that a plurality of subjects in the image are targeted and a plurality of trained models are selected. In the fourth trained model, the region-divided image is used as an input image, and the learning for inferring the corresponding trained model 404 is completed in advance.

Next, in step S413, the selected trained model 404 outputs an image in which the image quality of the input image is improved for each region.

Next, in step S414, the image processing unit 403 functions as a compositing unit, and synthesizes the image output in step S413 into the corresponding region.

The target of image processing may be the entire divided region, or only the region of a specific subject may be the target of processing. In the example of FIG. 19, image processing may be performed on each of the photoreceptor cells and the capillaries, or only either the photoreceptor cells or the capillaries may be the target of the image processing. The fourth trained model may be trained to infer what kind of combination is selected.

Although the examples of photoreceptor cells and capillaries have been described, the subjects may be other combinations, and the subjects may be three or more.

It is desirable that images other than the target area of image processing do not affect the inference processing of the trained model. For example, as the input image to the image processing unit 403, an image masking an area other than the processing target may be used as the input image. Further, the image processing unit 403 may perform image processing on the original input image as it is and cut out the target area after the processing, or may cut out the target area from the original image and change the image size to use the input image. The brightness value of the masked area may be replaced with an intermediate value, a zero value, or a maximum value, or converted into a negative brightness value, etc., and the masked area may be clearly learned.

Although the front image has been described as an example, the image may be a tomographic image or a three-dimensional image. The tomographic image before the division is shown in FIG. 20 (a), and the boundary line after the division is shown in FIG. 20 (b). Since each layer has a different grain size and contrast, the optimum trained model also differs. By changing the trained model for each divided layer, image processing suitable for each layer can be performed. The same can be performed for the three-dimensional volume image shown in FIG. 6 and the three-dimensional tomographic image shown in FIG. 7. When the vitreous body outside the retina or the outside of the choroid is not the object of observation, the image processing may be performed only on the region of the retina, excluding the object of image processing. Similarly, if there is bleeding in the fundus or opacity due to cataract, the area may be divided from the image and excluded from the image processing target.

According to the present embodiment, by dividing each subject in the image and performing image processing for each subject, it is possible to select an image process suitable for each subject in the image.

[Third embodiment] (dividing the area in the image)
A third embodiment of the present invention will be described. In the present embodiment, the image generation unit 400 divides an area in the image, selects a different trained model, and performs image processing. Further, it is not necessary to perform image processing on a part of the divided image and not perform image processing on the other part. This embodiment can be applied when the appearance differs depending on the location of the image, such as the central portion and the peripheral portion of the image.

When the appearance of the image is different, in the example of origin of the ophthalmologic imaging apparatus 200, when the measurement light is blocked by the lens barrel and a shadow is reflected, or when the distortion or color of the peripheral portion of the image is different, the brightness or The contrast may differ depending on the image position. In addition, when a ghost due to reflection is reflected, an artifact may appear in the image. In addition, examples derived from the eye 118 to be inspected include tilting of the eye 118 to be inspected, focus shift due to unevenness, and reflection of eyelashes.

An example in which the measurement light is kicked and a shadow is reflected on the lens barrel is shown in FIG. 21 (a), an example in which the brightness and contrast differ depending on the image position is shown in FIG. 21 (b), and an example of the focus shift in the peripheral portion is shown. It is shown in 21 (c). Further, an example of left focus shift due to the inclination of the subject is shown in FIG. 22 (a), and an example of reflection of eyelashes is shown in FIG. 22 (b).

In the example of FIG. 21A, the measurement light is kicked by the lens barrel, the shadow 31 of the lens barrel is reflected in the image 30, and the peripheral portion is darkened. In the case of a combination of a round lens barrel and a square imaging range, vignetting is likely to occur especially in the diagonal direction and the longitudinal direction in the image 30.

In the example of FIG. 21B, the peripheral portion is darker than the central portion. The darkening of the peripheral portion is caused by the in-plane distribution of the coupling efficiency in the optical path of the measurement light including the subject, such as distortion in the peripheral portion and angle dependence of the optical system. In the case of a color image, color unevenness and bleeding occur.

In the example of FIG. 21C, the peripheral portion is blurred with respect to the central portion. Blurring of the peripheral portion occurs, for example, when the focus is deviated because the difference in depth between the central portion and the peripheral portion is large with respect to the depth of field.

Similarly, when the tilt is large on the left and right sides of the screen, blurring occurs in the screen as in the example of FIG. 22A. When the eyelashes are reflected on the screen, the shadow 31 is reflected as shown in FIG. 22 (b).

Since the appearance of each example differs depending on the location of the image, it may be difficult to handle with one trained model. In the present embodiment, the image region is divided and processing is performed with the corresponding trained model. The method of division is not limited to the central portion and the peripheral portion, and may be divided into a plurality of locations.

Using the fifth trained model, the reconstruction unit 401 can divide the image of the eye to be inspected 118 into a plurality of regions for each position or image quality. Similar to the third trained model, the fifth trained model is previously trained to divide and output a region having a different appearance from the input image by inference processing. The fifth trained model is different from the third trained model in that even if the subject is the same, regions that are different in appearance are divided. The fifth trained model is trained using the training data consisting of a set of pairs of the input image and the teacher image in which the area is divided.

The image processing selection unit 402 can select one of the plurality of trained models 404 for each region by using the sixth trained model. In the sixth trained model, the region-divided image is used as an input image, and the corresponding trained model 404 is inferred and trained in advance.

This embodiment has an effect that when the appearance of the image is different in the image, the image can be divided and a suitable trained model can be selected for image processing.

[Fourth Embodiment] (superimposing images of different image quality)
A fourth embodiment of the present invention will be described. The image of the eye to be inspected 118 has a first image and one or more second images that are a part of the first image. The second image is a higher quality image than the first image. As shown in FIG. 23, the first image 40 represents an image obtained by SLO, and the second images 41a, 41b, 41c represent an image obtained by AO-SLO or the like. In general, it takes time to acquire a high-quality image corresponding to the second image, a high-resolution device such as adaptive optics is required, and the angle of view is often narrowed. When the second image is superimposed on the first image, it may be difficult for one trained model to handle it because the image quality differs depending on the location.

This embodiment can be performed by the same processing as the flowchart of FIG. 18 of the second embodiment. Only the points that this embodiment differs from the second embodiment will be described.

In step S411, the reconstruction unit 401 aligns the second images 41a, 41b, and 41c with the first image 40, and then the first image 40 has the second images 41a, 41b, and 41c. Divide into areas and non-existent areas. Hereinafter, the second images 41a, 41b, and 41c are referred to as the second image 41.

In step S412, the image processing selection unit 402 selects the trained model 404 corresponding to each region. The image processing selection unit 402 selects the trained model 404 having the second image 41 as an input image in the region where the second image 41 exists, and the first image processing selection unit 402 in the region where the second image 41 does not exist. The trained model 404 with the image 40 as the input image is selected.

In step S413, the trained model 404 selected in the region where the second image 41 exists uses the second image 41 as an input image and outputs an image with improved image quality of the input image. The trained model 404 selected in the region where the second image 41 does not exist uses the first image 40 as the input image and outputs an image in which the image quality of the input image is improved.

In step S414, the image processing unit 403 synthesizes each image processed in step S413 into a corresponding region.

The trained model 404 corresponding to the first image 40 comprises a set of a pair of a relatively wide angle of view and low image quality input image and a wide angle of view and high image quality teacher image corresponding to the first image 40. Complete the learning using the training data.

The trained model 404 corresponding to the second image 41 has a relatively narrow angle of view corresponding to the second image 41 and a higher image quality than the first image, and a teacher with a narrow angle of view and higher image quality. Complete the training using the training data from the set of image pairs. The high-quality teacher image may be obtained by, for example, superimposing a plurality of sheets, high-density scanning, or another high-quality measuring device. For example, AO may be used to obtain a high-quality teacher image. Further, two images in which the first image 40 is added in addition to the second image 41 may be used as the input image.

By selecting the trained model 404 for each image quality of the input image as in the present embodiment, image processing can be performed with an appropriate trained model suitable for the image quality.

Although the example in which the first image 40 and the second image 41 before compositing are present has been described, the area is divided based on one image obtained by compositing the first image 40 and the second image 41. You may. Further, when three or more types of images having different image quality are mixed, the number of region divisions and the trained model 404 may be increased.

[Fifth Embodiment] (Montage)
A fifth embodiment of the present invention will be described. This embodiment is suitable for cases where two or more images are acquired by shifting the imaging part and the image angle is widened by pasting them together. An example in which two images are pasted together is shown in FIG. 24 (a). In FIG. 24A, two images 51 and 52 acquired by shifting the imaging portion are aligned and pasted together. This embodiment can be performed by the same process as in FIG. 18 of the second embodiment. Only the points that this embodiment differs from the second embodiment will be described.

In step S411, the reconstruction unit 401 divides the regions 51a and 52a having one image into the regions 53b where the two images overlap.

In step S412, the image processing selection unit 402 selects the trained model 404 corresponding to the regions 51a and 52a and the trained model 404 corresponding to the region 53b.

In step S413, the image processing unit 403 performs image processing on each region using the selected trained model 404.

In step S414, the image processing unit 403 synthesizes the image processed in step S413 into the corresponding region.

The trained model 404 corresponding to the regions 51a and 52a is previously trained to output a high-quality image by inference processing for one input image. The trained model 404 corresponding to the region 53b is previously trained to output a high-quality image by inference processing for two input images. Since the trained model 404 corresponding to the region 53b has a larger number of input images than the trained model 404 corresponding to the regions 51a and 52a, relatively high image quality can be obtained. Although the explanation has been given in the example of pasting two images, the same may be applied to three or more images. As the number of overlapping sheets increases, the corresponding trained model 404 may be prepared. As an example, FIG. 24 (b) shows an example in which five images are pasted together. In the example of FIG. 24B, a trained model corresponding to each of 1 to 5 input images may be prepared.

The image of the eye to be inspected 118 is an image in which a part of a plurality of images is overlapped. The reconstruction unit 401 divides the image of the eye to be inspected 118 into a plurality of regions for each number of overlapping images.

By selecting the trained model 404 for each number of input images as in the present embodiment, it is possible to perform image processing with an appropriate trained model 404 corresponding to the number of superposed images.

[Sixth Embodiment] (Image for analysis and image for observation)
A sixth embodiment of the present invention will be described. This embodiment is applied when performing image analysis. FIG. 25 is a flowchart showing the processing of the image generation unit 400 according to the sixth embodiment. A portion where FIG. 25 is different from FIG. 3 will be described.

In step S422, the reconstruction unit 401 divides the area for image display and the area for analysis. The region division for analysis includes, for example, division of the region with respect to the center of the macula as shown in FIG. 26, and division of photoreceptor cells and capillaries as shown in FIG.

In step S423, the image processing selection unit 402 selects the trained model 404 in the image display area and the trained model 404 in the analysis area. When the area is divided in step S422, the image processing selection unit 402 selects the trained model 404 corresponding to each of the divided areas.

In step S424, the trained model 404 selected in the image display area outputs an image in which the image quality of the image in the image display area is improved. The trained model 404 selected in the analysis area outputs an image in which the image quality of the image in the analysis area is improved.

In step S425, the image processing unit 403 functions as an analysis unit and analyzes the image of the analysis area output in step S424. For example, the film thickness of the retinal layer, the number and density of photoreceptor cells, and the like can be mentioned.

In step S426, the image processing unit 403 synthesizes the image display area output in step S424, and superimposes the analysis result of step S425 on the combined image. The display control unit 500 displays an image on which the analysis results are superimposed. This step can be omitted if the superimposed display of the analysis results is not required. Further, the image processing unit 403 may arrange or switch between the display image and the analysis result.

The trained model 404 for analysis corresponds to image processing that prioritizes ease of analysis. The trained model 404 for analysis has been trained using training data consisting of a set of pairs of input images and teacher images suitable for analysis. The teacher image may be an image that has undergone image processing for analysis. For example, other methods include binarization and images segmented by organization.

An image for displaying an image may be easier to see if the brightness of the image has gradation or smoothness, as in the case of photoreceptor cells as shown in FIG. On the other hand, the image for analysis may be easier to analyze if the brightness of the image is binarized with no gradation as shown in FIG. 27 or if the boundary is clear. By performing image processing by the trained model 404 for analysis separately from the image processing by the trained model 404 for image display, the analysis can be performed stably.

[7th Embodiment]
In the above-described embodiment, the imaging data is obtained by, but is not limited to, the spectral domain type OCT (SD-OCT) using a broadband light source and the AO-OCT for the eye 118 to be inspected. For example, a wavelength sweep type OCT (SS-OCT) or a fundus camera may be used. Further, an image using a contrast medium or spontaneous fluorescence, or an OCTA image (motion contrast image) obtained by converting an OCT signal may be used.

When images of different modality are mixed as input images, a trained model 404 corresponding to each may be prepared. In that case, the same processing as that shown in FIG. 18 may be performed. That is, in step S411, the reconstruction unit 401 divides the image area as needed. In step S412, the image processing selection unit 402 may select the trained model 404 corresponding to each modality. In step S413, the image processing unit 403 performs image processing on the selected corresponding trained model 404. In step S414, the image processing unit 403 synthesizes images of different modality.

Further, the processes corresponding to the plurality of embodiments described above may be combined. For example, after performing image processing for compensating for the focus position shift, image processing for reducing noise may be performed.

Furthermore, which trained model is used and the certainty of the inference process may be displayed. As an example, FIG. 28 shows a display example of the trained model. In FIG. 28, the ON state (● circle) and the OFF state (○ mark) are displayed by the changeover switch 552 for each trained model 551 used, and the certainty is displayed by the light and shade buttons 553. .. With such a form, the user can confirm which trained model 551 is used and the certainty. The display of FIG. 28 may be provided on a part of the display screen of FIG. 15, or the trained model 551 may be changed by the user by looking at the result. For example, the changeover switch 552 may be used to switch the model ON / OFF, and the recalculation button 554 may be used to perform recalculation. These functions of the image processing unit 403 may be realized by software.

As described above, according to the present embodiment, the trained model 551 used for each image can be displayed, so that the trained model 551 can be easily confirmed and readjusted.

According to the first to seventh embodiments, the ophthalmologic imaging apparatus 200 can efficiently perform image processing using the trained model. The ophthalmologic imaging apparatus 200 selects a trained model 404 based on the content of the image to be displayed and information at the time of imaging, and the user selects an image processing method by improving the image quality with the selected trained model 404. The trouble can be saved.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

It should be noted that all of the above embodiments merely show examples of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

100 optical system, 200 ophthalmic imaging device, 300 control unit, 400 image generation unit, 500 display control unit, 600 storage unit

Claims (18)

With multiple image processing means
It has an image processing selection means for selecting one of the plurality of image processing means.
The selected image processing means is a medical image processing apparatus characterized in that an image of an eye to be examined is used as an input image and an image having improved image quality of the input image is output. The medical image processing apparatus according to claim 1, wherein the image of the eye to be inspected is an image of the anterior eye portion of the eye to be inspected, the posterior eye portion of the eye to be inspected, or the fine structure of the eye to be inspected. The medical image processing apparatus according to claim 2, wherein the microstructure is a blood vessel, a capillary, a photoreceptor cell, a blood vessel wall, or a drusen. The medical image processing apparatus according to any one of claims 1 to 3, wherein the plurality of image processing means is a plurality of first trained models. The medical use according to any one of claims 1 to 4, wherein the image processing selection means selects one of the plurality of image processing means using the second trained model. Image processing device. The medical image processing apparatus according to any one of claims 1 to 4, wherein the image processing selection means selects one of the plurality of image processing means according to a shooting mode. The image processing selection means selects a plurality of image processing means among the plurality of image processing means, and the image processing selection means selects a plurality of image processing means.
The medical image processing apparatus according to any one of claims 1 to 6, wherein each of the selected plurality of image processing means outputs an image in which the image quality of the input image is improved. Further having a dividing means for dividing the image of the eye to be inspected into a plurality of regions,
The image processing selection means selects one of the plurality of image processing means for each area, and the image processing selection means selects one of the plurality of image processing means.
The selected image processing means outputs an image in which the image quality of the input image is improved for each of the areas.
The medical image processing apparatus according to any one of claims 1 to 7, wherein the medical image processing apparatus further includes a compositing means for synthesizing images output by the plurality of selected image processing means. .. The dividing means uses a third trained model to divide the image of the eye to be inspected into a plurality of regions for each type of subject.
The medical image processing according to claim 8, wherein the image processing selection means selects one of the plurality of image processing means for each region by using a fourth trained model. apparatus. The dividing means uses a fifth trained model to divide the image of the eye to be inspected into a plurality of regions for each position or image quality.
The medical image processing according to claim 8, wherein the image processing selection means selects one of the plurality of image processing means for each region by using the sixth trained model. apparatus. The image of the eye to be inspected has a first image and one or more second images that are a part of the first image.
The second image has a higher image quality than the first image.
The dividing means divides the first image into a region in which the second image exists and a region in which the second image does not exist.
The image processing selection means selects an image processing means that uses the second image as an input image in the region where the second image exists, and the first image processing means in the region where the second image does not exist. Select an image processing method that uses an image as an input image,
The image processing means selected in the region where the second image exists takes the second image as an input image and outputs an image in which the image quality of the input image is improved.
8. The image processing means selected in the region where the second image does not exist uses the first image as an input image and outputs an image in which the image quality of the input image is improved. The medical image processing device described. The image of the eye to be inspected is an image in which a part of a plurality of images is overlapped.
The medical image processing apparatus according to claim 8, wherein the dividing means divides the image of the eye to be inspected into a plurality of regions for each number of overlapping images. The medical image processing apparatus according to claim 8, wherein the division means divides an area for image display and an area for analysis. Further having an analysis means for analyzing an image output by the image processing means
The medical image processing apparatus according to claim 8 or 13, wherein the synthesizing means superimposes the result of the analysis on the synthesized image. The image processing selection means selects an image processing means that uses a still image as an input image when the image of the eye to be inspected is a still image, and inputs a moving image when the image of the eye to be inspected is a moving image. The medical image processing apparatus according to any one of claims 1 to 6, wherein the image processing means is selected. One of claims 1 to 6, wherein the image processing selection means selects a general-purpose image processing means when the certainty of selecting the plurality of image processing means is equal to or less than a threshold value. The medical image processing apparatus according to. A processing method for a medical image processing apparatus having a plurality of image processing means.
A step of selecting one of the plurality of image processing means, and
A processing method of a medical image processing apparatus, wherein the selected image processing means includes a step of using an image of an eye to be examined as an input image and outputting an image having improved image quality of the input image. A program for causing a computer to function as each means of the medical image processing apparatus according to any one of claims 1 to 16.

Images