WO2020049828A1

WO2020049828A1 - Image processing apparatus, image processing method, and program

Info

Publication number: WO2020049828A1
Application number: PCT/JP2019/023650
Authority: WO
Inventors: 弘樹内田; 好彦岩瀬; 治嵯峨野; 律也富田
Original assignee: キヤノン株式会社
Priority date: 2018-09-06
Filing date: 2019-06-14
Publication date: 2020-03-12
Also published as: JP2023076615A; JP7488934B2

Abstract

An image processing apparatus comprising: an image quality improvement unit which generates, from a first image of an eye to be examined, a second image in which at least one of noise reduction and contrast enhancement is performed as compared with the first image, by using a learned model; and a display control unit which displays, on a display unit, the first image and the second image by switching, arranging side by side, or overlapping the first image and the second image.

Description

Image processing apparatus, image processing method, and program

The present invention relates to an image processing device, an image processing method, and a program.

(2) As a method for non-destructively and non-invasively acquiring a tomographic image of a subject such as a living body, an apparatus (OCT apparatus) using optical coherence tomography (OCT) has been put to practical use. An OCT apparatus is widely used as an ophthalmic apparatus for acquiring an image for ophthalmic diagnosis.

In OCT, a tomographic image of a subject can be obtained by causing light reflected from a measurement target and light reflected from a reference mirror to interfere with each other and analyzing the intensity of the interference light. As such an OCT, a time domain OCT (TD-OCT: Time @ Domain @ OCT) is known. In TD-OCT, depth information of a subject is obtained by sequentially changing the position of a reference mirror.

Also, a spectral domain OCT (SD-OCT: Spectral Domain OCT) and a wavelength-swept OCT (SS-OCT: Swept Source OCT) are known. In the SD-OCT, interference light that is caused to interfere using low coherence light is separated, and depth information is replaced with frequency information and acquired. In SS-OCT, interference light is acquired using light whose wavelength has been previously separated using a wavelength-swept light source. Note that SD-OCT and SS-OCT are also collectively referred to as Fourier domain OCT (FD-OCT: Fourier Domain OCT).

By using OCT, a tomographic image based on depth information of the subject can be obtained. Further, by integrating the acquired three-dimensional tomographic images in the depth direction and projecting them on a two-dimensional plane, a front image of the measurement target can be generated. Conventionally, in order to improve the image quality of these images, it has been performed to acquire the images a plurality of times and to perform the superimposition processing. However, in such a case, it takes time to take a plurality of shots.

Patent Literature 1 discloses a technique for converting a previously acquired image into a higher-resolution image using an artificial intelligence engine in order to cope with rapid advances in medical technology and simple imaging in an emergency. . According to such a technique, for example, an image acquired by less photographing can be converted to an image with higher resolution.

JP 2018-5841A

However, even an image having a high resolution may not be an image suitable for image diagnosis. For example, even if the image has a high resolution, an object to be observed may not be able to be properly grasped when there is much noise or when the contrast is low.

Therefore, one of the objects of the present invention is to provide an image processing device, an image processing method, and a program that can generate an image more suitable for image diagnosis than before.

The image processing apparatus according to one embodiment of the present invention uses the learned model to perform at least one of noise reduction and contrast enhancement from the first image of the subject's eye as compared to the first image. An image quality improving unit that generates a second image, and a display control unit that causes the display unit to switch between the first image and the second image and display them side by side or in a superimposed manner.

An image processing device according to another embodiment of the present invention uses a learned model to convert a first image, which is a front image generated based on information in a range in the depth direction of the subject's eye, into the first image. Generating a second image in which at least one of noise reduction and contrast enhancement has been performed compared to the image, based on an image quality improving unit and a depth direction range for generating the first image, A selecting unit that selects a learned model used by the image quality improving unit from a plurality of learned models.

Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings.

1 shows a schematic configuration of an OCT apparatus according to a first embodiment. 2 illustrates a schematic configuration of a control unit according to the first embodiment. 4 shows an example of teacher data according to the first embodiment. 4 shows an example of teacher data according to the first embodiment. 4 illustrates an example of a configuration of a learned model according to the first embodiment. 5 is a flowchart of a series of image processing according to the first embodiment. 13 shows an example of a report screen for switching and displaying images before and after the image quality improvement processing. 13 shows an example of a report screen for switching and displaying images before and after the image quality improvement processing. 13 shows an example of a report screen in which images before and after the image quality improvement processing are displayed side by side. 7 shows an example of a report screen that simultaneously displays a plurality of images to which image quality improvement processing has been applied. 7 shows an example of a report screen that simultaneously displays a plurality of images to which image quality improvement processing has been applied. 9 illustrates a schematic configuration of a control unit according to the second embodiment. 9 is a flowchart of a series of image processing according to the second embodiment. An example of changing the image quality improvement processing will be described. An example of changing the image quality improvement processing will be described. 7 shows an example of a report screen that simultaneously displays a plurality of images to which image quality improvement processing has been applied. 7 shows an example of a report screen that simultaneously displays a plurality of images to which image quality improvement processing has been applied. 13 is a flowchart of a series of image processing according to the third embodiment. 9 shows a schematic configuration of a control unit according to a fourth embodiment. 13 is a flowchart of a series of image processing according to the fourth embodiment. 15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. 15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. 15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. 15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. 17 illustrates an example of a user interface according to a fifth embodiment. 5 shows an example of En-Face images of a plurality of OCTAs. 5 shows an example of En-Face images of a plurality of OCTAs. 17 illustrates an example of a user interface according to a fifth embodiment. 17 illustrates an example of a user interface according to a fifth embodiment.

Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings.

However, dimensions, materials, shapes, relative positions of components, and the like described in the following embodiments are arbitrary, and can be changed according to the configuration of an apparatus to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate the same or functionally similar elements.

In the following embodiments, the subject's eye is taken as an example, but other human organs may be used as the subject. Also, an OCTA (OCT @ Angiography) image of an eye to be examined will be described as an example of an image on which image quality improvement processing is performed using a learned model related to a machine learning model (machine learning engine). OCTA is an angiography using OCT without using a contrast agent. In OCTA, an OCTA image (frontal blood vessel image) is generated by integrating three-dimensional motion contrast data acquired based on depth information of a subject in a depth direction and projecting the integrated data on a two-dimensional plane.

Here, the motion contrast data is data obtained by repeatedly photographing substantially the same part of the subject and detecting a temporal change of the subject during the photographing. In addition, the substantially identical portion refers to a position that is the same as an allowable level for generating the motion contrast data, and includes a portion slightly shifted from the strictly identical portion. The motion contrast data is obtained, for example, by calculating a temporal change of the phase, vector, and intensity of the complex OCT signal from a difference, a ratio, a correlation, or the like.

Here, notes on image quality improvement processing using a learned model for a machine learning model are described. By performing image quality improvement processing on images using a trained model related to machine learning models, high-quality images can be obtained from a small number of images, but tissues that do not actually exist are drawn on the images, Sometimes an existing organization is erased. Therefore, there is a problem that it is difficult to determine the authenticity of the tissue drawn on the image in the image whose image quality has been improved by the image quality improvement processing using the learned model.

Therefore, in the following embodiments, a machine learning model is used to generate an image more suitable for image diagnosis than before, and for such an image, it is possible to easily determine the authenticity of the tissue drawn on the image. An image processing device is provided.

In the following embodiments, an OCTA image will be described. However, the image on which the image quality improvement processing is performed is not limited to this, and may be a tomographic image, a luminance En-Face image, or the like. Here, the En-Face image is a front image generated by projecting or integrating data in a predetermined depth range determined based on two reference planes on a two-dimensional plane in three-dimensional data of the subject. It is. The En-Face image includes, for example, a luminance En-Face image based on a luminance tomographic image and an OCTA image based on motion contrast data.

(Example 1)
Hereinafter, an optical coherence tomography apparatus (OCT apparatus) and an image processing method according to a first embodiment of the present invention will be described with reference to FIGS. FIG. 1 shows a schematic configuration of the OCT apparatus according to the present embodiment.

The OCT apparatus 1 according to the present embodiment includes an OCT imaging unit 100, a control unit (image processing device) 200, an input unit 260, and a display unit 270.

The OCT imaging unit 100 includes an imaging optical system of the SD-OCT apparatus, and causes interference between return light from the eye E to which the measurement light is irradiated via the scanning unit and reference light corresponding to the measurement light. Based on the light, a signal including information on the tomography of the eye E (tomographic information) is acquired. The OCT imaging unit 100 includes a light interference unit 110 and a scanning optical system 150.

The control unit 200 can control the OCT imaging unit 100, generate an image from signals obtained from the OCT imaging unit 100 and other devices (not shown), and process the generated / acquired image. . The display unit 270 is an arbitrary display such as an LCD display, and is a GUI for operating the OCT imaging unit 100 and the control unit 200, a generated image, an image subjected to arbitrary processing, and various information such as patient information. Can be displayed.

The input unit 260 is used to operate the control unit 200 by operating a GUI or inputting information. The input unit 260 includes, for example, a mouse, a touchpad, a trackball, a touch panel display, a pointing device such as a stylus pen, a keyboard, and the like. When a touch panel display is used, the display unit 270 and the input unit 260 can be integrally configured. In the present embodiment, the OCT imaging unit 100, the control unit 200, the input unit 260, and the display unit 270 are separate elements, but some or all of them may be integrally configured. Good.

The light interference unit 110 in the OCT imaging unit 100 includes a light source 111, a coupler 113, a collimating optical system 121, a dispersion compensating optical system 122, a reflection mirror 123, a lens 131, a diffraction grating 132, an imaging lens 133, and a line sensor 134. Is provided. The light source 111 is a low coherence light source that emits near-infrared light. Light emitted from the light source 111 propagates through the optical fiber 112a and enters the coupler 113, which is an optical branching unit. The light incident on the coupler 113 is split into measurement light traveling toward the scanning optical system 150 and reference light traveling toward the reference optical system including the collimating optical system 121, the dispersion compensating optical system 122, and the reflection mirror 123. The measurement light enters the optical fiber 112b and is guided to the scanning optical system 150. On the other hand, the reference light enters the optical fiber 112c and is guided to the reference optical system.

The reference light that has entered the optical fiber 112c is emitted from the fiber end, enters the dispersion compensating optical system 122 via the collimating optical system 121, and is guided to the reflection mirror 123. The reference light reflected by the reflection mirror 123 follows the optical path in the reverse direction and enters the optical fiber 112c again. The dispersion compensating optical system 122 is for compensating the dispersion of the optical system in the scanning optical system 150 and the eye E to be inspected, and adjusting the dispersion of the measurement light and the reference light. The reflection mirror 123 is configured to be drivable in the optical axis direction of the reference light by a driving unit (not shown) controlled by the control unit 200, and sets the optical path length of the reference light to the optical path length of the measurement light. And the reference light and the measurement light can have the same optical path length.

On the other hand, the measurement light that has entered the optical fiber 112b is emitted from the end of the fiber and enters the scanning optical system 150. The scanning optical system 150 is an optical system configured to be relatively movable with respect to the eye E to be inspected. The scanning optical system 150 is configured to be able to be driven in the front, rear, up, down, left, and right directions with respect to the eye axis of the eye E by driving means (not shown) controlled by the control unit 200, and to perform alignment with respect to the eye E. It can be carried out. Note that the scanning optical system 150 may be configured to include the light source 111, the coupler 113, the reference optical system, and the like.

The scanning optical system 150 includes a collimating optical system 151, a scanning unit 152, and a lens 153. Light emitted from the fiber end of the optical fiber 112b is substantially collimated by the collimating optical system 151 and enters the scanning unit 152.

The scanning unit 152 has two galvanometer mirrors capable of rotating a mirror surface, one of which deflects light in the horizontal direction and the other deflects light in the vertical direction. The scanning unit 152 deflects the incident light under the control of the control unit 200. Thus, the scanning unit 152 scans the measurement light on the fundus Er of the eye E in two directions, that is, the main scanning direction perpendicular to the paper surface (X direction) and the sub-scanning direction in the paper surface direction (Y direction). Can be. The main scanning direction and the sub-scanning direction are not limited to the X direction and the Y direction, but are perpendicular to the depth direction (Z direction) of the eye E, and the main scanning direction and the sub-scanning direction intersect each other. Any direction is acceptable. Therefore, for example, the main scanning direction may be the Y direction, and the sub scanning direction may be the X direction.

(4) The measurement light scanned by the scanning unit 152 forms an illumination spot on the fundus Er of the subject's eye E via the lens 153. When receiving the in-plane deflection by the scanning unit 152, each illumination spot moves (scans) on the fundus Er of the eye E to be inspected. The return light of the measurement light reflected and scattered from the fundus Er at the position of the illumination spot follows the optical path in reverse, enters the optical fiber 112b, and returns to the coupler 113.

As described above, the reference light reflected by the reflection mirror 123 and the return light of the measurement light from the fundus Er of the eye E are returned to the coupler 113 and interfere with each other to become interference light. The interference light passes through the optical fiber 112d and is emitted to the lens 131. The interference light is made substantially parallel by the lens 131 and enters the diffraction grating 132. The diffraction grating 132 has a periodic structure and splits the input interference light. The split interference light is imaged on the line sensor 134 by the imaging lens 133 whose focus state can be changed. The line sensor 134 outputs a signal corresponding to the intensity of light emitted to each sensor unit to the control unit 200. The control unit 200 can generate a tomographic image of the eye E based on the interference signal output from the line sensor 134.

により Through the above series of operations, it is possible to acquire tomographic information in the depth direction at one point of the eye E. Such an operation is called A-scan.

Further, by driving the galvanomirror of the scanning unit 152, interference light at one point adjacent to the eye E is generated, and tomographic information in the depth direction at one point adjacent to the eye E is acquired. By repeating this series of controls, two-dimensional tomographic information of the eye E to be examined in the transverse direction and the depth direction can be obtained by performing the A-scan a plurality of times in an arbitrary transverse direction (main scanning direction). . Such an operation is called a B scan. The control unit 200 can configure one B-scan image by collecting a plurality of A-scan images based on the interference signal acquired by the A-scan. Hereinafter, this B-scan image is referred to as a two-dimensional tomographic image.

Furthermore, the galvanomirror of the scanning unit 152 can be minutely driven in the sub-scanning direction orthogonal to the main scanning direction to acquire tomographic information at another position (adjacent scanning line) of the eye E. The control unit 200 can obtain a three-dimensional tomographic image of the eye E in a predetermined range by collecting a plurality of B-scan images by repeating this operation.

Next, the control unit 200 will be described with reference to FIG. FIG. 2 shows a schematic configuration of the control unit 200. The control unit 200 includes an acquisition unit 210, an image processing unit 220, a drive control unit 230, a storage unit 240, and a display control unit 250.

The acquisition unit 210 can acquire the data of the output signal of the line sensor 134 corresponding to the interference signal of the eye E from the OCT imaging unit 100. Note that the data of the output signal acquired by the acquisition unit 210 may be an analog signal or a digital signal. When the acquisition unit 210 acquires an analog signal, the control unit 200 can convert the analog signal into a digital signal.

The acquisition unit 210 can acquire the tomographic data generated by the image processing unit 220 and various images such as a two-dimensional tomographic image, a three-dimensional tomographic image, a motion contrast image, and an En-Face image. Here, the tomographic data is data including information on a tomographic image of a subject, a signal obtained by performing a Fourier transform on an interference signal by OCT, a signal obtained by performing an arbitrary process on the signal, a tomographic image based on the signal, and the like. Including

Further, the acquisition unit 210 may include a shooting condition group (for example, shooting date and time, shooting site name, shooting area, shooting angle of view, shooting method, image resolution and gradation, image size, image filter, , And information on the data format of the image). Note that the photographing condition group is not limited to the illustrated one. Further, the photographing condition group does not need to include all of the illustrated ones, and may include some of them.

Specifically, the acquisition unit 210 acquires the imaging conditions of the OCT imaging unit 100 at the time of capturing an image. The acquiring unit 210 can also acquire a group of photographing conditions stored in a data structure forming an image according to the data format of the image. Note that when the imaging conditions are not stored in the data structure of the image, the acquiring unit 210 can also acquire the imaging information group including the imaging conditions group from a storage device that separately stores the imaging conditions.

The acquisition unit 210 can also acquire information for identifying the eye to be examined, such as the subject identification number, from the input unit 260 or the like. Note that the acquisition unit 210 may acquire various data, various images, and various information from the storage unit 240 and other devices (not illustrated) connected to the control unit 200. The acquisition unit 210 can cause the storage unit 240 to store the acquired various data and images.

The image processing unit 220 generates a tomographic image, an En-Face image, or the like from the data acquired by the acquisition unit 210 or the data stored in the storage unit 240, or performs image processing on the generated or acquired image. Can be. The image processing unit 220 includes a tomographic image generation unit 221, a motion contrast generation unit 222, an En-Face image generation unit 223, and an image quality improvement unit 224.

The tomographic image generation unit 221 performs wave number conversion, Fourier transform, absolute value conversion (acquisition of amplitude), and the like on the data of the interference signal acquired by the acquisition unit 210 to generate tomographic data, and generates tomographic data based on the tomographic data. A tomographic image of the optometry E can be generated. Here, the data of the interference signal acquired by the acquisition unit 210 may be data of a signal output from the line sensor 134 or acquired from a device (not shown) connected to the storage unit 240 or the control unit 200. It may be data of the interference signal obtained. Note that any known method may be used as a method for generating a tomographic image, and a detailed description thereof will be omitted.

断層 Further, the tomographic image generation unit 221 can generate a three-dimensional tomographic image based on the generated tomographic images of a plurality of parts. The tomographic image generation unit 221 can generate a three-dimensional tomographic image by arranging, for example, tomographic images of a plurality of parts in one coordinate system. Here, the tomographic image generation unit 221 may generate a three-dimensional tomographic image based on tomographic images of a plurality of parts acquired from a device (not illustrated) connected to the storage unit 240 and the control unit 200.

The motion contrast generation unit 222 can generate a two-dimensional motion contrast image using a plurality of tomographic images obtained by photographing substantially the same location. In addition, the motion contrast generation unit 222 can generate a three-dimensional motion contrast image by arranging the generated two-dimensional motion contrast images of the respective parts in one coordinate system.

In the present embodiment, the motion contrast generation unit 222 generates a motion contrast image based on decorrelation values between a plurality of tomographic images obtained by photographing substantially the same part of the eye E.

Specifically, the motion contrast generation unit 222 acquires a plurality of tomographic images that have been aligned with respect to a plurality of tomographic images obtained by photographing substantially the same locations where photographing times are continuous with each other. Note that various known methods can be used for alignment. For example, one of a plurality of tomographic images is selected as a reference image, and while changing the position and angle of the reference image, the degree of similarity with other tomographic images is calculated, and the position of each tomographic image with respect to the reference image is calculated. A shift amount is calculated. By correcting each tomographic image based on the calculation result, positioning of a plurality of tomographic images is performed. Note that the alignment processing may be performed by a component that is separate from the motion contrast generation unit 222. The method of positioning is not limited to this, and may be performed by any known method.

The motion contrast generation unit 222 calculates a decorrelation value for each of two tomographic images whose imaging times are continuous from each other among a plurality of aligned tomographic images by the following Equation 1.

Here, A (x, z) indicates the amplitude at the position (x, z) of the tomographic image A, and B (x, z) indicates the amplitude at the same position (x, z) of the tomographic image B. The resulting decorrelation value M (x, z) takes a value from 0 to 1, and becomes closer to 1 as the difference between the two amplitude values increases. In the present embodiment, a case has been described in which a two-dimensional tomographic image on the XZ plane is used, but a two-dimensional tomographic image on the YZ plane or the like may be used. In this case, the position (x, z) may be replaced with the position (y, z) or the like. The decorrelation value may be obtained based on the luminance value of the tomographic image, or may be obtained based on the value of an interference signal corresponding to the tomographic image.

The motion contrast generation unit 222 determines the pixel value of the motion contrast image based on the decorrelation value M (x, z) at each position (pixel position), and generates a motion contrast image. In the present embodiment, the motion contrast generation unit 222 calculates the decorrelation value for tomographic images whose imaging times are continuous with each other, but the method of calculating motion contrast data is not limited to this. The two tomographic images for which the decorrelation value M is obtained need only have an imaging time of each corresponding tomographic image within a predetermined time interval, and the imaging times need not be continuous. Therefore, for example, for the purpose of extracting an object having a small temporal change, two tomographic images whose imaging interval is longer than a normal specified time are extracted from a plurality of acquired tomographic images, and a decorrelation value is calculated. You may. Instead of the decorrelation value, a variance value, a value obtained by dividing the maximum value by the minimum value (maximum value / minimum value), or the like may be obtained.

The method of generating the motion contrast image is not limited to the above-described method, and any other known method may be used.

The En-Face image generation unit 223 can generate an En-Face image (OCTA image) as a front image from the three-dimensional motion contrast image generated by the motion contrast generation unit 222. Specifically, the En-Face image generation unit 223 projects the three-dimensional motion contrast image on a two-dimensional plane based on, for example, two arbitrary reference planes in the depth direction (Z direction) of the subject's eye E. It is possible to generate an OCTA image that is a front image that has been obtained. In addition, the En-Face image generation unit 223 can similarly generate an En-Face image of luminance from the three-dimensional tomographic image generated by the tomographic image generation unit 221.

More specifically, the En-Face image generation unit 223 determines, for example, a representative value of the pixel value in the depth direction at each position in the XY directions of the area surrounded by the two reference planes, and determines the representative value as the representative value. A pixel value at each position is determined based on the position, and an En-Face image is generated. Here, the representative value includes a value such as an average value, a median value, or a maximum value of pixel values within a range in a depth direction of a region surrounded by two reference planes.

Note that the reference plane may be a plane along a layer boundary of a slice of the eye E to be examined, or may be a plane. Hereinafter, the range in the depth direction between the reference planes for generating the En-Face image is referred to as the generation range of the En-Face image. In addition, the method of generating an En-Face image according to the present embodiment is an example, and the En-Face image generation unit 223 may generate an En-Face image using any known method.

The image quality improving unit 224 generates a high-quality OCTA image based on the OCTA image generated by the En-Face image generating unit 223, using a learned model described later. Further, the image quality improving unit 224 is a high-quality tomographic image or high-quality luminance based on the tomographic image generated by the tomographic image generating unit 221 and the luminance En-Face image generated by the En-Face image generating unit 223. May be generated. Note that the image quality improving unit 224 obtains not only the OCTA image or the like captured using the OCT capturing unit 100, but also the obtaining unit 210 obtained from another device (not shown) connected to the storage unit 240 or the control unit 200. A high-quality image can be generated based on various images. Further, the image quality improvement unit 224 may perform image quality improvement processing of not only an OCTA image and a tomographic image but also a three-dimensional motion contrast image and a three-dimensional tomographic image.

The drive control unit 230 can control driving of components such as the light source 111 of the OCT imaging unit 100, the scanning optical system 150, the scanning unit 152, and the imaging lens 133, which are connected to the control unit 200. The storage unit 240 can store various data acquired by the acquisition unit 210 and various images and data such as tomographic images and OCTA images generated and processed by the image processing unit 220. In addition, the storage unit 240 stores information on the subject's eye, such as attributes (name, age, and the like) of the subject and measurement results (eye axis length, intraocular pressure, and the like) acquired using other examination equipment, imaging parameters, and images. Analysis parameters and parameters set by the operator can be stored. The image and the information may be stored in an external storage device (not shown). In addition, the storage unit 240 can also store a program or the like for performing the function of each component of the control unit 200 by being executed by the processor.

The display control unit 250 can cause the display unit 270 to display various information acquired by the acquisition unit 210 and various images such as tomographic images, OCTA images, and three-dimensional motion contrast images generated and processed by the image processing unit 220. it can. Further, the display control unit 250 can cause the display unit 270 to display information and the like input by the user.

The control unit 200 may be configured using, for example, a general-purpose computer. Note that the control unit 200 may be configured using a computer dedicated to the OCT apparatus 1. The control unit 200 includes a storage medium including a CPU (Central Processing Unit) and an MPU (Micro Processing Unit) (not shown), and a memory such as an optical disk and a ROM (Read Only Memory). Each component other than the storage unit 240 of the control unit 200 may be configured by a software module executed by a processor such as a CPU or an MPU. In addition, each of the constituent elements may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like. The storage unit 240 may be configured by an arbitrary storage medium such as an optical disk and a memory.

Note that the control unit 200 may include one processor such as a CPU and storage media such as a ROM, or may include a plurality of storage media. Therefore, each component of the control unit 200 functions when at least one or more processors and at least one storage medium are connected and at least one or more processors execute a program stored in at least one or more storage media. May be configured. The processor is not limited to a CPU or an MPU, but may be a GPU (Graphics Processing Unit) or the like.

Next, a learned model related to a machine learning model according to a machine learning algorithm such as deep learning according to the present embodiment will be described with reference to FIGS. 3A to 4. The learned model according to the present embodiment generates and outputs an image in which image quality improvement processing has been performed based on the input image according to the tendency of learning.

The image quality improvement processing in this specification refers to converting an input image into an image with an image quality more suitable for image diagnosis, and a high-quality image refers to an image converted into an image with an image quality more suitable for image diagnosis. Say. Here, the content of the image quality suitable for the image diagnosis depends on what to be diagnosed by various image diagnoses. For this reason, it cannot be said unconditionally, but for example, image quality suitable for image diagnosis is low in noise, high in contrast, the object to be photographed is shown in colors and gradations that are easy to observe, the image size is large, Includes image quality that may be high resolution. Further, the image quality may include an image in which objects and gradations which do not actually exist and have been drawn in the process of image generation are removed from the image.

A trained model is a model in which a machine learning model according to an arbitrary machine learning algorithm such as deep learning is previously trained (learned) using appropriate teacher data (learning data). However, it is assumed that the learned model does not perform any further learning and can perform additional learning. The teacher data is composed of one or more pairs of input data and output data (correct data). In the present embodiment, a pair of input data and output data is composed of an OCTA image and an OCTA image obtained by performing a superposition process such as an averaging process on a plurality of OCTA images including the OCTA image.

画素 The superimposed image that has been subjected to the superimposition process becomes a high-quality image suitable for image diagnosis because pixels that are commonly drawn in the original image group are emphasized. In this case, the generated high-quality image is a high-contrast image in which the difference between the low-brightness region and the high-brightness region is clear as a result of emphasizing the pixels drawn in common. In addition, for example, in a superimposed image, random noise generated at each shooting can be reduced, or a region that is not well drawn in an original image at a certain time can be interpolated by another original image group.

ペア Note that among pairs forming the teacher data, pairs that do not contribute to high image quality can be removed from the teacher data. For example, when a high-quality image, which is output data forming a pair of teacher data, has an image quality that is not suitable for image diagnosis, an image output by a trained model learned using the teacher data is also not suitable for image diagnosis. Image quality may be lost. Therefore, by removing from the teacher data a pair whose output data is unsuitable for image diagnosis, it is possible to reduce the possibility that the trained model generates an image having image quality unsuitable for image diagnosis.

When the average luminance and the luminance distribution of the image group that is a pair are significantly different, the trained model learned using the teacher data is used to generate an image that is not suitable for image diagnosis having a luminance distribution that is significantly different from the low-quality image. There is a possibility to output. For this reason, a pair of input data and output data having significantly different average luminance and luminance distribution can be removed from the teacher data.

Further, when the structure or position of the imaging target drawn in the group of images that is a pair differs greatly, the learned model learned using the teacher data sets the imaging target in a structure or position that is significantly different from the low-quality image. There is a possibility of outputting an image that is not suitable for the drawn image diagnosis. For this reason, a pair of input data and output data in which the structure or position of the imaging target to be drawn greatly differs can be removed from the teacher data.

By using the trained model that has been trained in this way, the image quality improving unit 224 can increase the contrast and reduce noise by superimposing processing when an OCTA image acquired in one shot (inspection) is input. It is possible to generate a high-quality OCTA image that has been reduced. Therefore, the image quality improving unit 224 can generate a high-quality image suitable for image diagnosis based on the low-quality image that is the input image.

Next, the image at the time of learning will be described. An image group forming a pair group of an OCTA image 301 and a high-quality OCTA image 302, which constitutes teacher data, is created by a rectangular area image of a fixed image size corresponding to a positional relationship. The creation of the image will be described with reference to FIGS. 3A and 3B.

First, a case will be described in which one of the group of pairs forming the teacher data is an OCTA image 301 and a high-quality OCTA image 302. In this case, as shown in FIG. 3A, a pair is formed by using the entire OCTA image 301 as input data and the entire high-quality OCTA image 302 as output data. Note that in the example shown in FIG. 3A, a pair of input data and output data is formed by the entirety of each image, but the pair is not limited to this.

For example, as shown in FIG. 3B, a pair may be formed by using a rectangular area image 311 of the OCTA image 301 as input data and a rectangular area image 321 as a corresponding imaging area in the OCTA image 302 as output data.

At the time of learning, the scan range (shooting angle of view) and the scan density (the number of A-scans and the number of B-scans) are normalized to make the image size uniform, so that the rectangular area size at the time of learning can be made uniform. The rectangular area images shown in FIGS. 3A and 3B are examples of the rectangular area size when learning is performed separately.

In addition, the number of rectangular areas can be set to one in the example shown in FIG. 3A and plural in the example shown in FIG. 3B. For example, in the example illustrated in FIG. 3B, a pair may be formed using the rectangular area image 312 of the OCTA image 301 as input data and the rectangular area image 322 that is a corresponding imaging area in the high-quality OCTA image 302 as output data. it can. In this manner, a pair of rectangular area images different from each other can be created from a pair of an OCTA image and a high-quality OCTA image one by one. In addition, in the original OCTA image and the high-quality OCTA image, by creating a large number of rectangular area image pairs while changing the position of the area to different coordinates, it is possible to enrich the group of pairs forming the teacher data. .

Although the rectangular area is discretely shown in the example shown in FIG. 3B, the original OCTA image and the high-quality OCTA image are divided into a group of continuous rectangular area images having a constant image size without gaps. Can be. Alternatively, the original OCTA image and the high-quality OCTA image may be divided into rectangular area image groups at random positions corresponding to each other. As described above, by selecting an image of a smaller area as a rectangular area as a pair of input data and output data, a large amount of pair data is generated from the OCTA image 301 and the high-quality OCTA image 302 constituting the original pair. it can. Therefore, the time required for training the machine learning model can be reduced.

Next, as an example of the learned model according to the present embodiment, a convolutional neural network (CNN) that performs image quality improvement processing on an input tomographic image will be described with reference to FIG. FIG. 4 shows an example of the configuration 401 of the learned model used by the image quality improving unit 224.

学習 The learned model shown in FIG. 4 is composed of a plurality of layer groups responsible for processing the input value group and outputting the processed value group. Note that the types of layers included in the configuration 401 of the learned model include a convolution (Convolution) layer, a downsampling (Downsampling) layer, an upsampling (Upsampling) layer, and a synthesis (Merger) layer.

The convolution layer is a layer that performs a convolution process on an input value group according to parameters such as a set filter kernel size, the number of filters, a stride value, and a dilation value. The number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image.

(4) The downsampling layer is a layer that performs processing to reduce the number of output value groups from the number of input value groups by thinning out or combining input value groups. Specifically, for example, there is a Max @ Pooling process.

(4) The upsampling layer is a layer that performs processing for increasing the number of output value groups beyond the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. Specifically, as such processing, for example, there is a linear interpolation processing.

The composition layer is a layer that performs a process of inputting a value group such as an output value group of a certain layer or a pixel value group constituting an image from a plurality of sources, concatenating them, and adding them to combine them.

As parameters set in the convolutional layer group included in the configuration 401 shown in FIG. 4, for example, by setting the kernel size of the filter to 3 pixels in width, 3 pixels in height, and 64 to the number of filters, a certain accuracy Image quality improvement processing is possible. However, it should be noted that if the parameter setting for the layer group or the node group forming the neural network is different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may be different. In other words, in many cases, appropriate parameters are different depending on the mode of implementation, and can be changed to preferable values as needed.

In addition to the method of changing the parameters as described above, by changing the configuration of the CNN, there is a case where the CNN can obtain better characteristics. The better characteristics include, for example, a higher accuracy of the image quality improvement processing, a shorter time of the image quality improvement processing, a shorter time required for training of the machine learning model, and the like.

Although not illustrated, as a modification example of the configuration of the CNN, for example, a batch normalization (Batch Normalization) layer or an activation layer using a normalized linear function (Rectifier Linear Unit) may be incorporated after the convolutional layer. .

(4) When data is input to a trained model of such a machine learning model, data according to the design of the machine learning model is output. For example, output data having a high possibility of corresponding to the input data is output according to the tendency trained using the teacher data. In the learned model according to the present embodiment, when the OCTA image 301 is input, a high-quality OCTA image 302 is output according to the tendency of training using the teacher data.

When learning is performed by dividing the image region, the learned model outputs a rectangular region image that is a high-quality OCTA image corresponding to each rectangular region. In this case, the image quality improving unit 224 first divides the OCTA image 301, which is the input image, into a rectangular area image group based on the image size at the time of learning, and inputs the divided rectangular area image group to the trained model. After that, the image quality improving unit 224 compares each of the rectangular area image groups, which are high-quality OCTA images output from the learned model, with the same positional relationship as each of the rectangular area image groups input to the learned model. And combine them. Accordingly, the image quality improving unit 224 can generate a high-quality OCTA image 302 corresponding to the input OCTA image 301.

Next, a series of image processing according to the present embodiment will be described with reference to FIGS. FIG. 5 is a flowchart of a series of image processing according to the present embodiment.

First, in step S501, the acquiring unit 210 acquires a plurality of three-dimensional tomographic information obtained by photographing the eye E several times. The obtaining unit 210 may obtain the tomographic information of the eye E using the OCT imaging unit 100, or may obtain the tomographic information from the storage unit 240 or another device connected to the control unit 200.

Here, a case where the tomographic information of the eye E is acquired using the OCT imaging unit 100 will be described. First, the operator sits down on the patient, who is the subject, in front of the scanning optical system 150, starts OCT imaging after performing alignment, inputting patient information and the like to the control unit 200, and the like. The drive control unit 230 of the control unit 200 drives the galvanomirror of the scanning unit 152, scans the substantially same portion of the eye to be examined a plurality of times, and acquires a plurality of tomographic information (interference signals) at the substantially same portion of the eye to be examined. . After that, the drive control unit 230 minutely drives the galvanomirror of the scanning unit 152 in the sub-scanning direction orthogonal to the main scanning direction, and acquires a plurality of pieces of tomographic information at another position (adjacent scanning line) of the eye E. I do. By repeating this control, the acquisition unit 210 acquires a plurality of three-dimensional tomographic information in a predetermined range of the eye E.

Next, in step S502, the tomographic image generation unit 221 generates a plurality of three-dimensional tomographic images based on the obtained plurality of three-dimensional tomographic information. Note that when the acquisition unit 210 acquires a plurality of three-dimensional tomographic images from another device connected to the storage unit 240 or the control unit 200 in step S501, step S502 may be omitted.

In step S503, the motion contrast generation unit 222 generates three-dimensional motion contrast data (three-dimensional motion contrast image) based on the plurality of three-dimensional tomographic images. Note that the motion contrast generation unit 222 may obtain a plurality of pieces of motion contrast data based on three or more tomographic images acquired for substantially the same location, and generate an average value of the pieces of motion contrast data as final motion contrast data. If the acquisition unit 210 acquires the three-dimensional motion contrast data from the storage unit 240 or another device connected to the control unit 200 in step S501, steps S502 and S503 may be omitted.

In step S504, the En-Face image generation unit 223 generates an OCTA image for the three-dimensional motion contrast data according to the instruction of the operator or based on a predetermined En-Face image generation range. If the acquisition unit 210 acquires an OCTA image from another device connected to the storage unit 240 or the control unit 200 in step S501, steps S502 to S504 may be omitted.

In step S505, the image quality improving unit 224 performs an image quality improving process on the OCTA image using the learned model. The image quality improving unit 224 inputs the OCTA image to the trained model, and generates a high-quality OCTA image based on the output from the trained model. When the learned model is learning by dividing the image region, the image quality improving unit 224 first divides the OCTA image, which is the input image, into a rectangular region image group based on the image size at the time of learning. Then, the divided rectangular area image group is input to the learned model. After that, the image quality improving unit 224 compares each of the rectangular area image groups, which are high-quality OCTA images output from the learned model, with the same positional relationship as each of the rectangular area image groups input to the learned model. And combine them to generate a final high quality OCTA image.

In step S506, the display control unit 250 causes the display unit 270 to switch and display the high-quality OCTA image (second image) generated by the image quality improvement unit 224 with the original OCTA image (first image). . As described above, in the image quality improvement processing using the machine learning model, a blood vessel that does not actually exist may be drawn on the OCTA image, or a blood vessel that originally exists may be erased. On the other hand, the display control unit 250 causes the display unit 270 to switch the generated high-quality OCTA image to the original OCTA image and display the same, so that the blood vessel newly generated by the image quality improvement processing or the original blood vessel is displayed. It is possible to easily determine whether the blood vessel is also present in the image. When the display processing by the display control unit 250 ends, a series of image processing ends.

Next, an operation method of the control unit 200 will be described with reference to FIGS. 6A to 7. 6A and 6B show an example of a report screen for switching and displaying images before and after the image quality improvement processing. A report screen 600 shown in FIG. 6A shows a tomographic image 611 and an OCTA image 601 before the image quality improvement processing. A report screen 600 shown in FIG. 6B shows a tomographic image 611 and an OCTA image 602 (high-quality OCTA image) after the image quality improvement processing.

In the report screen 600 shown in FIG. 6A, when the operator presses the right button of the mouse on the OCTA image 601 using a mouse as an example of the input unit 260, a pop-up menu 620 for selecting whether or not to perform the image quality improvement processing Is displayed. When the operator selects the image quality improvement processing on the pop-up menu 620, the image quality improvement unit 224 executes the image quality improvement processing on the OCTA image 601.

Then, as illustrated in FIG. 6B, the display control unit 250 switches the OCTA image 601 displayed on the report screen 600 before performing the image quality improvement processing to the OCTA image 602 after performing the image quality improvement processing and displays the same. . By pressing the right mouse button again on the OCTA image 602, the pop-up menu 620 can be opened and the OCTA image 601 can be switched to the image 601 before the image quality improvement processing is performed and displayed.

Although the example in which the image is displayed before and after the image quality improvement process is displayed by the pop-up menu 620 displayed in response to the pressing of the right button of the mouse, the image is switched by any method other than the pop-up menu. Is also good. For example, the image may be switched by a button (for example, the button 3420 in FIG. 18, FIG. 20A and FIG. 20B) arranged on the report screen, a pull-down menu, a radio button, a check box, a keyboard operation, or the like. Further, the image may be switched and displayed by a mouse wheel operation or a touch operation of the touch panel display.

The operator can arbitrarily switch and display the OCTA image 601 before performing the image quality improvement processing and the OCTA image 602 after performing the image quality improvement processing by the above method. Therefore, the operator can easily compare and compare the OCTA images before and after the image quality improvement processing, and can easily confirm a change in the OCTA image due to the image quality improvement processing. Therefore, the operator can easily identify a blood vessel that does not actually exist in the OCTA image due to the image quality improvement processing, or even if the blood vessel that originally exists has disappeared. It is possible to easily determine the authenticity of the organization described in the above.

In the above-described display method, the images before and after the image quality improvement processing are switched and displayed. However, similar effects can be obtained by displaying these images side by side or overlapping them. FIG. 7 shows an example of a report screen when images before and after image quality improvement processing are displayed side by side. On a report screen 700 shown in FIG. 7, an OCTA image 701 before the image quality improvement processing and an OCTA image 702 after the image quality improvement processing are displayed side by side.

でも Even in this case, the operator can easily compare and compare the images before and after the image quality improvement processing, and can easily confirm a change in the image due to the image quality improvement processing. Therefore, the operator can easily identify a blood vessel that does not actually exist in the OCTA image by the image quality improvement processing, or even if the blood vessel that originally exists disappears, and can easily identify the blood vessel. The authenticity of the depicted organization can be easily determined. When displaying the images before and after the image quality improvement processing in a superimposed manner, the display control unit 250 sets the transparency of at least one of the images before and after the image quality improvement processing, and displays the images before and after the image quality improvement processing on the display unit 270. They can be superimposed and displayed.

As described above, the image quality improving unit 224 may perform the image quality improving process using the learned model not only on the OCTA image but also on the tomographic image and the En-Face image of the luminance. In this case, as a pair of the teacher data of the trained model, the tomographic image and the luminance En-Face image before superimposition are used as input data, and the tomographic image and the luminance En-Face image after superimposition are used as output data. Can be used. In this case, the trained model may be a single trained model trained using teacher data such as an OCTA image or a tomographic image, or a plurality of trained models trained for each type of image. It may be. When a plurality of learned models are used, the image quality improvement unit 224 can use a learned model corresponding to the type of an image on which the image quality improvement processing is performed. Note that the image quality improvement unit 224 may perform image quality improvement processing using a learned model on a three-dimensional motion contrast image or a three-dimensional tomographic image, and learning data in this case can be prepared in the same manner as described above.

In FIG. 7, a tomographic image 711 before the image quality improvement processing and a tomographic image 712 after the image quality improvement processing are displayed side by side. The display control unit 250 switches between a tomographic image before and after the image quality improvement processing and an En-Face image of the luminance before and after the image quality improvement processing as shown in the OCTA images before and after the image quality improvement processing shown in FIGS. You may. In addition, the display control unit 250 may cause the display unit 270 to superimpose the tomographic image before and after the image quality improvement processing or the En-Face image of the luminance. Even in these cases, the operator can easily compare the images before and after the image quality improvement processing, and can easily confirm a change in the image due to the image quality improvement processing. Therefore, the operator can easily identify a tissue that does not actually exist in the image due to the image quality improvement process, or even if the tissue that originally exists disappears, and can easily identify the tissue on the image. The authenticity of the performed organization can be easily determined.

As described above, the control unit 200 according to the present embodiment includes the image quality improvement unit 224 and the display control unit 250. The image quality improving unit 224 uses the learned model to generate, from the first image of the subject's eye, a second image in which at least one of noise reduction and contrast enhancement has been performed compared to the first image. . The display control unit 250 causes the display unit 270 to switch between the first image and the second image and display them side by side or overlapped. The display control unit 250 can switch between the first image and the second image and display the first image and the second image on the display unit 270 according to an instruction from the operator.

Thereby, the control unit 200 can generate a high-quality image in which noise is reduced or contrast is enhanced from the original image. For this reason, the control unit 200 can generate an image more suitable for image diagnosis than before, such as a clearer image, an image in which a site to be observed or a lesion is emphasized, and the like.

Also, the operator can easily compare the images before and after the image quality improvement processing, and can easily confirm a change in the image due to the image quality improvement processing. Therefore, the operator can easily identify a tissue that does not actually exist in the image due to the image quality improvement process, or even if the tissue that originally exists disappears, and can easily identify the tissue on the image. The authenticity of the performed organization can be easily determined.

In the learned model according to the present embodiment, the superimposed image is used as the output data of the teacher data, but the teacher data is not limited to this. For example, a high-quality image obtained by performing a maximum posterior probability estimation process (MAP estimation process) on an original image group may be used as output data of teacher data. In the MAP estimation processing, a likelihood function is obtained from the probability density of each pixel value in a plurality of images, and a true signal value (pixel value) is estimated using the obtained likelihood function.

The high-quality image obtained by the MAP estimation process becomes a high-contrast image based on pixel values close to the true signal value. In addition, since the estimated signal value is obtained based on the probability density, in the high-quality image obtained by the MAP estimation processing, randomly generated noise is reduced. For this reason, by using the high-quality image obtained by the MAP estimation processing as the teacher data, the learned model can reduce the noise or increase the contrast from the input image and obtain a high-quality image suitable for image diagnosis. Images can be generated. The method of generating the pair of the input data and the output data of the teacher data may be performed in the same manner as the case where the superimposed image is used as the teacher data.

Also, a high-quality image obtained by applying a smoothing filter process to the original image may be used as the output data of the teacher data. In this case, the trained model can generate a high-quality image with reduced random noise from the input image. Further, as the output data of the teacher data, an image obtained by applying a gradation conversion process to the original image may be used. In this case, the learned model can generate a high-quality image in which contrast is enhanced from the input image. The method of generating the pair of the input data and the output data of the teacher data may be performed in the same manner as the case where the superimposed image is used as the teacher data.

Note that the input data of the teacher data may be an image acquired from an imaging device having the same image quality tendency as the OCT imaging unit 100. Further, the output data of the teacher data may be a high-quality image obtained by a high-cost process such as a successive approximation method, or a subject corresponding to the input data may have a higher performance than the OCT imaging unit 100. It may be a high-quality image obtained by shooting with a shooting device. Furthermore, the output data may be a high-quality image acquired by performing a noise reduction process based on a rule based on the structure of the subject. Here, the noise reduction process may include, for example, a process of replacing a high-luminance pixel of only one pixel which clearly appears in a low-luminance region with an average value of neighboring low-luminance pixel values. . For this reason, the trained model uses an image captured by an imaging device having a higher performance than an imaging device used for capturing an input image, or an image acquired in an imaging process that requires more man-hours than the input image capturing process. It may be data.

Note that the image quality improvement unit 224 generates a high-quality image in which noise is reduced or contrast is enhanced using the learned model, but the image quality improvement processing by the image quality improvement unit 224 has been described. Is not limited to this. The image quality improving unit 224 only needs to be able to generate an image having an image quality more suitable for image diagnosis as described above by the image quality improving process.

When the display control unit 250 causes the display unit 270 to display the images before and after the image quality improvement processing side by side, according to an instruction from the operator, the display control unit 250 selects any one of the images before and after the image quality improvement processing displayed side by side on the display unit 270. May be enlarged and displayed. More specifically, for example, when the operator selects the OCTA image 701 on the report screen 700 illustrated in FIG. 7, the display control unit 250 can enlarge and display the OCTA image 701 on the report screen 700. Further, when the operator selects the OCTA image 702 after the image quality improvement processing, the display control unit 250 can enlarge and display the OCTA image 702 on the report screen 700. In this case, the operator can observe the image to be observed among the images before and after the image quality improvement processing in more detail.

Further, when the generation range of an En-Face image such as an OCTA image is changed in response to an instruction from the operator, the control unit 200 changes the images displayed side by side to an image based on the changed generation range and a high image quality. The image may be changed and displayed. More specifically, when the operator changes the generation range of the En-Face image via the input unit 260, the En-Face image generation unit 223 uses the changed generation range to generate the En-en before the image quality improvement processing. Generate a Face image. The image quality improvement unit 224 generates a high-quality En-Face image from the En-Face image newly generated by the En-Face image generation unit 223 using the learned model. After that, the display control unit 250 changes the En-Face images before and after the image quality improvement processing displayed side by side on the display unit 270 to the newly generated En-Face images before and after the image quality improvement processing and displays the images. In such a case, it is possible to observe the En-Face images before and after the image quality improvement processing based on the changed range in the depth direction while arbitrarily changing the range in the depth direction desired by the operator.

(Modification 1)
As described above, in an image on which image quality improvement processing has been performed using a trained model, a tissue that does not actually exist is depicted, or a tissue that originally exists disappears. Therefore, an erroneous diagnosis may occur when the operator performs an image diagnosis based on the image. Therefore, when displaying the OCTA image, the tomographic image, and the like after the image quality improvement processing on the display unit 270, the display control unit 250 confirms that the image is an image on which the image quality improvement processing has been performed using the learned model. It may be displayed. In this case, occurrence of erroneous diagnosis by the operator can be suppressed. The display mode may be any mode as long as the mode can be understood as a high-quality image acquired using the learned model.

(Modification 2)
The first embodiment has described the example in which the image quality improvement processing is applied to an OCTA image, a tomographic image, and the like obtained by one imaging (inspection). On the other hand, image quality improvement processing using a learned model can be applied to a plurality of OCTA images, tomographic images, and the like obtained by a plurality of imagings (inspections). Modification 2 will be described with reference to FIGS. 8A and 8B, in which a plurality of OCTA images, tomographic images, and the like are simultaneously displayed with images to which image quality improvement processing using a learned model is applied.

FIGS. 8A and 8B show an example of a time-series report screen for displaying a plurality of OCTA images obtained by imaging the same subject's eye a plurality of times over time. On the report screen 800 shown in FIG. 8A, a plurality of OCTA images 801 before the image quality improvement processing is performed are displayed in chronological order. The report screen 800 also includes a pop-up menu 820, and the operator can select whether or not to apply the image quality improvement processing by operating the pop-up menu 820 via the input unit 260.

(4) When the operator selects the application of the image quality improvement processing, the image quality improvement unit 224 applies the image quality improvement processing using the learned model to all the displayed OCTA images. Then, as illustrated in FIG. 8B, the display control unit 250 switches the plurality of OCTA images 802 after the image quality improvement processing is performed to the displayed plurality of OCTA images 801 and displays them.

When the operator selects not to apply the image quality improvement processing in the pop-up menu 820, the display control unit 250 replaces the plurality of OCTA images 801 before the image quality improvement processing with the plurality of OCTA images 801 displayed after the image quality improvement processing. Of the OCTA image 802 of FIG.

In this modification, an example has been described in which a plurality of OCTA images before and after image quality improvement processing using a learned model are simultaneously switched and displayed. However, a plurality of tomographic images before and after the image quality improvement processing using the learned model, an En-Face image of luminance, and the like may be simultaneously switched and displayed. The operation method is not limited to the method using the pop-up menu 820, but employs any operation method such as a button, a pull-down menu, a radio button, a check box, a keyboard, a mouse wheel, or a touch panel operation arranged on the report screen. May do it.

(Example 2)
The trained model outputs output data likely to correspond to the input data according to the tendency of learning. In this regard, when the learned model learns using a group of images having similar image quality trends as teacher data, an image having a higher image quality is output more effectively for images having similar similarities. be able to. Therefore, in the second embodiment, the image quality improvement processing is performed by using a plurality of learned models that are learned using the imaging data such as the imaging region and the teacher data formed by the group of pairs grouped for each generation range of the En-Face image. Thus, the image quality improvement processing is performed more effectively.

Hereinafter, the OCT apparatus according to the present embodiment will be described with reference to FIGS. Note that the configuration of the OCT apparatus according to the present embodiment is the same as that of the OCT apparatus 1 according to the first embodiment except for a control unit. Therefore, the same reference numerals are used for the same configurations as those illustrated in FIG. And the explanation is omitted. Hereinafter, the OCT apparatus according to the present embodiment will be described focusing on differences from the OCT apparatus 1 according to the first embodiment.

FIG. 9 shows a schematic configuration of the control unit 900 according to the present embodiment. The configuration of the control unit 900 according to the present embodiment other than the image processing unit 920 and the selection unit 925 is the same as each configuration of the control unit 200 according to the first embodiment. Therefore, the same components as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof is omitted.

The image processing unit 920 of the control unit 900 includes a selection unit 925 in addition to the tomographic image generation unit 221, the motion contrast generation unit 222, the En-Face image generation unit 223, and the image quality improvement unit 224.

The selecting unit 925 selects a learned model to be used by the image quality improving unit 224 from among the plurality of learned models based on a shooting condition of an image to be subjected to the image quality improving process by the image quality improving unit 224 and a generation range of the En-Face image. Select The image quality improvement unit 224 performs image quality improvement processing on a target OCTA image, tomographic image, or the like using the learned model selected by the selection unit 925, and generates a high-quality OCTA image or high-quality tomographic image. .

Next, a plurality of learned models according to the present embodiment will be described. As described above, the learned model outputs output data that is highly likely to correspond to the input data according to the tendency of learning. In this regard, when the learned model learns using a group of images having similar image quality trends as teacher data, an image having a higher image quality is output more effectively for images having similar similarities. be able to. Therefore, in the present embodiment, a pair group is grouped for each imaging condition including an imaging region, an imaging method, an imaging region, an imaging angle of view, a scan density, an image resolution, and the like, and an En-Face image generation range. A plurality of learned models learned using the learned teacher data are prepared.

More specifically, for example, a plurality of learned models such as a learned model using an OCTA image in which a macular portion is an imaging region as teacher data and a learned model using an OCTA image in which a nipple is an imaging region as teacher data are used. Prepare a model. Note that the macula and the nipple are examples of the imaging region, and may include other imaging regions. Further, a learned model may be prepared in which an OCTA image for each specific imaging region in an imaging region such as a macula or a nipple is used as teacher data.

Also, for example, when the retina is photographed at a wide angle of view and low density, and when the retina is photographed at a narrow angle of view and high density, the depiction of a structure such as a blood vessel depicted in the OCTA image is significantly different. Therefore, a learned model may be prepared in which learning is performed for each piece of teacher data according to the shooting angle of view and the scan density. Further, as examples of the imaging method, there are imaging methods such as SD-OCT and SS-OCT, and the image quality, the imaging range, and the degree of depth reach differ depending on the difference between these imaging methods. For this reason, a learned model in which learning has been performed for each piece of teacher data according to the imaging method may be prepared.

Further, it is rare to generate an OCTA image in which blood vessels of all layers of the retina are extracted at once, and it is general to generate an OCTA image in which only blood vessels existing in a predetermined depth range are extracted. . For example, in a depth range such as a shallow layer, a deep layer, an outer layer, and a choroid shallow layer of the retina, an OCTA image in which blood vessels are extracted in respective depth ranges is generated. On the other hand, the form of the blood vessel depicted in the OCTA image varies greatly depending on the depth range. For example, blood vessels visualized in the shallow layer of the retina form a low-density, thin and clear blood vessel network, while blood vessels visualized in the shallow layer of the choroid are dense and clearly distinguish individual blood vessels. It is difficult. For this reason, a learned model may be prepared in which learning is performed for each teacher data according to the generation range of an En-Face image such as an OCTA image.

Here, the example in which the OCTA image is used as the teacher data has been described. However, when the image quality improvement processing is performed on the tomographic image, the En-Face image of the luminance, and the like, as in the first embodiment, these images are used as the teacher data. be able to. In this case, a plurality of learned models that have learned for each teacher data according to the shooting conditions of these images and the generation range of the En-Face image are prepared.

Next, a series of image processing according to the present embodiment will be described with reference to FIG. FIG. 10 is a flowchart of a series of image processing according to the present embodiment. Note that a description of processing similar to a series of image processing according to the first embodiment will be appropriately omitted.

First, in step S1001, similarly to step S501 according to the first embodiment, the acquisition unit 210 acquires a plurality of three-dimensional tomographic information obtained by imaging the eye E multiple times. The obtaining unit 210 may obtain the tomographic information of the eye E using the OCT imaging unit 100, or may obtain the tomographic information from the storage unit 240 or another device connected to the control unit 200.

(4) The acquisition unit 210 acquires a group of imaging conditions related to tomographic information. Specifically, the acquisition unit 210 can acquire imaging conditions such as an imaging region and an imaging method when imaging is performed on tomographic information. Note that the acquisition unit 210 may acquire a group of imaging conditions stored in a data structure configuring data of the tomographic information according to the data format of the tomographic information. Further, when the imaging condition is not stored in the data structure of the tomographic information, the acquisition unit 210 can acquire the imaging information group from a server or a database that stores a file describing the imaging condition. In addition, the acquisition unit 210 may estimate the imaging information group from an image based on the tomographic information by any known method.

In the case where the acquisition unit 210 acquires a plurality of three-dimensional tomographic images, three-dimensional motion contrast data, OCTA images, and the like, the acquisition unit 210 acquires a group of imaging conditions related to the acquired images and data. When only a plurality of learned models that have been trained for each teacher data according to the generation range of the OCTA image or the En-Face image of the luminance are used for the image quality improvement processing, the acquisition unit 210 sets the imaging conditions of the tomographic image. It is not necessary to acquire a group.

Steps S1002 to S1004 are the same as steps S502 to S504 according to the first embodiment, and thus description thereof is omitted. In step S1004, when the En-Face image generation unit 223 generates an OCTA image, the process proceeds to step S1005.

In step S1005, the selection unit 925 selects a learned model to be used by the image quality improvement unit 224, based on the imaging condition group and generation range for the generated OCTA image and the information on the teacher data regarding the plurality of learned models. More specifically, for example, when the imaging region of the OCTA image is a nipple, the selection unit 925 selects a learned model in which learning has been performed using the OCTA image of the nipple as teacher data. Further, for example, when the generation range of the OCTA image is the shallow layer of the retina, the selection unit 925 selects a learned model in which learning has been performed using the OCTA image whose generation range is the shallow layer of the retina as teacher data. .

Note that the selecting unit 925 sets an image having a similar image quality as the teacher data even if the imaging condition group and the generation range of the generated OCTA image do not completely match the information of the teacher data of the trained model. A learned model that has undergone learning may be selected. In this case, for example, the selection unit 925 may include a table in which the correspondence between the imaging condition group and the generation range related to the OCTA image and the learned model to be used is described.

In step S1006, the image quality improving unit 224 performs an image quality improving process on the OCTA image generated in step S1004 using the learned model selected by the selecting unit 925, and generates a high-quality OCTA image. The method of generating a high-quality OCTA image is the same as that in step S505 according to the first embodiment, and a description thereof will not be repeated.

Step S1007 is the same as step S506 according to the first embodiment, and a description thereof will not be repeated. In step S1007, when a high-quality OCTA image is displayed on the display unit 270, a series of image processing according to the present embodiment ends.

As described above, the control unit 900 according to the present embodiment includes the selection unit 925 that selects a learned model used by the image quality improving unit 224 from a plurality of learned models. The selecting unit 925 selects a learned model used by the image quality improving unit 224 based on a range in the depth direction for generating an OCTA image to be subjected to the image quality improving process.

For example, the selecting unit 925 can select a learned model based on a display region in an OCTA image to be subjected to image quality improvement processing and a range in a depth direction for generating the OCTA image. In addition, for example, the selection unit 925 may use the learning used by the image quality improvement unit 224 based on the imaging region including the display region in the OCTA image to be subjected to the image quality improvement process and the range in the depth direction for generating the OCTA image. May be selected. Further, for example, the selection unit 925 may select a learned model used by the image quality improvement unit 224 based on the imaging condition of the OCTA image to be subjected to the image quality improvement process.

For this reason, the control unit 900 performs image quality improvement processing by using a plurality of learned models that have been learned using teacher data configured by pairs of groups that are grouped according to shooting conditions and generation ranges of En-Face images. Image quality improvement processing can be performed more effectively.

In the present embodiment, an example has been described in which the selection unit 925 selects a learned model based on an imaging condition such as an imaging region of the OCTA image or a generation range. However, the learned model is selected based on conditions other than the above. May be changed. The selection unit 925 determines, for example, a projection method (maximum intensity projection method or average intensity projection method) when generating an OCTA image or an En-Face image of luminance, and the presence or absence of an artifact removal process caused by a blood vessel shadow. A learned model may be selected. In this case, it is possible to prepare a learned model in which learning is performed for each teacher data according to the projection method and the presence or absence of the artifact removal processing.

(Modification 3)
In the second embodiment, the selection unit 925 automatically selects an appropriate learned model according to the shooting conditions, the generation range of the En-Face image, and the like. On the other hand, there are cases where the operator desires to manually select the image quality improvement processing to be applied to the image. Therefore, the selection unit 925 may select a learned model according to an instruction of the operator.

In some cases, the operator desires to change the image quality improvement processing applied to the image. Therefore, the selection unit 925 may change the learned model and change the image quality improvement processing applied to the image in accordance with the instruction of the operator.

Hereinafter, with reference to FIGS. 11A and 11B, an operation method when manually changing the image quality improvement processing applied to an image will be described. 11A and 11B show an example of a report screen that switches and displays images before and after the image quality improvement processing. A report screen 1100 shown in FIG. 11A shows an OCTA image 1101 to which an image quality improvement process using a tomographic image 1111 and an automatically selected learned model has been applied. The report screen 1100 shown in FIG. 11B shows the tomographic image 1111 and the OCTA image 1102 to which the image quality improvement processing using the learned model according to the instruction of the operator has been applied. Further, the report screen 1100 shown in FIGS. 11A and 11B shows a process designation unit 1120 for changing the image quality improvement process applied to the OCTA image.

Here, the OCTA image 1101 displayed on the report screen 1100 shown in FIG. 11A depicts a deep blood vessel (Deep @ Capillary) in the macula. On the other hand, the image quality improvement processing applied to the OCTA image using the learned model automatically selected by the selection unit 925 is suitable for a shallow layer blood vessel (RPC) of the papilla. Therefore, regarding the OCTA image 1101 displayed on the report screen 1100 shown in FIG. 11A, the image quality improvement processing applied to the OCTA image is not optimal for the blood vessels extracted in the OCTA image.

Therefore, the operator selects Deep @ Capillary in the process specification unit 1120 via the input unit 260. The selection unit 925 changes the trained model used by the image quality improving unit 224 to a trained model that has been trained using the OCTA image relating to the deep blood vessels of the macula as teacher data in response to a selection instruction from the operator.

(4) The image quality improvement unit 224 performs the image quality improvement process on the OCTA image again using the learned model changed by the selection unit 925. The display control unit 250 causes the display unit 270 to display the high-quality OCTA image 1102 newly generated by the image quality improvement unit 224, as shown in FIG. 11B.

In this way, by configuring the selection unit 925 to change the learned model in accordance with the operator's instruction, the operator can re-designate an appropriate image quality improvement process for the same OCTA image. it can. The specification of the image quality improvement processing may be performed many times.

Here, an example has been described in which the control unit 900 is configured so that the image quality improvement processing applied to the OCTA image can be manually changed. On the other hand, the control unit 900 may be configured to be able to manually change the image quality improvement processing applied to the tomographic image, the luminance En-Face image, and the like.

The report screens shown in FIGS. 11A and 11B have a mode in which the images before and after the image quality improvement processing are switched and displayed. However, the report screens in which the images before and after the image quality improvement processing are displayed side by side or overlapped are displayed. It may be a screen. Furthermore, the mode of the process designating unit 1120 is not limited to the modes shown in FIGS. 11A and 11B, and may be any mode that can indicate the image quality improvement processing or the learned model. Further, the types of the image quality improvement processing shown in FIGS. 11A and 11B are examples, and may include other types of the image quality improvement processing according to the teacher data for the learned model.

Also, similarly to the second modification, a plurality of images to which the image quality improvement processing has been applied may be simultaneously displayed. At this time, it is also possible to configure so that it is possible to specify which image quality improvement processing is to be applied. An example of the report screen in this case is shown in FIGS. 12A and 12B.

FIGS. 12A and 12B show an example of a report screen for switching and displaying a plurality of images before and after the image quality improvement processing. The OCTA image 1201 before the image quality improvement processing is shown on the report screen 1200 shown in FIG. 12A. The OCTA image 1202 to which the image quality improvement processing according to the operator's instruction is applied is shown on the report screen 1200 shown in FIG. 12B. A report screen 1200 shown in FIGS. 12A and 12B shows a process designation unit 1220 for changing the image quality improvement process applied to the OCTA image.

In this case, the selection unit 925 selects a learned model corresponding to the image quality improvement processing instructed using the processing designation unit 1220 as a learned model used by the image quality improvement unit 224. The image quality improvement unit 224 performs image quality improvement processing on the plurality of OCTA images 1201 using the learned model selected by the selection unit 925. The display control unit 250 causes the generated plurality of high-quality OCTA images 1202 to be displayed on the report screen 1200 at a time as shown in FIG. 12B.

Although the image quality improvement processing for the OCTA image has been described, the learned model may be selected and changed in accordance with the operator's instruction with respect to the image quality improvement processing for the tomographic image, the luminance En-Face image, and the like. . A plurality of images before and after the image quality improvement processing may be displayed side by side on the report screen, or may be displayed in an overlapping manner. Also in this case, a plurality of images to which the image quality improvement processing has been applied according to the instruction from the operator can be displayed at a time.

(Example 3)
In the first and second embodiments, the image quality improving unit 224 automatically executes the image quality improving process after capturing a tomographic image or an OCTA image. However, the image quality improvement processing performed by the image quality improvement unit 224 using the learned model may take a long time in some cases. Also, generation of motion contrast data by the motion contrast generation unit 222 and generation of an OCTA image by the En-Face image generation unit 223 require time. Therefore, when displaying an image after the image quality improvement processing is completed after shooting, it may take a long time from shooting to display.

On the other hand, in the imaging of the subject's eye using the OCT apparatus, the imaging may fail due to blinking or unintended movement of the subject's eye. Therefore, the convenience of the OCT apparatus can be improved by confirming the success or failure of imaging at an early stage. Therefore, in the third embodiment, prior to generation and display of a high-quality OCTA image, an En-Face image or an OCTA image having a luminance based on tomographic information obtained by photographing the eye to be inspected is displayed at an early stage. The OCT apparatus is configured so that a captured image can be confirmed.

Hereinafter, an OCT apparatus according to the present embodiment will be described with reference to FIG. Since the configuration of the OCT apparatus according to the present embodiment is the same as that of the OCT apparatus 1 according to the first embodiment, the same reference numerals are used and the description is omitted. Hereinafter, the OCT apparatus according to the present embodiment will be described focusing on differences from the OCT apparatus 1 according to the first embodiment.

FIG. 13 is a flowchart of a series of image processing according to the present embodiment. First, in step S1301, the acquiring unit 210 acquires a plurality of three-dimensional tomographic information by photographing the eye E by the OCT photographing unit 100.

Step S1302 is the same as step S502 according to the first embodiment, and a description thereof will not be repeated. When a three-dimensional tomographic image is generated in step S1302, the process proceeds to step S1303.

In step S1303, the En-Face image generation unit 223 generates a front image (luminance En-Face image) of the fundus by projecting the three-dimensional tomographic image generated in step S1302 onto a two-dimensional plane. After that, in step S1304, the display control unit 250 causes the display unit 270 to display the generated luminance En-Face image.

Steps S1305 and S1306 are the same as steps S503 and S504 according to the first embodiment, and a description thereof will not be repeated. When an OCTA image is generated in step S1306, the process proceeds to step S1307. In step S1307, the display control unit 250 switches the OCTA image before the image quality improvement processing generated in step S1306 to the luminance En-Face image and causes the display unit 270 to display it.

In step S1308, as in step S505 according to the first embodiment, the image quality improving unit 224 performs image quality improving processing on the OCTA image generated in step S1306 using the learned model, and obtains a high-quality OCTA image. Generate In step S1309, the display control unit 250 causes the display unit 270 to display the generated high-quality OCTA image by switching to the OCTA image before the image quality improvement processing.

As described above, before the acquisition unit 210 acquires the OCTA image, the display control unit 250 according to the present embodiment uses the En- of the luminance En- which is the front image generated based on the tomographic data in the depth direction of the subject's eye. The face image (third image) is displayed on the display unit 270. In addition, immediately after the acquisition of the OCTA image, the display control unit 250 switches the displayed luminance En-Face image to the OCTA image, and causes the display unit 270 to display the OC-image. Further, after a high-quality OCTA image is generated by the image quality improving unit 224, the display control unit 250 switches the displayed OCTA image to a high-quality OCTA image and causes the display unit 270 to display the OCTA image.

This allows the operator to check the front image of the subject's eye immediately after imaging, and immediately determine the success or failure of imaging. Further, since the OCTA image is displayed immediately after the OCTA image is generated, the operator determines at an early stage whether or not a plurality of three-dimensional tomographic information for generating motion contrast data has been appropriately acquired. You can judge.

For the tomographic image, the luminance En-Face image, and the like, by displaying the tomographic image, the luminance En-Face image, and the like before performing the image quality improvement processing, the operator determines the success or failure of the imaging at an early stage. be able to.

In the present embodiment, the motion contrast data generation process (step S1305) is started after the brightness En-Face image display process (step S1304), but the timing of the motion contrast data generation process is not limited to this. . For example, the motion contrast generation unit 222 may start the generation process of the motion contrast data in parallel with the generation process (Step S1303) and the display process (Step S1304) of the luminance En-Face image. Similarly, the image quality improving unit 224 may start the image quality improving process (Step S1308) in parallel with the OCTA image display process (Step S1307).

(Example 4)
In the first embodiment, the example in which the OCTA images before and after the image quality improvement processing are switched and displayed has been described. On the other hand, in the fourth embodiment, the images before and after the image quality improvement processing are compared.

FIG. 14 shows a schematic configuration of the control unit 1400 according to the present embodiment. The configuration of the control unit 1400 according to the present embodiment other than the image processing unit 1420 and the comparison unit 1426 is the same as each configuration of the control unit 200 according to the first embodiment. Therefore, the same components as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof is omitted.

The image processing unit 1420 of the control unit 1400 is provided with a comparing unit 1426 in addition to the tomographic image generating unit 221, the motion contrast generating unit 222, the En-Face image generating unit 223, and the image quality improving unit 224.

The comparing unit 1426 compares the image before the image quality improvement processing is performed by the image quality improvement unit 224 (original image) with the image after the image quality improvement processing is performed. More specifically, the comparing unit 1426 compares the images before and after the image quality improvement processing, and calculates the difference between the pixel values at the corresponding pixel positions of the images before and after the image quality improvement processing.

{The comparison unit 1426 generates a color map image colored according to the magnitude of the difference value. For example, when the pixel value of the image after the image quality improvement processing is larger than that of the image before the image quality improvement processing, the color tone of the warm color (yellow to orange to red) is changed to the pixel of the image after the image quality improvement processing. If the value is small, use a cool (yellow-green to green-blue) color tone. By using such a color scheme, it is possible to easily identify a portion shown in a warm color system on the color map image as a tissue restored (or newly created) by the image quality improvement processing. Similarly, it is possible to easily identify a portion indicated by a cool color system on the color map image as noise (or a tissue that has been erased) removed by the image quality improvement processing.

The color arrangement of the color map image is an example. For example, the color scheme of the color map image is arbitrarily set according to a desired configuration, such as performing a different color tone according to the magnitude of the pixel value in the image after the image quality improvement processing with respect to the pixel value in the image before the image quality improvement processing. May be.

The display control unit 250 can display the color map image generated by the comparison unit 1426 on the display unit 270 by superimposing the color map image on the image before the image quality improvement processing or the image after the image quality improvement processing.

Next, a series of image processing according to the present embodiment will be described with reference to FIG. Steps S1501 to S1505 are the same as steps S501 to S505 according to the first embodiment, and a description thereof will not be repeated. If a high-quality OCTA image is generated by the image quality improving unit 224 in step S1505, the process proceeds to step S1506.

In step S1506, the comparing unit 1426 compares the OCTA image generated in step S1504 with the high-quality OCTA image generated in step S1505 to calculate a difference between pixel values, and based on the difference between pixel values. Generate a color map image. Note that the comparing unit 1426 compares the images using another method such as a pixel value ratio or a correlation value between the images before and after the high image quality processing, instead of the difference between the pixel values in the images before and after the high image quality processing. A color map image may be generated based on the result.

In step S1507, the display control unit 250 causes the display unit 270 to superimpose the color map image on the image before the image quality improvement processing or the image after the image quality improvement processing. At this time, the display control unit 250 can set the transparency of the color map and superimpose the color map image on the target image so as not to hide the image on which the color map image is superimposed.

In addition, the display control unit 250 sets a high transparency at a place where the difference between the images before and after the image quality improvement processing is small (the pixel value of the color map image is low) in the color map image, The transparency may be set so that is completely transparent. By doing so, both the image displayed below the color map image and the color map image can be viewed well. As for the transparency of the color map image, the comparing unit 1426 may generate a color map image including the transparency setting.

As described above, the control unit 1400 according to the present embodiment includes the comparison unit 1426 that compares the first image with the second image on which the image quality improvement processing has been performed. The comparing unit 1426 calculates a difference between the first image and the second image, and generates a color map image that is color-coded based on the difference. The display control unit 250 controls the display on the display unit 270 based on the comparison result by the comparison unit 1426. More specifically, the display control unit 250 causes the display unit 270 to display a color map image superimposed on the first image or the second image.

Thereby, by observing the color map image superimposed on the image before and after the image quality improvement processing, it is possible to more easily confirm the change of the image due to the image quality improvement processing. For this reason, the operator should be able to more easily identify a tissue that does not actually exist in the image due to the image quality improvement process, or even if the original tissue disappears. And the authenticity of the organization can be determined more easily. In addition, the operator can easily identify, according to the color arrangement of the color map image, whether the part is a part newly drawn or erased by the image quality improvement processing.

The display control unit 250 can enable or disable the superimposed display of the color map image in accordance with an instruction from the operator. The on / off operation of the superimposed display of the color map image may be simultaneously applied to a plurality of images displayed on the display unit 270. In this case, the comparison unit 1426 generates a color map image for each of the images before and after the corresponding image quality improvement process, and the display control unit 250 converts the color map image into an image before the image quality improvement process or an image after the image quality improvement process. Can be superimposed. The display control unit 250 may cause the display unit 270 to display an image before the image quality improvement processing or an image after the image quality improvement processing before displaying the color map image.

In the present embodiment, an OCTA image has been described as an example. However, a similar process can be performed when an image quality improvement process is performed on a tomographic image, an En-Face image of luminance, or the like. The comparison processing and the color map display processing according to the present embodiment can be applied to the OCT apparatuses according to the second and third embodiments.

(Modification 4)
Further, the comparison unit 1426 may compare the images before and after the image quality improvement processing, and the display control unit 250 may display a warning on the display unit 270 according to the comparison result by the comparison unit 1426. More specifically, when the difference between the pixel values in the images before and after the image quality improvement processing calculated by the comparison unit 1426 is larger than a predetermined value, the display control unit 250 causes the display unit 270 to display a warning. According to such a configuration, in the generated high-quality image, if a tissue that does not actually exist is generated by the learned model, or an organization that originally exists is erased, The operator can be alerted. Note that the comparison between the difference and the predetermined value may be performed by the comparing unit 1426 or may be performed by the display control unit 250. Further, instead of the difference, a statistical value such as an average value of the difference may be compared with a predetermined value.

Further, when the difference between the images before and after the image quality improvement process is larger than a predetermined value, the display control unit 250 may not display the image after the image quality improvement process is performed on the display unit 270. In this case, in the generated high-quality image, if a tissue that does not actually exist is generated by the learned model or an existing tissue is erased, the high-quality image is generated. Erroneous diagnosis based on an image can be suppressed. Note that the comparison between the difference and the predetermined value may be performed by the comparing unit 1426 or may be performed by the display control unit 250. Further, instead of the difference, a statistical value such as an average value of the difference may be compared with a predetermined value.

(Example 5)
Next, an image processing apparatus (control unit 200) according to a fifth embodiment will be described with reference to FIGS. 20A and 20B. In the present embodiment, an example will be described in which the display control unit 250 displays the processing result of the image quality improving unit 224 on the display unit 270. In the present embodiment, description will be given with reference to FIGS. 20A and 20B, but the display screen is not limited to this. Like a follow-up observation, the image quality improvement processing (image quality improvement processing) can be similarly applied to a display screen in which a plurality of images obtained at different dates and times are displayed side by side. The image quality improvement processing can be similarly applied to a display screen in which the examiner confirms the success or failure of imaging immediately after imaging, such as an imaging confirmation screen. The display control unit 250 can cause the display unit 270 to display the plurality of high-quality images generated by the image quality improvement unit 224 and the low-quality images that have not been subjected to the high quality improvement. Thus, a low-quality image and a high-quality image can be output according to the instruction of the examiner.

Hereinafter, an example of the interface 3400 will be described with reference to FIGS. 20A and 20B. Reference numeral 3400 denotes the entire screen, 3401 denotes a patient tab, 3402 denotes an imaging tab, 3403 denotes a report tab, and 3404 denotes a setting tab. A hatched line in the report tab 3403 indicates an active state of the report screen. In the present embodiment, an example in which a report screen is displayed will be described. Im3405 displays an SLO image, and Im3406 displays an OCTA En-Face image indicated by Im3407 on the SLO image Im3405. Here, the SLO image is a front image of the fundus oculi acquired by an SLO (Scanning Laser Ophthalmoscope) optical system (not shown). Im3407 and Im3408 denote OCTA En-Face images, Im3409 denotes a luminance En-Face image, and Im3411 and Im3412 denote tomographic images.

Reference numerals

3413 and 3414 superimpose and display the boundaries of the upper and lower ranges of the OCTA En-Face image shown in Im3407 and Im3408, respectively, on the tomographic image. Button 3420 is a button for designating execution of the image quality improvement processing. Of course, as described later, the button 3420 may be a button for instructing display of a high-quality image.

In the present embodiment, the execution of the image quality improvement processing is performed by designating the button 3420, or the presence or absence of the execution is determined based on information stored (stored) in the database. First, an example will be described in which the button 3420 is designated in accordance with an instruction from the examiner to switch between the display of a high-quality image and the display of a low-quality image. Note that the target image of the image quality improvement processing will be described as an OCTA En-Face image.

(4) When the examiner designates the report tab 3403 and transits to the report screen, the low quality OCTA En-Face images Im3407 and Im3408 are displayed. Thereafter, when the examiner specifies the button 3420, the image quality improving unit 224 executes the image quality improvement processing on the images Im3407 and Im3408 displayed on the screen. After the image quality improvement processing is completed, the display control unit 250 displays the high quality image generated by the image quality improvement unit 224 on the report screen. In addition, since Im3406 is obtained by superimposing and displaying Im3407 on the SLO image Im3405, Im3406 also displays an image that has been subjected to high quality processing. Then, the display of the button 3420 is changed to the active state, and a display that indicates that the image quality improvement processing has been executed is displayed.

Here, the execution of the processing in the image quality improving unit 224 need not be limited to the timing at which the examiner has designated the button 3420. Since the types of the OCTA En-Face images Im3407 and Im3408 to be displayed when the report screen is opened are known in advance, the image quality improvement processing may be executed when the screen transitions to the report screen. Then, at the timing when the button 3420 is pressed, the display control unit 250 may display a high quality image on the report screen. Furthermore, there is no need for two types of images to be subjected to the image quality improvement processing in response to an instruction from the examiner or when transitioning to the report screen. Multiple OCTA En-Faces, such as surface (Im2910), deep (Im2920), outer (Im2930), and choroidal vascular networks (Im2940) as shown in FIG. 19A and FIG. Processing may be performed on an image. In this case, the image on which the image quality improvement processing has been performed may be temporarily stored in a memory or may be stored in a database.

Next, a description will be given of a case where the image quality improvement processing is executed based on the information stored (recorded) in the database. When the state in which the image quality improvement process is performed is stored in the database, a high quality image obtained by performing the image quality improvement process is displayed by default when the report screen is displayed. Then, the button 3420 is displayed as an active state by default, so that the examiner can recognize that the high-quality image obtained by executing the high-quality processing is displayed. . If the examiner wants to display a low-quality image before the high-quality processing, the examiner can display the low-quality image by specifying the button 3420 to release the active state. To return to the high-quality image, the examiner specifies the button 3420.

(4) Whether or not the image quality improvement processing is performed on the database is specified for each layer, such as common to all the data stored in the database and for each photographing data (for each inspection). For example, when the state in which the image quality improvement processing is executed is stored for the entire database, the state in which the examiner does not execute the image quality improvement processing is stored for individual photographing data (individual examination). In this case, when the image data is displayed next time, the image data is displayed without performing the image quality improvement processing. A user interface (not shown) (for example, a save button) may be used to save the execution state of the image quality improvement processing for each piece of imaging data (for each inspection). In addition, when transitioning to other imaging data (other examination) or other patient data (for example, changing to a display screen other than the report screen in response to an instruction from the examiner), the display state (for example, the button 3420 Based on the (state), the state in which the image quality improving process is performed may be stored. Accordingly, when the presence / absence of the execution of the image quality improvement processing is not specified in the imaging data unit (inspection unit), the processing is performed based on the information specified for the entire database, and in the imaging data unit (inspection unit). If specified, processing can be executed individually based on that information.

In the present embodiment, an example is shown in which Im3407 and Im3408 are displayed as the OCTA En-Face images, but the displayed OCTA En-Face images can be changed by the examiner's designation. Therefore, a description will be given of an image change when the execution of the image quality improvement processing is designated (the button 3420 is in the active state).

The image is changed using a user interface (not shown) (for example, a combo box). For example, when the examiner changes the type of image from the surface layer to the choroidal vascular network, the image quality improving unit 224 performs an image quality improving process on the choroidal vascular network image, and the display control unit 250 generates the image quality improving unit 224. The high-quality image on the report screen. That is, the display control unit 250 changes the display of the high-quality image in the first depth range to the high-quality image in the second depth range that is at least partially different from the first depth range in response to an instruction from the examiner. The display may be changed to an image display. At this time, the display control unit 250 changes the first depth range to the second depth range in response to an instruction from the examiner, thereby displaying the high-quality image in the first depth range on the second depth range. May be changed to display a high-quality image in a depth range of. As described above, if a high-quality image has already been generated for an image that is likely to be displayed when the report screen transitions, the display control unit 250 may display the generated high-quality image. .

The method of changing the type of image is not limited to the method described above, and it is also possible to generate an OCTA En-Face image in which a different depth range is set by changing a reference layer and an offset value. In this case, when the reference layer or the offset value is changed, the image quality improving unit 224 performs an image quality improving process on the En-Face image of an arbitrary OCTA, and the display control unit 250 Is displayed on the report screen. The reference layer and the offset value can be changed using a user interface (not shown) (for example, a combo box or a text box). Further, by dragging (moving the layer boundary) any one of the

boundaries

3413 and 3414 superimposed on the tomographic images Im3411 and Im3412, the generation range of the OCTA En-Face image can be changed.

する When the boundary line is changed by dragging, the execution instruction of the image quality improvement processing is continuously executed. Therefore, the image quality improving unit 224 may always process the execution command, or may execute the command after the layer boundary is changed by dragging. Alternatively, the execution of the image quality improvement processing is continuously instructed, but when the next instruction comes, the previous instruction may be canceled and the latest instruction may be executed.

The high-quality processing may take a relatively long time. Therefore, even if the command is executed at any timing described above, it may take a relatively long time before a high-quality image is displayed. Therefore, from when a depth range for generating an OCTA En-Face image is set in response to an instruction from the examiner until the high-quality image is displayed, the OCTA corresponding to the set depth range is displayed. An En-Face image (low-quality image) may be displayed. That is, when the depth range is set, the OCTA En-Face image (low-quality image) corresponding to the set depth range is displayed, and when the high-quality processing ends, the OCTA En-Face image is displayed. The display of the (low-quality image) may be changed to the display of a high-quality image. In addition, information indicating that the high-quality image processing is being performed may be displayed from when the depth range is set until the high-quality image is displayed. Note that these are not limited to the case where the execution of the image quality improvement processing has already been designated (the button 3420 is in the active state). For example, the execution of the image quality improvement processing is performed in response to an instruction from the examiner. Is applicable until the high-quality image is displayed.

In the present embodiment, an example is shown in which different layers are displayed on Im3407 and Im3408 as En-Face images of OCTA, and low-quality and high-quality images are switched and displayed. However, the present invention is not limited to this. For example, a low-quality OCTA En-Face image may be displayed side by side on the Im3407, and a high-quality OCTA En-Face image may be displayed side by side on the Im3408. When the images are switched and displayed, the images are switched at the same place, so that it is easy to compare the changed portions. When the images are displayed side by side, the images can be displayed at the same time, so that the entire image can be easily compared.

Next, with reference to FIGS. 20A and 20B, a description will be given of the execution of the image quality improvement processing at the screen transition. FIG. 20B is a screen example in which the Enta-Face image Im3407 of OCTA in FIG. 20A is enlarged and displayed. 20B, a button 3420 is displayed as in FIG. 20A. The screen transition from FIG. 20A to FIG. 20B is performed, for example, by double-clicking the En-Face image Im3407 of OCTA, and the screen transition from FIG. 20B to FIG. Note that the screen transition is not limited to the method shown here, and a user interface (not shown) may be used.

(4) When the execution of the image quality improvement processing is designated at the time of the screen transition (the button 3420 is active), the state is maintained even at the time of the screen transition. That is, in a case where a transition is made to the screen of FIG. 20B while the high-quality image is being displayed on the screen of FIG. 20A, the high-quality image is also displayed on the screen of FIG. 20B. Then, the button 3420 is activated. The same applies to the transition from FIG. 20B to FIG. 20A. In FIG. 20B, the display can be switched to a low quality image by designating a button 3420.

Regarding the screen transition, not only the screen shown here, but also a transition to a screen that displays the same shooting data such as a display screen for follow-up observation or a display screen for panorama, the display state of the high-quality image is maintained. Transition is performed as is. That is, an image corresponding to the state of button 3420 on the display screen before the transition is displayed on the display screen after the transition. For example, if the button 3420 on the display screen before the transition is in the active state, a high-quality image is displayed on the display screen after the transition. Further, for example, if the active state of the button 3420 on the display screen before the transition is released, a low-quality image is displayed on the display screen after the transition. When the button 3420 on the display screen for follow-up observation is activated, a plurality of images obtained at different dates and times (different examination dates) displayed side by side on the display screen for follow-up observation are switched to high-quality images. You may. That is, when the button 3420 on the display screen for follow-up observation is activated, the configuration may be such that the button 3420 is collectively reflected on a plurality of images obtained at different dates and times.

FIG. 18 shows an example of a display screen for follow-up observation. When the tab 3801 is selected according to an instruction from the examiner, a display screen for follow-up observation is displayed as shown in FIG. At this time, the examiner can change the depth range of the En-Face image by selecting from a predetermined depth range set (3802 and 3803) displayed in the list box. For example, in the list box 3802, the retinal surface layer is selected, and in the list box 3803, the retinal deep layer is selected. The upper display area displays the analysis result of the En-Face image of the retinal surface layer, and the lower display area displays the analysis result of the En-Face image of the deeper retina. That is, when the depth range is selected, the display of the analysis results of the plurality of En-Face images in the selected depth range is simultaneously changed for a plurality of images at different dates and times.

At this time, if the display of the analysis result is set to the non-selection state, the display may be changed to a parallel display of a plurality of En-Face images at different dates and times. When the button 3420 is designated in response to an instruction from the examiner, the display of the plurality of En-Face images is changed to the display of a plurality of high-quality images at once.

When the display of the analysis result is in the selected state, when the button 3420 is designated according to the instruction from the examiner, the display of the analysis result of the plurality of En-Face images is performed by the analysis of the plurality of high-quality images. It is changed to the display of the result at once. Here, the display of the analysis result may be a display in which the analysis result is superimposed on the image with an arbitrary transparency. At this time, the display of the analysis result may be changed, for example, to a state where the analysis result is superimposed on the displayed image with arbitrary transparency. The change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

種類 In addition, the type and offset position of the layer boundary used to specify the depth range can be changed collectively from a user interface such as 3805 and 3806. Note that the tomographic images are also displayed together, and the layer boundary data superimposed on the tomographic images is moved according to an instruction from the examiner, so that the depth ranges of a plurality of En-Face images at different dates and times are collectively determined. It may be changed. At this time, when a plurality of tomographic images at different dates and times are displayed side by side and the above-described movement is performed on one tomographic image, the layer boundary data may be similarly moved on other tomographic images.

The presence or absence of the image projection method or the projection artifact suppression process may be changed by selecting the user interface such as a context menu, for example.

Alternatively, the selection screen may be displayed by selecting the selection button 3807, and the image selected from the image list displayed on the selection screen may be displayed. Note that an arrow 3804 displayed at the top of FIG. 18 is a mark indicating that the test is currently selected, and the reference test (Baseline) is the test selected at the time of Follow-up imaging (one of FIG. 18). Left image). Of course, a mark indicating the reference inspection may be displayed on the display unit.

If the “Show @ Difference” check box 3808 is specified, a measured value distribution (map or sector map) for the reference image is displayed on the reference image. Further, in this case, a difference measurement value map between the measurement value distribution calculated for the reference image and the measurement distribution calculated for the image displayed in the region is displayed in an area corresponding to the other inspection dates. I do. As the measurement result, a trend graph (a graph of the measurement values for the images on each inspection day obtained by the measurement over time) may be displayed on the report screen. That is, time-series data (for example, a time-series graph) of a plurality of analysis results corresponding to a plurality of images at different dates and times may be displayed. At this time, the analysis results regarding the dates and times other than the multiple dates and times corresponding to the multiple displayed images are also distinguished from the multiple analysis results corresponding to the multiple displayed images (for example, time-series (The color of each point on the graph differs depending on whether an image is displayed or not). Further, a regression line (curve) of the trend graph and a corresponding mathematical expression may be displayed on a report screen.

In the present embodiment, the description has been given of the OCTA En-Face image, but the present invention is not limited to this. An image related to processing such as display, image quality improvement, and image analysis according to the present embodiment may be an En-Face image of luminance. Further, not only the En-Face image but also a different image such as a tomographic image, an SLO image, a fundus photograph, or a fluorescent fundus photograph may be used. In that case, the user interface for executing the image quality improvement processing is to instruct execution of the image quality improvement processing for a plurality of different types of images, or to select an arbitrary image from the plurality of types of the different images. There may be one that instructs execution of the high image quality processing.

With such a configuration, the display control unit 250 can display the image processed by the image quality improving unit 224 according to the present embodiment on the display unit 270. At this time, as described above, when at least one of a plurality of conditions regarding the display of the high-quality image, the display of the analysis result, the depth range of the front image to be displayed, and the like is selected, the display screen is displayed. The transition may be made, or the selected state may be maintained.

Further, as described above, when at least one of the plurality of conditions is in a selected state, the state in which at least one is selected is maintained even if another condition is changed to a selected state. You may. For example, when the display of the analysis result is in the selected state, the display control unit 250 increases the display of the analysis result of the low image quality image in response to an instruction from the examiner (for example, when the button 3420 is specified). The display may be changed to a display of the analysis result of the image quality image. In addition, when the display of the analysis result is in the selected state, the display control unit 250 displays the analysis result of the high-quality image in response to an instruction from the examiner (for example, when the designation of the button 3420 is released). May be changed to the display of the analysis result of the low-quality image.

In addition, when the display of the high-quality image is in the non-selection state, the display control unit 250 responds to an instruction from the examiner (for example, when the display of the analysis result is released), and The display of the analysis result may be changed to the display of a low-quality image. Further, when the display of the high-quality image is in the non-selection state, the display control unit 250 displays the low-quality image in response to an instruction from the examiner (for example, when the display of the analysis result is specified). The display of the analysis result of the low-quality image may be changed. In addition, when the display of the high-quality image is in the selected state, the display control unit 250 analyzes the high-quality image in response to an instruction from the examiner (for example, when the display of the analysis result is released). The display of the result may be changed to the display of a high-quality image. In addition, when the display of the high-quality image is in the selected state, the display control unit 250 increases the display of the high-quality image in response to an instruction from the examiner (for example, when the display of the analysis result is specified). The display may be changed to a display of the analysis result of the image quality image.

{Suppose that the display of the high-quality image is in the non-selected state and the display of the first type of analysis result is in the selected state. In this case, in response to an instruction from the examiner (for example, when the display of the second type of analysis result is specified), the display control unit 250 generates the first type of analysis result of the low image quality image. The display may be changed to the display of the second type of analysis result of the low image quality image. Also, consider the case where the display of a high-quality image is in a selected state and the display of the first type of analysis result is in a selected state. In this case, in response to an instruction from the examiner (for example, when the display of the second type of analysis result is specified), the display control unit 250 outputs the first type of analysis result of the high-quality image. The display may be changed to the display of the second type of analysis result of the high quality image.

Note that, as described above, the display screen for follow-up observation may be configured so that these display changes are collectively reflected on a plurality of images obtained at different dates and times. Here, the display of the analysis result may be a display in which the analysis result is superimposed on the image with an arbitrary transparency. At this time, the display of the analysis result may be changed, for example, to a state where the analysis result is superimposed on the displayed image with arbitrary transparency. The change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

(Modification 5)
In the various embodiments and modifications described above, the display control unit 250 displays the image selected according to the instruction from the examiner, out of the high-quality image generated by the image quality improvement unit 224 and the input image. Can be displayed. In addition, the display control unit 250 may switch the display on the display unit 270 from a captured image (input image) to a high-quality image according to an instruction from the examiner. That is, the display control unit 250 may change the display of the low-quality image to the display of the high-quality image according to an instruction from the examiner. In addition, the display control unit 250 may change the display of the high-quality image to the display of the low-quality image according to an instruction from the examiner.

Further, the image quality improvement unit 224 starts the image quality improvement processing (input of an image to the image quality improvement engine) by the image quality improvement engine (learned model for the image quality improvement) in response to an instruction from the examiner. Then, the display control unit 250 may cause the display unit 270 to display the high-quality image generated by the image quality improving unit 224. On the other hand, when an input image is captured by the imaging device (OCT imaging unit 100), the high-quality engine automatically generates a high-quality image based on the input image, and the display control unit 250 May be displayed on the display unit 270 in accordance with the instruction. Here, the image quality improvement engine includes a learned model that performs the above-described image quality improvement processing (image quality improvement processing).

処理 Note that these processes can be similarly performed on the output of the analysis result. That is, the display control unit 250 may change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image according to an instruction from the examiner. Further, the display control unit 250 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image according to an instruction from the examiner. Of course, the display control unit 250 may change the display of the analysis result of the low-quality image to the display of the low-quality image according to an instruction from the examiner. In addition, the display control unit 250 may change the display of the low-quality image to the display of the analysis result of the low-quality image according to an instruction from the examiner. In addition, the display control unit 250 may change the display of the analysis result of the high-quality image to the display of the high-quality image according to an instruction from the examiner. Further, the display control unit 250 may change the display of the high-quality image to the display of the analysis result of the high-quality image according to an instruction from the examiner.

The display control unit 250 may change the display of the analysis result of the low-quality image to the display of another type of analysis result of the low-quality image in accordance with an instruction from the examiner. In addition, the display control unit 250 may change the display of the analysis result of the high-quality image to the display of another type of analysis result of the high-quality image according to an instruction from the examiner.

Here, the analysis result of the high-quality image may be displayed by superimposing and displaying the analysis result of the high-quality image on the high-quality image with any transparency. The display of the analysis result of the low-quality image may be a display in which the analysis result of the low-quality image is superimposed and displayed on the low-quality image with arbitrary transparency. At this time, the display of the analysis result may be changed, for example, to a state where the analysis result is superimposed on the displayed image with arbitrary transparency. The change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

(Modification 6)
Analysis results such as a desired layer thickness and various blood vessel densities may be displayed on the report screens in the various embodiments and modifications described above. In addition, the optic papilla, macula, blood vessel region, nerve fiber bundle, vitreous region, macula region, choroid region, sclera region, cribriform region, retinal layer boundary, retinal layer boundary edge, photoreceptor cells, blood cells, The value (distribution) of a parameter relating to a site of interest including at least one of a blood vessel wall, a blood vessel inner wall boundary, a blood vessel outer boundary, a ganglion cell, a corneal region, a corner region, and Schlemm's canal may be displayed as an analysis result. At this time, for example, by analyzing a medical image to which various types of artifact reduction processing are applied, a highly accurate analysis result can be displayed. Note that the artifact is, for example, a false image region caused by light absorption by a blood vessel region or the like, a projection artifact, a band-like artifact in a front image generated in a main scanning direction of measurement light due to a state (movement, blink, etc.) of an eye to be inspected, or the like. There may be. In addition, the artifact may be, for example, any image-failure region that randomly appears on a medical image of a predetermined part of the subject for each imaging. Further, the values (distributions) of parameters relating to an area including at least one of the various artifacts (missing areas) as described above may be displayed as an analysis result. In addition, a parameter value (distribution) relating to a region including at least one of abnormal sites such as drusen, new blood vessels, vitiligo (hard vitiligo), and pseudo drusen may be displayed as an analysis result.

The analysis result may be displayed as an analysis map, a sector indicating a statistical value corresponding to each divided region, or the like. The analysis result may be generated using a learned model (analysis result generation engine, a learned model for generating the analysis result) obtained by learning the analysis result of the medical image as learning data. . At this time, the trained model is a learning model using learning data including a medical image and an analysis result of the medical image, learning data including a medical image and an analysis result of a medical image of a type different from the medical image, and the like. May be obtained. Further, the learned model is obtained by learning using learning data including input data in which a plurality of different types of medical images of a predetermined part are set, such as a luminance front image and a motion contrast front image. Is also good. Here, the luminance front image corresponds to the luminance En-Face image, and the motion contrast front image corresponds to the OCTA En-Face image. Further, a configuration may be adopted in which an analysis result obtained using a high-quality image generated by a learned model for improving image quality is displayed.

The input data included in the learning data may be a high-quality image generated by a learned model for improving image quality, or may be a set of a low-quality image and a high-quality image. Further, the learning data includes, for example, at least an analysis value (for example, an average value or a median value) obtained by analyzing the analysis area, a table including the analysis value, an analysis map, and the position of the analysis area such as a sector in the image. Information including one may be data obtained by labeling input data as correct answer data (for supervised learning). In addition, according to the instruction from the examiner, the analysis result obtained by the learned model for generating the analysis result may be displayed.

In addition, various diagnosis results such as glaucoma and age-related macular degeneration may be displayed on the report screens in the various embodiments and the modifications described above. At this time, for example, by analyzing the medical image to which the various artifact reduction processes described above are applied, a highly accurate diagnosis result can be displayed. In the diagnosis result, the position of the specified abnormal part or the like may be displayed on the image, or the state or the like of the abnormal part may be displayed by characters or the like. Further, a classification result (for example, a Curtin classification) of an abnormal part or the like may be displayed as a diagnosis result.

Note that the diagnosis result may be generated using a learned model (diagnosis result generation engine, a learned model for generating a diagnosis result) obtained by learning a diagnosis result of a medical image as learning data. . The learned model is obtained by learning using learning data including a medical image and a diagnosis result of the medical image, and learning data including a medical image and a diagnosis result of a medical image of a different type from the medical image. It may be obtained. Further, a configuration may be adopted in which a diagnosis result obtained using a high-quality image generated by a learned model for improving image quality is displayed.

The input data included in the learning data may be a high-quality image generated by a learned model for improving image quality, or may be a set of a low-quality image and a high-quality image. The learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal part), the position of the lesion in the image, the position of the lesion with respect to the attention area, the findings (interpretation findings, etc.), Information that includes at least one of the following: information that includes at least one of grounds for negating the diagnosis name (negative medical support information), and the like, as correct data (for supervised learning). You may. It should be noted that the diagnosis result obtained by the learned model for generating the diagnosis result may be displayed according to the instruction from the examiner.

In addition, on the report screens in the various embodiments and modified examples described above, the object recognition result (object detection result) and the segmentation result of the noted part, the artifact, the abnormal part, and the like as described above may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, a color or the like may be superimposed and displayed on an object in an image. Note that the object recognition result and the segmentation result may be generated using a learned model obtained by learning learning data obtained by labeling a medical image with information indicating object recognition and segmentation as correct data. . Note that the above-described generation of the analysis result and the generation of the diagnosis result may be obtained by using the above-described object recognition result and the segmentation result. For example, a process of generating an analysis result and a process of generating a diagnosis result may be performed on a region of interest obtained by the processing of object recognition and segmentation.

The above-described learned model may be a learned model obtained by learning using learning data including input data in which a plurality of different types of medical images of a predetermined part of the subject are set. At this time, as input data included in the learning data, for example, input data in which a front image of a motion contrast of a fundus and a luminance front image (or a luminance tomographic image) are set can be considered. Further, as input data included in the learning data, for example, input data in which a tomographic image of a fundus (B-scan image) and a color fundus image (or a fluorescent fundus image) are set may be considered. Further, the plurality of different types of medical images may be any medical images obtained by different modalities, different optical systems, different principles, or the like.

The learned model described above may be a learned model obtained by learning using learning data including input data in which a plurality of medical images of different parts of the subject are set. At this time, as input data included in the learning data, for example, input data in which a tomographic image of the fundus (B-scan image) and a tomographic image of the anterior segment (B-scan image) are set can be considered. Further, as input data included in the learning data, for example, input data in which a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic papilla of the fundus are set. Is also conceivable.

The input data included in the learning data may be different parts of the subject and a plurality of different types of medical images. At this time, the input data included in the learning data may be, for example, input data that sets a tomographic image of the anterior ocular segment and a color fundus image. Further, the above-described learned model may be a learned model obtained by learning using learning data including input data in which a plurality of medical images of a predetermined part of the subject with different imaging angles of view are set. The input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a predetermined region into a plurality of regions, such as a panoramic image. At this time, by using a wide-field-of-view image such as a panoramic image as learning data, it is possible to acquire the feature amount of the image with high accuracy because the amount of information is larger than that of the narrow-field-of-view image. The result of the processing can be improved. The input data included in the learning data may be input data in which a plurality of medical images at different dates and times of a predetermined part of the subject are set.

The display screen on which at least one of the analysis result, the diagnosis result, the object recognition result, and the segmentation result is displayed is not limited to the report screen. Such a display screen includes, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). May be displayed. For example, by displaying the at least one result obtained using the above-described learned model on a shooting confirmation screen, the examiner can check the result with high accuracy even immediately after shooting. Further, the change of the display of the low-quality image and the high-quality image described above may be, for example, the change of the display of the analysis result of the low-quality image and the analysis result of the high-quality image.

Here, the various learned models described above can be obtained by machine learning using learning data. The machine learning includes, for example, deep learning (Deep @ Learning) including a multi-layer neural network. For at least a part of the multi-layer neural network, for example, a convolutional neural network (CNN: Convolutional Neural Network) can be used. In addition, at least a part of the multi-layer neural network may use a technology related to an auto encoder (self-encoder). Further, a technology related to back propagation (error back propagation method) may be used for learning. However, the machine learning is not limited to the deep learning, but may be any model that can extract (represent) the feature amount of learning data such as an image by learning.

Further, the high-quality image engine (learned model for high-quality image) may be a learned model obtained by additionally learning learning data including at least one high-quality image generated by the high-quality image engine. Good. At this time, whether or not to use the high-quality image as learning data for additional learning may be configured to be selectable by an instruction from the examiner.

(Modification 7)
The preview screens in the various embodiments and modifications described above may be configured so that the learned model is used for at least one frame of the live moving image. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the learned model corresponding to each live moving image may be used. Thus, for example, even for a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of imaging. For this reason, for example, failure in re-imaging can be reduced, so that the accuracy and efficiency of diagnosis can be improved. Note that the plurality of live moving images may be, for example, a moving image of the anterior segment for alignment in the XYZ directions, and a front moving image of the fundus for focus adjustment or OCT focus adjustment of the fundus observation optical system. Further, the plurality of live moving images may be, for example, tomographic moving images of a fundus for coherence gate adjustment of OCT (adjustment of an optical path length difference between a measurement optical path length and a reference optical path length).

The moving image to which the learned model can be applied is not limited to a live moving image, and may be, for example, a moving image stored (saved) in a storage unit. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus stored (saved) in the storage unit may be displayed on the display screen. For example, when it is desired to appropriately observe the vitreous body, first, a reference frame may be selected based on the condition that the vitreous body exists on the frame as much as possible. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a configuration may be adopted in which high-quality images (high-quality frames) sequentially generated by the learned model for improving image quality are successively displayed for at least one frame of the moving image.

As the above-described method of positioning between frames, the same method may be applied to the method of positioning in the X direction and the method of positioning in the Z direction (depth direction), or all different methods may be used. May be applied. In addition, the alignment in the same direction may be performed a plurality of times by different methods. For example, after the rough alignment is performed, the precise alignment may be performed. Further, as a method of positioning, for example, a plurality of positions obtained by dividing a tomographic image (coarse in the Z direction) using a retinal layer boundary obtained by performing a segmentation process on a tomographic image (B scan image) and dividing the tomographic image are obtained. Positioning (precision in the X and Z directions) using correlation information (similarity) between the region and the reference image, and one-dimensional projection images generated for each tomographic image (B scan image) were used (in the X direction). ) Alignment (X-direction) alignment using a two-dimensional front image and the like. In addition, the configuration may be such that, after coarse positioning is performed in pixel units, precise positioning is performed in subpixel units.

Here, during the various adjustments, there is a possibility that the imaging target such as the retina of the eye to be inspected has not been imaged well yet. For this reason, since there is a large difference between the medical image input to the learned model and the medical image used as the learning data, a high-quality image may not be obtained with high accuracy. Therefore, when an evaluation value such as the image quality evaluation of a tomographic image (B scan) exceeds a threshold, display of a high-quality moving image (continuous display of high-quality frames) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the image quality improvement button may be changed to a state in which the examiner can specify (active state).

In addition, a different learned model for improving image quality is prepared for each shooting mode having a different scanning pattern or the like, and a learned model for improving image quality corresponding to the selected shooting mode is selected. Is also good. Further, one learned model for improving image quality obtained by learning learning data including various medical images obtained in different imaging modes may be used.

(Modification 8)
In the various embodiments and the modified examples described above, when the learned model is under additional learning, it may be difficult to output (inference / prediction) using the learned model itself under additional learning. For this reason, it is preferable to prohibit the input of a medical image to the learned model during the additional learning. Further, the same learned model as the learned model during the additional learning may be prepared as another spare learned model. At this time, during the additional learning, it is preferable that a medical image can be input to the spare learned model. Then, after the additional learning is completed, the learned model after the additional learning is evaluated, and if there is no problem, the spare learned model may be replaced with the learned model after the additional learning. If there is a problem, a spare learned model may be used.

学習 Also, a learned model obtained by learning for each imaging region may be selectively used. Specifically, a first learned model obtained using learning data including a first imaging region (lung, eye to be examined, and the like) and a learning including a second imaging region different from the first imaging region A plurality of learned models including the second learned model obtained using the data can be prepared. Then, the control unit 200 may include a selection unit that selects any one of the plurality of learned models. At this time, the control unit 200 may include a control unit that executes the selected learned model as additional learning. The control means searches for data in which an imaging part corresponding to the selected learned model and an imaging image of the imaging part are paired in accordance with an instruction from the examiner, and retrieves the obtained data as learning data. Can be executed as additional learning on the selected learned model. The imaging part corresponding to the selected learned model may be obtained from the information in the header of the data or manually input by the examiner. The data search may be performed via a network from a server or the like of an external facility such as a hospital or a laboratory. Thus, additional learning can be efficiently performed for each imaging region using the imaging image of the imaging region corresponding to the learned model.

Note that the selection unit and the control unit may be configured by a software module executed by a processor such as a CPU or an MPU of the control unit 200. Further, the selection unit and the control unit may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like.

学習 Also, when acquiring learning data for additional learning from a server or the like of an external facility such as a hospital or a research institute via a network, it is desirable to reduce a decrease in reliability due to falsification or a system trouble at the time of additional learning. Therefore, the validity of the learning data for additional learning may be detected by confirming the matching by digital signature or hashing. Thereby, the learning data for additional learning can be protected. At this time, if the validity of the learning data for additional learning cannot be detected as a result of checking the consistency by digital signature or hashing, a warning to that effect is issued, and additional learning using the learning data is performed. Absent. The server may be in any form, such as a cloud server, a fog server, an edge server, etc., regardless of the installation location.

(Modification 9)
In the various embodiments and modifications described above, the instruction from the examiner may be an instruction by voice or the like in addition to a manual instruction (for example, an instruction using a user interface or the like). At this time, for example, a machine learning model including a speech recognition model (speech recognition engine, learned model for speech recognition) obtained by machine learning may be used. The manual instruction may be an instruction by character input using a keyboard, a touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (a character recognition engine, a learned model for character recognition) obtained by machine learning may be used. The instruction from the examiner may be an instruction by a gesture or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, learned model for gesture recognition) obtained by machine learning may be used.

The instruction from the examiner may be a line of sight detection result of the examiner on the monitor. The gaze detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from the periphery of the monitor. At this time, the pupil detection from the moving image may use the above-described object recognition engine. Further, the instruction from the examiner may be an instruction based on an electroencephalogram, a weak electric signal flowing through the body, or the like.

In such a case, for example, as the learning data, character data or voice data (waveform data) indicating an instruction to display a result of processing of the various learned models as described above is used as input data, and various learned models are used. It may be learning data in which an execution instruction for actually displaying the result of the model processing on the display unit is correct data. Further, as the learning data, for example, character data or audio data indicating an instruction to display a high-quality image obtained by a learned model for improving image quality is used as input data, and an execution instruction and a button for displaying a high-quality image are input. Learning data in which an execution instruction for changing 3420 to the active state may be correct data. Of course, as the learning data, any data may be used as long as the instruction content indicated by the character data or the voice data and the execution instruction content correspond to each other. Moreover, you may convert audio | voice data into character data using an acoustic model or a language model. Further, a process of reducing noise data superimposed on audio data may be performed using waveform data obtained by a plurality of microphones. In addition, an instruction using characters or voice and an instruction using a mouse, a touch panel, or the like may be configured to be selectable according to an instruction from the examiner. In addition, on / off of an instruction by a character, a voice, or the like may be configured to be selectable according to an instruction from the examiner.

Here, machine learning includes deep learning as described above, and a recursive neural network (RNN) may be used as at least a part of the multi-layer neural network, for example. Here, as an example of a machine learning model according to the present modification, an RNN that is a neural network that handles time-series information will be described with reference to FIGS. 16A and 16B. Further, Long @ short-term @ memory (hereinafter, LSTM) which is a kind of RNN will be described with reference to FIGS. 17A and 17B.

FIG. 16A shows the structure of RNN which is a machine learning model. RNN3520 has a loop structure to the network, enter the data ^x t 3510 at time t, and outputs the data ^h t 3530. Since the RNN 3520 has a loop function in the network, the state at the current time can be taken over to the next state, so that the time series information can be handled. FIG. 16B shows an example of input / output of the parameter vector at time t. The data x ^t 3510 includes N data (Params 1 to Params N). Further, the data ^h t 3530 output from RNN3520 includes data of N (Params1 ~ ParamsN) corresponding to the input data.

However, since the RNN cannot handle long-term information at the time of error back propagation, LSTM may be used. The LSTM can learn long-term information by providing a forgetting gate, an input gate, and an output gate. Here, FIG. 17A shows the structure of the LSTM. In the LSTM3540, the information that the network takes over at the next time t is the internal state ct ^{-1 of the} network called a cell and the output data ht ^-1 . Note that lowercase letters (c, h, x) in the figure represent vectors.

Next, FIG. 17B shows details of the LSTM3540. In FIG. 17B, FG indicates a forgetting gate network, IG indicates an input gate network, and OG indicates an output gate network, each of which is a sigmoid layer. Therefore, a vector in which each element takes a value from 0 to 1 is output. The forgetting gate network FG determines how much past information is retained, and the input gate network IG determines which value to update. The CU is a cell update candidate network, and is an activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gate network OG selects the element of the cell candidate and selects how much information to transmit at the next time.

The LSTM model described above is a basic model, and is not limited to the network shown here. The coupling between the networks may be changed. Instead of the LSTM, a QRNN (Quasi \ Current \ Neural \ Network) may be used. Further, the machine learning model is not limited to the neural network, and boosting, a support vector machine, or the like may be used. When the instruction from the examiner is an input using characters or voice, a technology related to natural language processing (for example, Sequence to Sequence) may be applied. Further, a dialogue engine (a dialogue model, a trained model for a dialogue) that responds to the examiner with an output using characters or voices may be applied.

(Modification 10)
In the various embodiments and modifications described above, a high-quality image or the like may be stored in the storage unit according to an instruction from the examiner. At this time, after registering the file name after an instruction from the examiner to save a high-quality image or the like, any part of the file name (for example, the first part, the last part, Location), a file name including information (for example, characters) indicating that the image is an image generated by a process using a learned model for image quality improvement (image quality improvement process) is included in the instruction from the examiner. It may be displayed in an editable state accordingly.

In addition, when displaying a high-quality image on the display unit in various display screens such as a report screen, the displayed image is a high-quality image generated by a process using a learned model for improving the image quality. The display indicating the presence may be displayed together with the high-quality image. In this case, the user can easily identify from the display that the displayed high-quality image is not the image itself obtained by shooting, thereby reducing erroneous diagnosis or improving diagnosis efficiency. Can be. Note that the display indicating that the image is a high-quality image generated by the process using the learned model for improving the image quality is a display that can identify the input image and the high-quality image generated by the process. Any mode may be used. Further, not only the processing using the learned model for high image quality but also the processing using the various learned models described above are the results generated by the processing using the type of the learned model. A display indicating the presence may be displayed together with the result.

At this time, a display screen such as a report screen may be stored in the storage unit in accordance with an instruction from the examiner. For example, the report screen is stored as one image in which high-quality images and the like and a display indicating that these images are high-quality images generated by a process using a learned model for high image quality are arranged. It may be stored in.

In addition, the display indicating that the image is a high-quality image generated by processing using the learned model for high image quality is obtained by learning the learning model for high image quality with what learning data. A display indicating whether or not there is may be displayed on the display unit. The display may include an explanation of the types of the input data and the correct answer data of the learning data and an arbitrary display related to the correct answer data such as the imaging part included in the input data and the correct answer data. It should be noted that not only the processing using the learned model for improving the image quality, but also the processing using the various learned models as described above, the learning model of that type performs learning based on what learning data. A display indicating whether or not the information is displayed may be displayed on the display unit.

In addition, information (for example, characters) indicating that the image is generated by the process using the learned model for improving the image quality is displayed or stored in a state of being superimposed on the high-quality image or the like. You may. At this time, the portion to be superimposed on the image may be any region (for example, the end of the image) that does not overlap with the region where the target region to be photographed is displayed. Alternatively, a non-overlapping area may be determined and superimposed on the determined area.

If the button 3420 is set as an initial display screen of the report screen so that the button 3420 is set to the active state (the high-quality processing is turned on), a high-quality image or the like is displayed in accordance with an instruction from the examiner. A report image corresponding to the included report screen may be transmitted to the server. In addition, when the button 3420 is set to an active state by default, at the end of the examination (for example, when the photographing confirmation screen or the preview screen is changed to the report screen according to an instruction from the examiner). Alternatively, a report image corresponding to a report screen including a high-quality image or the like may be configured to be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, a depth range for generating an En-Face image on the initial display screen of the report screen, presence / absence of superimposition of the analysis map, whether or not the image is a high-quality image, and a display screen for follow-up observation) The report image generated based on at least one of the settings, such as whether or not the report image may be transmitted to the server.

(Modification 11)
In the above-described various embodiments and modified examples, among the various learned models as described above, an image (for example, a high-quality image, an analysis map such as an analysis map) obtained by the first type of learned model is shown. The image, the image indicating the object recognition result, and the image indicating the segmentation result) may be input to a second type of learned model different from the first type. At this time, a result (for example, an analysis result, a diagnosis result, an object recognition result, and a segmentation result) by the processing of the second type of learned model may be generated.

In addition, of the various types of learned models described above, the first type of the first type is obtained by using a result (for example, an analysis result, a diagnosis result, an object recognition result, and a segmentation result) obtained by processing the first type of the learned model. An image to be input to a second type of learned model different from the first type may be generated from the image input to the learned model. At this time, the generated image is likely to be an image suitable as an image to be processed by the second type of learned model. For this reason, an image obtained by inputting the generated image to the second type of learned model (for example, a high-quality image, an image showing an analysis result such as an analysis map, an image showing an object recognition result, and a segmentation result (The image shown) can be improved.

{Circle around (2)} Similar image search using an external database stored in a server or the like may be performed using, as a search key, an analysis result, a diagnosis result, or the like obtained by processing the learned model as described above. In the case where a plurality of images stored in the database are already managed in such a manner that feature amounts of the plurality of images are attached as additional information by machine learning or the like, the images themselves are used as search keys. A similar image search engine (similar image inspection model, trained model for similar image search) may be used.

(Modification 12)
Note that the generation processing of the motion contrast data in the above embodiment and the modification is not limited to the configuration performed based on the luminance value of the tomographic image. The above various processes are performed on tomographic data including an interference signal acquired by the OCT imaging unit 100, a signal obtained by performing a Fourier transform on the interference signal, a signal obtained by subjecting the signal to arbitrary processing, and a tomographic image based on these signals. May be applied. In these cases, the same effect as the above configuration can be obtained.

Although a fiber optical system using a coupler is used as the dividing means, a spatial optical system using a collimator and a beam splitter may be used. The configuration of the OCT imaging unit 100 is not limited to the above configuration, and a part of the configuration included in the OCT imaging unit 100 may be configured separately from the OCT imaging unit 100.

Also, in the above-described embodiment and modified examples, the configuration of the Mach-Zehnder interferometer is used as the interference optical system of the OCT imaging unit 100, but the configuration of the interference optical system is not limited to this. For example, the interference optical system of the OCT apparatus 1 may have a configuration of a Michelson interferometer.

Further, in the above-described embodiments and modified examples, a spectral domain OCT (SD-OCT) device using an SLD as a light source has been described as an OCT device, but the configuration of the OCT device according to the present invention is not limited to this. For example, the present invention can be applied to any other type of OCT device such as a wavelength-swept OCT (SS-OCT) device using a wavelength-swept light source capable of sweeping the wavelength of emitted light. Further, the present invention can also be applied to a Line-OCT apparatus using line light.

In addition, in the above-described embodiment and the modification, the acquisition unit 210 acquires the interference signal acquired by the OCT imaging unit 100, the three-dimensional tomographic image generated by the image processing unit 220, and the like. However, the configuration in which the acquisition unit 210 acquires these signals and images is not limited to this. For example, the acquisition unit 210 may acquire these signals from a server or an imaging device connected to the control unit via a LAN, a WAN, the Internet, or the like.

The learned model can be provided in the

control units

200, 900, and 1400, which are image processing devices. The learned model can be constituted by, for example, a software module executed by a processor such as a CPU. Further, the learned model may be provided in another server or the like connected to the

control units

200, 900, and 1400. In this case, the

control units

200, 900, and 1400 can perform image quality improvement processing using the learned model by connecting to a server having the learned model via an arbitrary network such as the Internet.

(Modification 13)
Further, the image processed by the image processing device or the image processing method according to the various embodiments and the modified examples described above includes a medical image acquired using an arbitrary modality (imaging device, imaging method). The medical image to be processed can include a medical image acquired by an arbitrary imaging device or the like, and an image created by the image processing apparatus or the image processing method according to the above-described embodiment and the modification.

{Furthermore, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. Further, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black and white image or a color image. Further, the medical image may be an image representing the structure (form) of the predetermined part or an image representing the function thereof. The images representing functions include, for example, images representing blood flow dynamics (blood flow, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasonic Doppler images. Note that the predetermined site of the subject may be determined according to the imaging target, and includes organs such as the human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, and liver, head, chest, Includes any parts such as legs and arms.

医 Also, the medical image may be a tomographic image of the subject or a front image. The front image is, for example, at least a part of the fundus front image, the front image of the anterior eye part, the fundus image obtained by fluorescence imaging, and data obtained by OCT (three-dimensional OCT data) in the depth direction of the imaging target. And an En-Face image generated using the data of FIG. The En-Face image is an OCTA En-Face image (motion contrast front image) generated using three-dimensional OCTA data (three-dimensional motion contrast data) using data in at least a part of the range in the depth direction of the imaging target. ). The three-dimensional OCT data and the three-dimensional motion contrast data are examples of three-dimensional medical image data.

撮影 The imaging device is a device for imaging an image used for diagnosis. The imaging apparatus detects, for example, a device that obtains an image of a predetermined portion by irradiating a predetermined portion of the subject with light, radiation such as X-rays, electromagnetic waves, or ultrasonic waves, or detects radiation emitted from a subject. And a device for obtaining an image of a predetermined part. More specifically, the imaging apparatuses according to the various embodiments and modifications described above include at least an X-ray imaging apparatus, a CT apparatus, an MRI apparatus, a PET apparatus, a SPECT apparatus, an SLO apparatus, an OCT apparatus, an OCTA apparatus, and a fundus. It includes a camera and an endoscope.

The OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. Further, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device and a wavelength sweep type OCT (SS-OCT) device. Further, the SLO device or OCT device may include a wavefront compensation SLO (AO-SLO) device using a wavefront compensation optical system, a wavefront compensation OCT (AO-OCT) device, or the like. Further, the SLO device and the OCT device may include a polarization SLO (PS-SLO) device, a polarization OCT (PS-OCT) device, and the like for visualizing information on a polarization phase difference and depolarization.

According to the above-described embodiment and one of the modified examples of the present invention, it is possible to generate an image more suitable for image diagnosis than in the related art.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

The present invention is not limited to the above embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to make the scope of the present invention public.

This application claims the priority based on Japanese Patent Application No. 2018-166817 filed on Sep. 6, 2018 and Japanese Patent Application No. 2019-068663 filed on Mar. 29, 2019. , The entire contents of which are incorporated herein.

200: control unit (image processing device), 224: image quality improvement unit, 250: display control unit

Claims

An image quality improvement unit configured to generate a second image in which at least one of noise reduction and contrast enhancement has been performed from the first image of the subject's eye using the learned model, as compared with the first image;
A display control unit that switches the first image and the second image on a display unit, and displays the images side by side or in an overlapping manner;
An image processing apparatus comprising:
The image processing apparatus according to claim 1, further comprising: a selection unit that selects a learned model used by the image quality improving unit from the plurality of learned models.
The first image is a front image generated based on information in a depth direction range of the subject's eye,
The image processing apparatus according to claim 2, wherein the selection unit selects a learned model used by the image quality improvement unit based on a range in a depth direction for generating the first image.
The said selection part selects the learned model used by the said image quality improvement part based on the display part in the said 1st image, and the range of the depth direction for producing | generating the said 1st image. An image processing apparatus according to claim 1.
5. The selection unit according to claim 4, wherein the selection unit selects a learned model used by the image quality improvement unit based on an imaging region including the display region and a range in a depth direction for generating the first image. The image processing apparatus according to any one of the preceding claims.
The image processing device according to claim 2, wherein the selection unit selects a learned model used by the image quality improvement unit based on a shooting condition of the first image.
The image processing device according to claim 2, wherein the selection unit selects a learned model used by the image quality improvement unit in response to an instruction from an operator.
The image processing device according to any one of claims 2 to 7, wherein the selection unit changes a learned model used by the image quality improvement unit in response to an instruction from an operator.
The display control unit according to any one of claims 1 to 8, wherein the display control unit switches between the first image and the second image according to an instruction from an operator and causes the display unit to display the first image and the second image. Image processing device.
An image quality improvement unit configured to generate a second image obtained by performing image quality improvement processing on the first image from the first image of the subject's eye using the learned model,
A display control unit configured to switch and display the first image and the second image on a display unit in response to an instruction from an operator;
An image processing apparatus comprising:
The image quality improving unit generates a plurality of the second images from a plurality of the first images,
The image processing device according to any one of claims 1 to 10, wherein the display control unit causes the display unit to switch and display the plurality of first images and the plurality of second images.
An acquisition unit that acquires the first image,
The display control unit,
Immediately after the acquisition of the first image by the acquisition unit, display the first image on the display unit,
The method according to claim 1, wherein after the second image is generated by the image quality improving unit, the displayed first image is switched to the second image and displayed on the display unit. An image processing apparatus according to the item.
The first image is a front image generated based on motion contrast data in a range in a depth direction of the subject's eye,
The display control unit,
Before the acquisition of the first image by the acquisition unit, a third image that is a front image generated based on tomographic data in the depth direction of the subject's eye is displayed,
The image processing apparatus according to claim 12, wherein immediately after the acquisition of the first image by the acquisition unit, the displayed third image is switched to the first image and displayed on the display unit.
The first image is a front image generated based on information in a depth direction range of the subject's eye,
When the range of the first image in the depth direction is changed in response to an instruction from the operator, the display control unit controls the first image and the second image displayed side by side on the display unit. The image according to any one of claims 1 to 8, wherein the image is displayed by being changed into a first image based on the changed range in the depth direction and a second image generated from the first image. The image processing apparatus according to any one of the preceding claims.
15. The display control unit, according to an instruction from an operator, enlarges any one of the first image and the second image displayed side by side on the display unit. The image processing device according to any one of the above.
The display control unit sets transparency to at least one of the first image and the second image, and causes the display unit to display the first image and the second image in a superimposed manner. An image processing apparatus according to any one of claims 1 to 8.
Using the trained model, at least one of noise reduction and contrast enhancement from the first image, which is a front image generated based on information in the depth direction range of the subject's eye, as compared to the first image. An image quality improvement unit that generates a second image that has been made,
A selecting unit that selects a learned model used by the image quality improving unit from a plurality of learned models based on a range in a depth direction for generating the first image;
An image processing apparatus comprising:
The image processing apparatus according to any one of claims 3 to 12, wherein the first image is one of a front image of the luminance of the eye to be inspected and a blood vessel image of the eye to be inspected.
A comparing unit configured to compare the first image and the second image and generate a color map image colored based on the comparison result;
The image processing device according to claim 1, wherein the color map image is superimposed on the first image or the second image.
An image quality improvement unit configured to generate a second image in which at least one of noise reduction and contrast enhancement has been performed from the first image of the subject's eye using the learned model, as compared with the first image;
A comparing unit that compares the first image and the second image;
A display control unit that controls display on a display unit based on a comparison result by the comparison unit;
An image processing apparatus comprising:
The comparison unit calculates a difference between the first image and the second image, generates a color map image that is color-coded based on the difference,
The image processing device according to claim 20, wherein the display control unit causes the display unit to display the color map image.
22. The image processing device according to claim 21, wherein the display control unit superimposes and displays the color map image on the first image or the second image.
23. The image processing device according to claim 21, wherein the display control unit superimposes and displays the color map image on the first image or the second image according to an instruction from an operator.
24. The image processing device according to claim 21, wherein the display control unit superimposes and displays the color map images corresponding to a plurality of the first images or a plurality of the second images.
The comparison unit calculates a difference between the first image and the second image,
The image processing device according to claim 20, wherein the display control unit causes the display unit to display a warning when the difference is larger than a predetermined value.
The comparison unit calculates a difference between the first image and the second image,
The image processing device according to claim 20, wherein the display control unit does not cause the display unit to display the second image when the difference is larger than a predetermined value.
27. The learning data of the learned model includes an image obtained by one of a superposition process, a maximum posterior probability estimation process, a smoothing filter process, and a gradation conversion process. The image processing device according to claim 1.
The learning data of the learned model is an image captured by an image capturing device having a higher performance than the image capturing device used for capturing the first image, or an image capturing process that requires more man-hours than the image capturing process of the first image. The image processing apparatus according to claim 1, wherein the image processing apparatus includes an image acquired by:
Generating, from the first image of the eye to be examined, a second image in which at least one of noise reduction and contrast enhancement has been performed as compared with the first image, using the learned model;
A step of switching between the first image and the second image on a display unit and displaying the image side by side or in an overlapping manner;
An image processing method comprising:
Generating a second image obtained by performing image quality improvement processing on the first image from the first image of the eye to be inspected, using the learned model;
A step of switching and displaying the first image and the second image on a display unit according to an instruction from an operator;
An image processing method comprising:
Using the trained model, at least one of noise reduction and contrast enhancement from the first image, which is a front image generated based on information in the depth direction range of the subject's eye, as compared to the first image. Generating a second image, one of which has been made;
Selecting, from a plurality of learned models, a learned model used for generating the second image based on a range in a depth direction for generating the first image;
An image processing method comprising:
Generating, from the first image of the eye to be examined, a second image in which at least one of noise reduction and contrast enhancement has been performed as compared with the first image, using the learned model;
Comparing the first image and the second image;
Controlling display on a display unit based on a comparison result between the first image and the second image;
An image processing method comprising:
A program which, when executed by a processor, causes the processor to execute each step of the image processing method according to any one of claims 29 to 32.