JP2022011912A - Image processing apparatus, image processing method and program - Google Patents

Abstract

To reduce excessive correction due to image processing of a medical image using a learned model.SOLUTION: An image processing apparatus comprises: an image quality improvement unit which acquires a second image of a medical image of a subject by performing image quality improvement on a first image of the medical image of the subject by using a learned model that is obtained by learning with the medical image of the subject as learning data; a detection unit which detects a target region in the medical image of the subject; and an image processing unit which performs image processing such that a difference between a pixel value of the target region and a pixel value of a region other than the target region becomes greater with respect to the target region in the second image and the pixel value of the target region becomes lower.SELECTED DRAWING: Figure 7

Description

The present invention relates to an image processing apparatus, an image processing method, and a program.

As an apparatus for obtaining a tomographic image of a subject, an apparatus (OCT apparatus) using an optical coherence tomography (OCT) is known. By using a medical tomographic imaging device such as an OCT device, it is possible to observe the state inside the retinal layer three-dimensionally, and such a medical tomographic imaging device can be used for ophthalmic retinal diseases such as AMD. It is useful for diagnosing. In recent years, OCTs used in clinical practice are roughly classified into two methods, SD-OCT (Spectral Domain OCT) and SS-OCT (Swept Source OCT), for example, as a method for acquiring images at high speed. In SD-OCT, a wideband light source is used, and an interferogram is acquired with a spectroscope. On the other hand, in SS-OCT, spectral interference is measured by a single-channel photodetector by using a high-speed wavelength sweep light source as a light source.

Recently, in both types of OCT, OCT angiography (OCTA), which images blood vessels without using a contrast medium, has attracted attention. OCTA generates motion contrast data from the OCT image acquired by OCT. Here, the motion contrast data is data obtained by repeatedly photographing the same cross section of the measurement target by OCT and detecting a temporal change of the measurement target between the images, for example, the phase, vector, and intensity of the complex OCT signal. The temporal change of is calculated from the difference, ratio, correlation, etc.

Further, the OCTA image is usually displayed as a two-dimensional OCTA front image by projecting the three-dimensional motion contrast data calculated from the acquired three-dimensional OCT image onto a two-dimensional plane. be. In this regard, Patent Document 1 discloses a technique for generating a two-dimensional front image by designating a range in the depth direction of the motion contrast data to be projected. Further, Patent Document 2 discloses a technique for generating a high-resolution image from a low-resolution image by an artificial intelligence engine obtained by machine learning using a low-resolution image and a high-resolution image.

Japanese Unexamined Patent Publication No. 2017-6179 JP-A-2018-5841

However, there may be room for improvement in various aspects such as analysis processing of data (OCT data) or motion contrast data obtained by using OCT. For example, in performing analysis processing of OCT data or motion contrast data, in particular, when correcting an image using an artificial intelligence engine, there is a possibility that overcorrection may occur.

Therefore, one of the objects of the present invention is to provide an image processing apparatus, an image processing method, and a program that reduce overcorrection by image processing of a medical image using a trained model.

The image processing apparatus according to at least one embodiment of the present invention uses a trained model obtained by training using the medical image of the subject as training data, and is higher than the first image of the medical image of the subject. For the image quality improving unit that performs image quality processing and acquires a second image of the medical image of the subject, the detection unit that detects the target area in the medical image of the subject, and the target area in the second image. The image processing unit is provided with an image processing unit that performs image processing so that the difference between the pixel value in the target area and the pixel value in the area other than the target area is widened and the pixel value in the target area is lower.

According to at least one embodiment of the present invention, overcorrection due to image processing of a medical image using a trained model can be reduced.

It is a block diagram which shows the schematic structure of the OCT apparatus which concerns on 1st Embodiment. It is a figure explaining the schematic structure of the photographing apparatus which concerns on 1st Embodiment. It is a block diagram which shows the schematic structure of the image quality improvement part which concerns on 1st Embodiment. An example of the configuration of the neural network related to the high image quality engine is shown. An example of learning data related to high image quality processing is shown. An example of an input image related to high image quality processing is shown. It is a flow chart which shows the schematic flow of the image processing which concerns on 1st Embodiment. It is a flow chart which shows an example of the flow of image processing which concerns on 1st Embodiment. It is a flow chart which shows an example of the flow of image processing which concerns on 1st Embodiment. It is a flow chart which shows an example of the flow of image processing which concerns on 1st Embodiment. It is a flow chart which shows an example of the flow of image processing which concerns on 1st Embodiment. It is a flow chart which shows an example of the flow of image processing which concerns on 1st Embodiment. It is a flow chart which shows an example of the flow of image processing which concerns on 1st Embodiment. It is a flow chart which shows the schematic flow of the image processing which concerns on 2nd Embodiment. An example of the machine learning model according to the modification 8 is shown. An example of the machine learning model according to the modification 8 is shown.

Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of the components, etc. described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar. Further, in the following, the axial direction is described as Z, the horizontal direction of the fundus plane is described as X, and the vertical direction of the fundus plane is described as Y.

In the following, the machine learning model refers to a learning model based on a machine learning algorithm. Specific algorithms for machine learning include the nearest neighbor method, naive Bayesian method, decision tree, and support vector machine. In addition, deep learning (deep learning) in which features for learning and coupling weighting coefficients are generated by themselves using a neural network can also be mentioned. As appropriate, any of the above algorithms that can be used can be applied to the following embodiments and modifications. The teacher data refers to learning data and is composed of a pair of input data and output data. The correct answer data means the output data of the learning data (teacher data).

The trained model is a model in which training (learning) is performed in advance using appropriate teacher data (learning data) for a machine learning model that follows an arbitrary machine learning algorithm such as deep learning. .. However, although the trained model is obtained by using appropriate training data in advance, it is not that no further training is performed, and additional training can be performed. Additional learning can be done even after the device has been installed at the site of use.

(First Embodiment)
Hereinafter, the image processing system according to the first embodiment of the present invention will be described in detail with reference to FIGS. 1 to 13. In the present embodiment, as an example of the image processing system, an OCT device including an image processing device that processes a tomographic image of a subject acquired by OCT will be described.

(Configuration of image processing device)
The configuration of the image processing device 101 and the connection with other devices according to the present embodiment will be described with reference to FIG. The image processing device 101 is connected to a photographing device 100, an external storage device 102, an output unit 103, and an input unit 104. It should be noted that these connections may be wired connections or wireless connections. Further, these connections may be connections via a network. For example, the image processing device 101 may be connected to the photographing device 100 via a network such as the Internet. Further, for example, the external storage device 102 may be placed on a network such as the Internet so that the data can be shared by a plurality of image processing devices.

The image processing device 101 is provided with functional blocks such as an acquisition unit 101-1, an imaging control unit 101-2, an image processing unit 101-4, a display control unit 101-5, and a storage unit 101-3. There is. The image processing device 101 can be configured by using a general computer including a processor, a memory, and the like, but may be configured as a dedicated computer of the OCT device. Here, the image processing device 101 may be a built-in (internal) computer of the OCT device, or may be a separate (external) computer to which the OCT device is communicably connected. Further, the image processing device 101 may be, for example, a personal computer, or a desktop PC, a notebook PC, or a tablet PC (portable information terminal) may be used. The processor may be a CPU (Central Processing Unit). Further, the processor may be, for example, an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), an FPGA (Field-Programmable Gate Array), or the like.

Each function of the image processing device 101 may be realized by executing a software module stored in the storage unit 101-3 by a processor such as a CPU or MPU. The processor may be, for example, GPU, FPGA, or the like. Further, each function may be configured by a circuit or the like that performs a specific function such as an ASIC. For example, the image processing unit 101-4 may be realized by using dedicated hardware such as an ASIC, or the display control unit 101-5 may be realized by using a dedicated processor such as a GPU different from the CPU. The storage unit 101-3 may be composed of, for example, an optical disk such as a hard disk or an arbitrary storage medium such as a memory.

The acquisition unit 101-1 captures signal data such as an SLO image, a tomographic image, and an anterior eye portion image obtained by photographing the subject with the imaging device 100, an image such as an SLO image, a tomographic image, and an anterior eye observation image, and patient information. It is a functional block to acquire etc. Further, the acquisition unit 101-1 can generate an image such as an SLO image, a tomographic image, and an anterior eye observation image by using the acquired signal data. The acquisition unit 101-1 is provided with a tomographic image generation unit 101-11 and a motion contrast data generation unit 101-12. The acquisition unit 101-1 may acquire various data including images from an external device (not shown) such as a server connected to the image processing device 101. Here, the image processing device 101 and an external device (not shown) may be connected via an arbitrary network such as the Internet. Further, the acquisition unit 101-1 may acquire various data stored in the storage unit 101-3.

The tomographic image generation unit 101-11 performs signal processing on the signal data (interference signal) of the acquired tomographic image to generate a tomographic image, and stores the generated tomographic image in the storage unit 101-3. As a method for generating a tomographic image from an interference signal, any known method may be used.

The motion contrast data generation unit 101-12 generates motion contrast data based on a plurality of tomographic images generated by the tomographic image generation unit 101-11 at substantially the same position (regions corresponding to each other in the subject). Hereinafter, a method of generating motion contrast data will be described.

First, the tomographic image generation unit 101-11 generates a plurality of tomographic images from a plurality of interference signals acquired by photographing substantially the same position of the subject a plurality of times. Here, the scanning group of the measurement light when the substantially same position of the subject is photographed a plurality of times in order to generate the motion contrast data is referred to as one cluster. Further, a plurality of tomographic images corresponding to a plurality of interference signals acquired by photographing the same position of the subject a plurality of times are referred to as a tomographic image for one cluster (one group). More specifically, the tomographic image generation unit 101-11 performs wave number conversion, fast Fourier transform (FFT: Fast Fourier Transform), and absolute value conversion (FFT: Fast Fourier Transform) and absolute value conversion (FFT: Fast Fourier Transform) for a plurality of interference signals acquired by the acquisition unit 101-1. By performing (acquisition of amplitude), a tomographic image for one cluster is generated. The method for generating a tomographic image is not limited to this, and any known method for performing image processing may be used.

Next, the alignment unit 101-41 of the image processing unit 101-4, which will be described later, aligns the tomographic images belonging to the same cluster and performs superposition processing. After that, the image feature acquisition unit 101-44 of the image processing unit 101-4, which will be described later, acquires layer boundary data from the superposition tomographic image. In the present embodiment, the variable shape model is used as the layer boundary acquisition method, but any known layer boundary acquisition method may be used. Note that the layer boundary acquisition process is not essential. For example, if the motion contrast image is generated only in three dimensions and the two-dimensional motion contrast image projected in the depth direction is not generated, the layer boundary acquisition process can be omitted. ..

The motion contrast data generation unit 101-12 calculates motion contrast data between adjacent tomographic images in the same cluster. In the present embodiment, the motion contrast data generation unit 101-12 obtains the decorrelation value Mxz as the motion contrast data based on the following equation (1).

Here, Axz indicates the amplitude (of the complex number data after FFT processing) at the position (x, z) of the tomographic image data A, and Bxz indicates the amplitude of the tomographic data B at the same position (x, z). 0 ≦ Mxz ≦ 1, and the larger the difference between the two amplitude values, the closer to 1. The motion contrast data generation unit 101-12 performs the decorrelation calculation process as in the equation (1) between arbitrary temporally adjacent tomographic images (belonging to the same cluster). It should be noted that the tomographic images to be subjected to the decorrelation calculation process do not necessarily have to be adjacent in time as long as they are images within a predetermined period. The motion contrast data generation unit 101-12 generates an image having the average of the obtained motion contrast data values (number of tomographic images per cluster-1) as the pixel value as the final motion contrast image.

Here, the motion contrast data is calculated based on the amplitude of the complex number data after the FFT processing, but the calculation method of the motion contrast data is not limited to the above. For example, the motion contrast data may be calculated based on the phase information of the complex number data, or the motion contrast data may be calculated based on both the amplitude and phase information. Alternatively, the motion contrast data may be calculated based on the real part or the imaginary part of the complex number data.

Further, in the present embodiment, the decorrelation value is calculated as the motion contrast data, but the calculation method of the motion contrast data is not limited to this, and any known calculation method may be used. For example, the motion contrast data may be calculated based on the difference between the two values, or the motion contrast data may be calculated based on the ratio of the two values. Further, the motion contrast data can be obtained, for example, as a dispersion value between two tomographic images or corresponding interference signals, or a value obtained by dividing the maximum value by the minimum value (maximum value / minimum value). Any known method may be used for these calculation methods.

Further, when controlling the scanning means so that the measurement light is scanned multiple times at substantially the same position, the time interval (time interval) between one scan (one B scan) and the next scan (next B scan). ) May be modified (determined). Thereby, for example, even if the blood flow velocity may differ depending on the state of the blood vessel, the blood vessel region can be visualized with high accuracy.

At this time, for example, the time interval may be configured to be changeable according to an instruction from the operator (inspector). Further, for example, any motion contrast image may be configured to be selectable from a plurality of motion contrast images corresponding to a plurality of preset time intervals in response to an instruction from the operator. Further, for example, the time interval when the motion contrast data is acquired and the motion contrast data may be associated with each other and stored in the storage unit 101-3. Further, for example, the display control unit 101-5 may display the time interval when the motion contrast data is acquired and the motion contrast image corresponding to the motion contrast data on the output unit 103. Further, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. At this time, for example, using a machine learning model, the time interval may be determined (output) from the motion contrast image. In such a machine learning model, for example, a plurality of motion contrast images corresponding to a plurality of time intervals are used as input data, and the difference from the plurality of time intervals to the time interval when a desired motion contrast image is acquired is correctly answered. Learning as data It can be obtained by learning the data.

Further, here, the pixel value of the final motion contrast image is obtained by obtaining the average value of the acquired plurality of decorrelation values, but the final pixel value is not limited to this. For example, an image having a median value or a maximum value of a plurality of acquired decorrelation values as a pixel value may be generated as a final motion contrast image.

The photographing control unit 101-2 is a functional block that controls photography with respect to the photographing apparatus 100. For example, the photographing control unit 101-2 can perform drive control of a light source, a scanning unit, a driving device for a focusing lens, and the like included in the photographing apparatus 100. The imaging control unit 101-2 can control the alignment operation of the stage unit 100-2 of the imaging device 100, which will be described later. Further, the imaging control unit 101-2 can also instruct the imaging device 100 regarding the setting of imaging parameters and the start or end of imaging.

The storage unit 101-3 can store a program for realizing various application software including an operating system (OS), a device driver of a peripheral device, and a program for performing processing described later. Further, the storage unit 101-3 can also store the information acquired by the acquisition unit 101-1 and various images processed by the image processing unit 101-4. For example, the storage unit 101-3 stores a medical image such as a tomographic image acquired by the acquisition unit 101-1, or stores the image quality improved by the image quality improvement unit 101-47 described later. be able to.

The image processing unit 101-4 is a functional block that performs image processing on various images such as a tomographic image, a motion contrast image, an SLO image, and an anterior eye observation image. The image processing unit 101-4 includes an alignment unit 101-41, a composition unit 101-42, a correction unit 101-43, an image feature acquisition unit 101-44, a projection unit 101-45, an analysis unit 101-46, and a height unit. The image quality improving unit 101-47 is provided.

The alignment unit 101-41 is a functional block that performs alignment processing between images. For example, the alignment unit 101-41 can align tomographic images belonging to the same cluster and perform superposition processing. The alignment process and the superposition process may be performed by any known method.

The compositing unit 101-42 is a functional block that synthesizes one image from each of a plurality of two-dimensional images. The compositing unit 101-42 is provided with, for example, a compositing method designation unit 101-421, an same modality image compositing unit 101-422, and a heterogeneous modality image compositing unit 101-423.

The composition method designation unit 101-421 specifies the type of the image to be combined (for example, tomographic image / motion contrast image / tomographic image and motion contrast image) and the composition processing method (for example, superposition / pasting / juxtaposition display). do. The same modality image compositing unit 101-422 performs compositing processing between tomographic images or motion contrast images, for example. The heterogeneous modality image compositing unit 101-423 performs compositing processing between images obtained by different types of modality, such as between a tomographic image and a motion contrast image.

The correction unit 101-43 is a functional block that performs a process of suppressing projection artifacts generated in the motion contrast image. Here, the projection artifact is a phenomenon in which the motion contrast in the blood vessels on the surface of the retina is reflected on the deep side (deep layer of the retina, outer layer of the retina, choroid), and a high decorrelation value occurs in the deep region where the blood vessels do not actually exist. Point to. For example, the correction unit 101-43 performs a process of reducing projection artifacts in the generated motion contrast data. Therefore, the correction unit 101-43 corresponds to an example of a processing means that performs a process of reducing projection artifacts on the generated motion contrast data.

The image feature acquisition unit 101-44 acquires layer boundary data from a tomographic image, a motion contrast tomographic image, and the like. Specifically, the image feature acquisition unit 101-44 performs image segmentation processing on a tomographic image or the like, extracts the layer structure in the tomographic area of the eye to be inspected, and acquires layer boundary data such as the boundary position. As the image segmentation processing method, any known method may be used.

The projection unit 101-45 is a functional block that projects or integrates a tomographic image or a motion contrast image within a set depth range to generate a luminance front image (luminance En-Face image) or an OCTA front image. The depth range can be set by using two reference planes based on the instruction of the operator and the layer boundary data acquired by the image feature acquisition units 101-44. The depth range to be set may be any depth range. For example, the depth range of the surface layer of the retina and the outer layer of the retina can be set, and two types of synthetic OCTA front images can be generated by the synthesis unit 101-42.

Here, as a method of projecting the data corresponding to the depth range set based on the two reference planes on the two-dimensional plane, for example, the representative value of the data in the depth range is set as the pixel value on the two-dimensional plane. Can be used. The representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Specifically, as the projection method, for example, maximum value projection (MIP) or average value projection (AIP: Average Integrity Projection) can be selected.

Further, the depth range related to the front image may be, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries relating to the detected retinal layer. .. Further, the depth range related to the En-Face image may be, for example, a range changed (offset) according to the instruction of the operator from the range between the two layer boundaries regarding the detected retinal layer. good. Here, the depth range for generating the front image can be changed by the operator selecting from a default depth range set displayed in a selection list (not shown) or the like. In addition, the operator can change the type and offset position of the layer boundary used to specify the projection range from the user interface (UI), or operate the input unit 104 to move the layer boundary data superimposed on the tomographic image. You can also change the projection range by doing so.

The generated luminance front image and OCTA front image can be output by the output unit 103. The motion contrast image output to the output unit 103 is not limited to the OCTA front image, and may be a three-dimensional motion contrast image rendered three-dimensionally or a motion contrast tomographic image corresponding to the tomographic image. good. When the generated motion contrast image or the like is displayed by the output unit 103, the presence or absence of the above-mentioned projection method or projection artifact suppression process may be changed by selecting from a UI such as a context menu. For example, the motion contrast image after the projection artifact suppression process may be displayed as a three-dimensional image.

Further, the analysis unit 101-46 is a functional block that analyzes various images such as a tomographic image and a motion contrast image. The analysis unit 101-46 is provided with an emphasis unit 101-461, an extraction unit 101-462, a measurement unit 101-463, and a comparison unit 101-464.

The enhancement unit 101-461 can perform enhancement processing for emphasizing data in an arbitrary region in an image, for example, in response to an instruction by an operator. The extraction unit 101-462 can extract feature units, regions, and the like in the image used by the analysis unit 101-46. For example, the extraction unit 101-462 can acquire the layer boundaries of the retina and choroid, the boundaries of the anterior and posterior surfaces of the lamina cribrosa, the positions of the fovea centralis and the center of the optic nerve head, and the like from the tomographic image. In addition, the extraction unit 101-462 can extract the blood vessel region from the OCTA front image. The measuring unit 101-463 calculates, for example, a measured value to be analyzed from various images. For example, the measuring unit 101-463 calculates the thickness of a specific layer, or uses the blood vessel centerline data obtained by thinning the extracted blood vessel region or the blood vessel region to measure the blood vessel density or the like. Can be calculated. Further, the comparison unit 101-464 can compare a plurality of images such as a plurality of tomographic images and a plurality of motion contrast images. Further, the comparison unit 101-464 can also compare the analysis results of a plurality of images such as the measured values calculated by the measurement unit 101-463.

The image quality improving unit 101-47 is a functional block for improving the image quality of various images. Therefore, the high image quality section 101-47 is an example of a high image quality section for improving the image quality of a medical image such as a tomographic image or an OCTA front image. The details of the configuration and functions of the image quality improving unit 101-47 will be described later.

The display control unit 101-5 can control the display of the output unit 103 connected to the image processing device 101. The display control unit 101-5 can display, for example, various data such as a tomographic image of the eye to be inspected and patient information on the output unit 103.

The external storage device 102 can store a program for tomography, patient information, image information, and the like. For example, the external storage device 102 may include patient information (patient name, age, gender, etc.), eye information (left eye, right eye, axial length, etc.) and captured images (tomographic image, SLO image, etc.). And OCTA images, etc.), composite images, shooting parameters, image data and measurement data of past inspections, parameters set by the operator, etc. can be stored in association with each other.

The input unit 104 includes, for example, a mouse, a keyboard, a touch panel, and the like, and the operator can give an instruction to the image processing device 101 and the photographing device 100 via the input unit 104. The output unit 103 can output data such as various images processed by the image processing device 101. The output unit 103 can be configured by an arbitrary display or the like, and can display, for example, patient information or various images related to the subject based on the control by the display control unit 101-5. In this case, the output unit 103 can function as an example of a display unit that displays various images and information. The output unit 103 may be provided with a touch UI or the like.

A part of the configuration of the OCT device may be configured as a separate device or may be configured as an integrated device. For example, the output unit 103 may be integrally configured with the input unit 104 as a touch panel type display.

(Configuration of shooting device)
The photographing apparatus 100 is an apparatus for photographing the eye to be inspected in order to obtain a tomographic image, an SLO image, or the like of the eye to be inspected. Hereinafter, the configurations of the measurement optical system and the spectroscope in the photographing apparatus 100 according to the present embodiment will be described with reference to FIGS. 1 and 2. In the present embodiment, an imaging device including an SD-OCT (Spectral Domain OCT) optical system is used as the imaging device 100. Not limited to this, an imaging device including an optical system such as SS-OCT or TD-OCT (Time Domain OCT) may be used.

As shown in FIG. 1, the photographing apparatus 100 is provided with a measurement optical system 100-1, a stage unit 100-2, and a base unit 100-3. The measurement optical system 100-1 is an optical system for acquiring an anterior eye portion image, an SLO fundus image of the eye to be inspected, and a tomographic image. The stage unit 100-2 holds the measurement optical system 100-1 so as to be movable back and forth and left and right, and the measurement optical system 100-1 can be moved according to the control of the imaging control unit 101-2. The base portion 100-3 has a built-in spectroscope 230, which will be described later.

The configuration of the measurement optical system 100-1 will be described with reference to FIG. 2. In the measurement optical system 100-1, the objective lens 201 is installed facing the eye 200 to be inspected, and the first dichroic mirror 202 and the second dichroic mirror 203 are arranged on the optical axis thereof. The optical path from the objective lens 201 is branched by these dichroic mirrors into an optical path 250 for the OCT optical system, an optical path 251 for the SLO optical system and the fixation lamp, and an optical path 252 for front eye observation for each wavelength band. ..

In the present embodiment, the optical path 252 for front eye observation is arranged in the reflection direction of the first dichroic mirror 202, and the optical path 250 for the OCT optical system, the SLO optical system, and the fixation lamp are arranged in the transmission direction of the first dichroic mirror 202. Optical path 251 is arranged. Further, the optical path 250 for the OCT optical system is arranged in the reflection direction of the second dichroic mirror 203, and the SLO optical system and the optical path 251 for the fixation lamp are arranged in the transmission direction of the second dichroic mirror 203. However, the arrangement of each optical path with respect to the dichroic mirror may be reversed.

The SLO optical system and the optical path 251 for the fixed-view lamp include an SLO scanning unit 204, lenses 205, 206, mirror 207, a third dichroic mirror 208, an APD (Avalanche Photodiode) 209, an SLO light source 210, and a fixed-view lamp 211. It is provided. The mirror 207 is configured by using a prism on which a perforated mirror or a hollow mirror is vapor-deposited, and separates the illumination light by the SLO light source 210 and the return light of the illumination light from the eye 200 to be inspected. The third dichroic mirror 208 branches the optical path from the third dichroic mirror 208 into the optical path of the SLO light source 210 and the optical path of the fixation lamp 211 for each wavelength band. In the present embodiment, the optical path of the SLO light source 210 is arranged in the reflection direction of the third dichroic mirror 208, and the fixation lamp 211 is arranged in the transmission direction of the third dichroic mirror 208. However, the arrangement of each optical path with respect to the dichroic mirror may be reversed.

The SLO scanning unit 204 scans the illumination light emitted from the SLO light source 210 on the eye 200 to be inspected, and includes an X scanner that scans in the X direction and a Y scanner that scans in the Y direction. The SLO scanning unit 204 is controlled by the photographing control unit 101-2. In the present embodiment, the X scanner is a polygon mirror that performs high-speed scanning, and the Y scanner is a galvano mirror. However, the configuration of the SLO scanning unit 204 is not limited to this, and the X scanner and the Y scanner may be configured by using an arbitrary deflection mirror according to a desired configuration.

The lens 205 is driven in the optical axis direction by a motor (not shown) for focusing the SLO optical system and the fixation lamp 211. The motor for driving the lens 205 is controlled by the photographing control unit 101-2.

The SLO light source 210 generates light having a wavelength near 780 nm. APD209 detects the return light from the eye 200 to be inspected. The fixation lamp 211 generates visible light to promote fixation of the subject.

The illumination light emitted from the SLO light source 210 is reflected by the third dichroic mirror 208, passes through the mirror 207, passes through the lenses 206 and 205, reaches the SLO scanning unit 204, and is reached by the SLO scanning unit 204 on the eye 200 to be inspected. It is scanned. The return light from the eye 200 to be inspected returns to the same path as the illumination light, is reflected by the mirror 207, and is guided to the APD 209.

The APD209 generates signal data of the SLO fundus image based on the return light and outputs it to the image processing apparatus 101. The acquisition unit 101-1 of the image processing device 101 generates an SLO image based on the signal data output from the APD 209. The SLO image of the fundus of the eye 200 to be inspected can be obtained by scanning the illumination light emitted from the SLO light source 210 on the fundus of the eye 200 to be inspected. On the other hand, it is also possible to acquire an SLO image of the anterior segment of the eye 200 to be inspected by scanning the anterior segment of the eye to be inspected with the illumination light emitted from the SLO light source 210.

The light emitted from the fixation lamp 211 passes through the third dichroic mirror 208 and the mirror 207, passes through the lenses 206 and 205, and forms a predetermined shape at an arbitrary position on the eye 200 to be inspected by the SLO scanning unit 204. By having the subject gaze at the light emitted from the fixation lamp 211, it is possible to promote the fixation of the subject.

In the optical path 252 for frontal eye observation, lenses 212, 213, split prism 214, and CCD 215 for frontal eye observation for detecting infrared light are arranged. The CCD215 has a sensitivity in the wavelength of the irradiation light for observing the anterior eye portion (not shown), specifically in the vicinity of 970 nm. The data signal output from the CCD 215 is output to the image processing device 101, and the image processing device 101 can generate an anterior segment observation image based on the input signal data.

The split prism 214 is arranged at a position conjugate with the pupil of the eye 200 to be inspected. The image processing apparatus 101 can detect the distance in the Z-axis direction (optical axis direction) of the measurement optical system 100-1 with respect to the eye to be inspected 200 by using the split image of the anterior eye portion based on the light passing through the split prism 214.

The optical path 250 for the OCT optical system is provided with an OCT optical system, and the OCT optical system is used to capture a tomographic image of the eye 200 to be inspected. More specifically, the OCT optical system is used to obtain an interference signal for producing a tomographic image. The optical path 250 for the OCT optical system is provided with an XY scanner 216, lenses 217, 218, and an optical fiber 224.

The XY scanner (OCT scanning unit) 216 is for scanning the measurement light on the eye 200 to be inspected. The XY scanner 216 is controlled by the photographing control unit 101-2. Although the XY scanner 216 is shown as a single mirror in FIG. 2, it is actually a galvano mirror that scans in the XY2 axis direction. However, the configuration of the XY scanner 216 is not limited to this, and may be configured by using an arbitrary deflection mirror according to a desired configuration. For example, the XY scanner 216 may be configured by using any deflection means such as a MEMS mirror capable of scanning the measurement light in a two-dimensional direction with one sheet.

The lens 217 is driven in the optical axis direction by a motor (not shown) in order to focus the measurement light emitted from the optical fiber 224 connected to the optical coupler 219 to the eye 200 to be inspected. By this focusing, the return light of the measurement light from the eye 200 to be inspected is simultaneously imaged and incident on the tip of the optical fiber 224 in a spot shape. The motor for driving the lens 217 is controlled by the photographing control unit 101-2.

Next, the configuration of the optical path from the OCT light source 220, the reference optical system, and the spectroscope will be described. The OCT optical system is further provided with an OCT light source 220, a reference mirror 221, a dispersion compensating glass 222, a lens 223, an optical coupler 219, a single-mode optical fiber 224 to 227 connected to the optical coupler and integrated, and a spectroscope 230. Has been done. The OCT optical system constitutes a Michelson interferometer by these configurations.

The OCT light source 220 is an SLD (Super Luminescent Diode), which is a typical low coherent light source. The center wavelength is 855 nm and the wavelength bandwidth is about 100 nm. Here, the bandwidth is an important parameter because it affects the resolution of the obtained tomographic image in the optical axis direction. Although SLD was selected as the type of light source here, ASE (Amplified Spontaneous Emission) or the like can be used as long as low coherent light can be emitted. Near-infrared light is suitable for the center wavelength in view of measuring the eye. Further, since the central wavelength affects the lateral resolution of the obtained tomographic image, it can be as short as possible. In this embodiment, the center wavelength of the OCT light source 220 is set to 855 nm for both reasons.

The light emitted from the OCT light source 220 is divided into measurement light incident on the optical fiber 224 side via the optical coupler 219 and reference light incident on the optical fiber 226 side through the optical fiber 225. The measurement light is applied to the eye 200 to be observed through the optical path 250 for the OCT optical system, and reaches the optical coupler 219 as return light through the same optical path due to reflection or scattering by the eye 200 to be observed. On the other hand, the reference light reaches the reference mirror 221 and is reflected through the optical fiber 226, the lens 223, and the dispersion compensating glass 222 inserted to match the wavelength dispersion of the measurement light and the reference light. The reference light reflected by the reference mirror 221 returns in the same optical path and reaches the optical coupler 219.

The measurement light and the reference light are combined by the optical coupler 219 to become interference light. Here, the measurement light and the reference light cause interference when the optical path length of the measurement light and the optical path length of the reference light are substantially the same. The reference mirror 221 is held so as to be adjustable in the optical axis direction by a motor and a drive mechanism (not shown) controlled by the photographing control unit 101-2, and it is possible to match the optical path length of the reference light with the optical path length of the measurement light. be. The interfering light is guided to the spectroscope 230 via the optical fiber 227.

Further, polarization adjusting units 228 and 229 are provided in the optical fibers 224 and 226. The polarization adjusting units 228 and 229 adjust the polarization of the measurement light and the reference light, respectively. The polarization adjusting units 228 and 229 have some portions in which the optical fiber is wound in a loop shape. In the polarization adjusting units 228 and 229, the loop-shaped portion is rotated around the longitudinal direction of the optical fiber to twist the optical fiber, and the polarization states of the measurement light and the reference light can be adjusted and matched respectively. ..

The spectroscope 230 is provided with a lens 232, 234, a diffraction grating 233, and a line sensor 231. The interference light emitted from the optical fiber 227 becomes parallel light through the lens 234, is separated by the diffraction grating 233, and is imaged on the line sensor 231 by the lens 232.

The interference light is measured by the line sensor 231 as intensity information for each wavelength. The intensity information for each wavelength measured by the line sensor 231 is output to the image processing device 101 as signal data (interference signal) of a tomographic image. The image processing device 101 can generate a tomographic image of the eye 200 to be inspected using the received signal data.

In this embodiment, the Michelson interferometer is used as the interferometer, but a Mach-Zehnder interferometer may be used. For example, a Mach-Zehnder interferometer can be used when the light amount difference is large, and a Michelson interferometer can be used when the light amount difference is relatively small, depending on the light amount difference between the measured light and the reference light.

(High image quality processing)
Hereinafter, the high image quality processing for the motion contrast data acquired by OCT will be specifically described, but the terms used in the description of the embodiment will be briefly defined. First, the information of the three-dimensional volume data regarding the interference signal and the tomographic image of the brightness based on the interference signal is referred to as OCT data, and the information of the three-dimensional volume data regarding the motion contrast is referred to as OCTA data. Further, the two-dimensional information that can be extracted from the three-dimensional volume data related to the interference signal and the tomographic image of the brightness based on the interference signal is referred to as an OCT image, and the two-dimensional information that can be extracted from the three-dimensional volume data related to the motion contrast is referred to as an OCTA image. In particular, an image generated by projecting or integrating three-dimensional volume data relating to an interference signal and a tomographic image of luminance based on the interference signal within a specified depth direction is referred to as an OCT front image (luminance En-Face). Further, an image generated by projecting or integrating three-dimensional volume data related to motion contrast is referred to as an OCTA front image. Further, a two-dimensional image including data in the depth direction is referred to as a tomographic image.

(High image quality section)
FIG. 3 is a block diagram showing a more detailed configuration of the image quality improving unit 101-47. The high image quality improving unit 101-47 is provided with a high image quality processing unit 301, an area detection unit 302, an ROI setting unit 303, a blend processing unit 304, and a BC adjusting unit 305.

The high image quality processing unit 301 performs high image quality processing on the medical image acquired by the acquisition unit 101-1 using a learned model for high image quality (high image quality engine, high image quality model) described later. And generate a high quality image. The high image quality processing unit 301 performs high image quality processing on the medical image (first image) of the subject by using the trained model obtained by learning using the medical image of the subject as learning data. It can function as an example of a high image quality improving unit for acquiring a high image quality image (second image) of a subject. The medical images to be processed for high image quality include OCT images, OCTA images, tomographic images, OCT front images, OCTA front images, SLO images, anterior eye observation images, and images such as analysis maps obtained by analyzing these images. May be.

The region detection unit 302 performs region detection processing on a medical image or a high-quality medical image using a learned model (region detection engine, region detection model) for region detection, which will be described later, and at least one region. (For example, the target area) is detected. For example, the area detection unit 302 can detect a target area and an area other than the target area. The region detection unit 302 can function as an example of a detection unit that detects a target region in a medical image or a high-quality image of a subject. For example, the region detection unit 302 detects a perfused region and a non-perfused region from the OCTA front image using a region detection engine. In the diagnosis of the fundus, by identifying the perfused area and the non-perfused area, it is possible to determine whether there is no blood flow where the blood vessel should be or where there should be no blood vessel (new blood vessel, etc.). You can judge. The target region is not limited to the perfused region and the non-perfused region, and may include a region where overcorrection occurs by the high image quality engine. For example, the target region may include a foveal avascular region, an optic nerve head, and the like in addition to the perfused region and the non-perfused region.

The area classification may be managed by the attribute information (label) indicating each area. For example, the area detection unit 302 may generate a label image having attribute information as a pixel value corresponding to each pixel position of the medical image. Further, in addition to the attribute information, a value indicating the certainty (reliability, probability) of each attribute information output from the area detection engine may be held in association with each pixel of the medical image.

The ROI setting unit 303 sets the ROI (Region Of Interest) based on the area detected by the area detection unit 302. The ROI can be at least one of the regions (target regions) detected by the region detection unit 302. The ROI may be automatically set by the area detection unit 302 as an area having predetermined attribute information, and in this case, the ROI setting unit 303 may be omitted.

For example, the blend processing unit 304 performs a blend process, which is a kind of image processing, on a set ROI which is a region having predetermined attribute information, using a medical image and a high-quality image of the medical image. As the blending treatment, any known treatment such as an α blending treatment may be used. In the blend processing unit 304, the difference between the pixel value of the target area and the pixel value of the area other than the target area is widened and the pixel value of the target area is lower than that of the target area in the medical image or the high-quality image. It can function as an example of an image processing unit that performs image processing.

For example, the BC adjustment unit 305 performs a BC (Brightness, Contrast) adjustment process, which is a correction process for at least one of brightness and contrast, on a set ROI which is a region having predetermined attribute information. Here, the contrast correction process may include any known contrast correction process, and may include, for example, correction using a tone curve or a gamma curve and level correction. The BC adjustment unit 305 also makes the difference between the pixel value of the target area and the pixel value of the area other than the target area wider and lowers the pixel value of the target area with respect to the target area in the medical image or the high-quality image. It can function as an example of an image processing unit that performs image processing.

In addition, only one of the blend processing unit 304 and the BC adjustment unit 305 may be provided. Further, when both the blend processing unit 304 and the BC adjustment unit 305 are provided, the image processing device 101 receives either the blend processing or the BC adjustment processing as the image processing to be executed according to the instruction by the operator, or It may be configured so that both can be selected. When both the blending process and the BC adjustment process are executed, the image processing device 101 switches or arranges the images subjected to the respective image processes according to the operation of the UI or the like and displays them on the output unit 103. It may be configured in.

(Medical image quality improvement engine)
Hereinafter, the high image quality engine used by the high image quality processing unit 301 will be described with reference to FIGS. 4 to 7. In the present embodiment, the high image quality engine is stored in the storage unit 101-3, but the high image quality engine may be provided in an external device connected to the image processing device 101. In this case, the high image quality processing unit 301 can perform high image quality processing by using the high image quality engine provided in the external device.

The high image quality engine according to this embodiment is a trained model obtained by performing training (learning) related to a machine learning algorithm. In the present embodiment, for training of a machine learning model related to a machine learning algorithm, input data which is a low-quality image having specific shooting conditions assumed as a processing target and output data which is a high-quality image corresponding to the input data. The training data composed of the pair group of is used. The specific shooting conditions specifically include a predetermined shooting portion, a shooting method, a shooting angle of view, an image size, and the like.

Here, a general trained model will be briefly described. The trained model is a machine learning model in which training (learning) is performed on an arbitrary machine learning algorithm using appropriate learning data in advance. The training data is composed of one or more pairs of input data and output data (correct answer data). The format and combination of the input data and output data of the pair group constituting the training data may be one of which is an image and the other of which is a numerical value, or one of which is composed of a plurality of image groups and the other of which is a character string. May be an image or the like, and may be suitable for a desired configuration.

Specifically, for example, learning data (hereinafter, first learning data) composed of a pair group of an image acquired by OCT and an imaging site label corresponding to the image can be mentioned. The imaged part label is a unique numerical value or character string representing the part. Further, as an example of other learning data, it is composed of a pair group of a low-quality image with a lot of noise acquired by normal OCT shooting and a high-quality image shot multiple times by OCT to improve the image quality. Examples include learning data (hereinafter, second learning data).

At this time, when the input data is input to the trained model, the output data according to the design of the trained model is output. The trained model outputs, for example, output data that is likely to correspond to the input data according to the tendency trained with the trained data. In addition, the trained model outputs, for example, the certainty (reliability, probability) corresponding to the input data as a numerical value for each type of output data according to the tendency trained using the training data. Can be done.

Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first training data, the machine learning model outputs a imaging site label of the imaging site captured in the image. Or output the probability for each imaged part label. Further, for example, when a low-quality image with a lot of noise acquired by normal OCT shooting is input to the machine learning model trained with the second training data, the machine learning model is shot multiple times by OCT to improve the image quality. Outputs a high-quality image equivalent to the processed image. The machine learning model can be configured so that the output data output by itself is not used as training data from the viewpoint of maintaining quality.

The machine learning algorithm also includes a method related to deep learning such as a convolutional neural network (CNN). In the method related to deep learning, if the parameter settings for the layers and nodes constituting the neural network are different, the tendency of training using the training data may be different in the degree of reproduction in the output data. For example, in a deep learning machine learning model using the first training data, if more appropriate parameters are set, the probability of outputting a correct imaged region label may be higher. Further, for example, in a machine learning model of deep learning using the second learning data, if more appropriate parameters are set, a higher quality image may be output.

Specifically, the parameters in the CNN are, for example, the kernel size of the filter, the number of filters, the stride value, and the dilation value set for the convolution layer, and the number of nodes output by the fully connected layer. Etc. can be included. The parameter group and the number of training epochs can be set to preferable values for the usage pattern of the trained model based on the training data. For example, based on the learning data, it is possible to set a parameter group and an epoch number that can output a correct imaged part label with a higher probability and output a higher quality image.

One of the methods for determining such a parameter group and the number of epochs is illustrated. First, 70% of the pair group constituting the learning data is used for training, and the remaining 30% is randomly set for evaluation. Next, the machine learning model is trained using the training pair group, and the training evaluation value is calculated using the evaluation pair group at the end of each training epoch. The training evaluation value is, for example, the average value of the value group in which the output when the input data constituting each pair is input to the machine learning model being trained and the output data corresponding to the input data are evaluated by the loss function. be. Finally, the parameter group and the number of epochs when the training evaluation value becomes the smallest are determined as the parameter group and the number of epochs of the machine learning model. In this way, by dividing the pair group that constitutes the training data into training and evaluation and determining the number of epochs, the machine learning model overfits the training pair group. Can be prevented.

Here, the high image quality engine according to the present embodiment is configured as a module that outputs a high image quality image obtained by improving the high image quality of the input low image quality image. Here, high image quality in the present specification means converting an input image into an image having an image quality more suitable for image diagnosis, and a high image quality image is converted into an image having an image quality more suitable for image diagnosis. Refers to the image. Further, the low-quality image is, for example, a two-dimensional image or a three-dimensional image acquired by X-ray photography, CT, MRI, OCT, PET, SPECT, or the like, or a three-dimensional moving image of CT continuously photographed. It was taken without setting the image quality. Specifically, the low-quality image is, for example, an image acquired by low-dose radiography with an X-ray imaging device or CT, an MRI imaging without using a contrast medium, a short-time OCT imaging, or a small number of imaging images. Includes OCTA images and the like acquired by the number of times.

Further, when a high-quality image having less noise or high contrast is used for image analysis such as blood vessel analysis processing of an image such as OCTA or area segmentation processing of an image such as CT or OCT, a low-quality image can be obtained. In many cases, analysis can be performed more accurately than using it. Therefore, the high-quality image output by the high-quality engine may be useful not only for image diagnosis but also for image analysis.

Further, the content of the image quality suitable for the image diagnosis depends on what is desired to be diagnosed by various image diagnoses. Therefore, although it cannot be said unconditionally, for example, the image quality suitable for image diagnosis has less noise, high contrast, the image to be photographed is shown in colors and gradations that are easy to observe, and the image size is large. Includes image quality such as high resolution. In addition, it is possible to include an image quality in which an object or a gradation that does not actually exist and is drawn in the process of image generation is removed from the image.

In the image processing method constituting the image quality improvement method by the image quality improvement processing unit 301 in the present embodiment, processing using various machine learning algorithms such as deep learning is performed. In the image processing method, in addition to the processing using the machine learning algorithm, the existing arbitrary image processing such as various image filter processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge-based image processing is performed. May be processed.

Hereinafter, a configuration example of the CNN related to the high image quality engine according to the present embodiment will be described with reference to FIG. FIG. 4 shows an example of the configuration of the high image quality engine. The configuration shown in FIG. 4 is composed of a plurality of layers responsible for processing and outputting an input value group. As shown in FIG. 4, the types of layers included in the configuration include a convolution layer, a Downsampling layer, an Upsampling layer, and a Merger layer.

The convolution layer is a layer that performs convolution processing on the input value group according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image. The downsampling layer is a process of reducing the number of output value groups to be smaller than the number of input value groups by thinning out or synthesizing input value groups. Specifically, for example, there is a Max Polling process. The upsampling layer is a process of increasing the number of output value groups to be larger than the number of input value groups by duplicating the input value group or adding interpolated values from the input value group. Specifically, for example, there is a linear interpolation process. The composite layer is a layer in which a value group such as an output value group of a certain layer or a pixel value group constituting an image is input from a plurality of sources, and the processing is performed by concatenating or adding them.

In such a configuration, the value group in which the pixel value group constituting the input image Im410 is output via the convolution processing block and the pixel value group constituting the input image Im410 are combined in the composite layer. After that, the combined pixel value group is formed into a high-quality image Im420 at the final convolution layer. Although not shown, as an example of changing the configuration of the CNN, for example, a batch normalization layer or an activation layer using a normalized linear function (Rectifier Liner Unit) may be incorporated after the convolution layer. You may.

Here, the GPU can perform efficient operations by processing more data in parallel. Therefore, when learning is performed a plurality of times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in the present embodiment, the GPU is used in addition to the CPU for the processing by the image processing unit 101-4, which is an example of the learning unit. Specifically, when a learning program including a learning model is executed, learning is performed by the CPU and the GPU collaborating to perform calculations. In the processing of the learning unit, the calculation may be performed only by the CPU or GPU. Further, the high image quality processing unit 301 may also be realized by using the GPU in the same manner as the learning unit.

Further, the learning unit may include an error detection unit and an update unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error back propagation method is a method of adjusting the coupling weighting coefficient and the like between the nodes of each neural network so that the above error becomes small.

When using some image processing methods such as image processing using CNN, it is necessary to pay attention to the image size. Specifically, it should be noted that different image sizes may be required for the input low-quality image and the output high-quality image in order to deal with the problem that the peripheral part of the high-quality image is not sufficiently high-quality. Should be.

Although not specified in this embodiment for the sake of clear explanation, when a high image quality engine that requires different image sizes for the image input to the image quality engine and the image to be output is adopted, the image size is appropriate. It is assumed that is being adjusted. Specifically, padding is performed on an input image such as an image used for training data for training a machine learning model or an image input to a high image quality engine, and a shooting area around the input image is captured. Adjust the image size by combining them. The area to be padded is filled with a constant pixel value, a neighboring pixel value, or mirror padding according to the characteristics of the high image quality method so that the image quality can be effectively improved.

Further, the high image quality improving method by the high image quality processing unit 301 may be carried out by only one image processing method, or may be carried out by combining two or more image processing methods. Further, a plurality of high image quality improvement method groups may be performed in parallel to generate a plurality of high image quality image groups, and then the highest image quality high image quality image may be finally selected as the high image quality image. The selection of the highest image quality image may be automatically performed using the image quality evaluation index, or a plurality of high image quality image groups may be displayed on the UI provided in the output unit 103 or the like. It may be performed according to the instruction of the examiner (operator).

Since the input image without high image quality may be more suitable for image diagnosis, the input image may be added to the final image selection target. Further, parameters may be input to the high image quality engine together with the low image quality image. A parameter for specifying the degree of high image quality and a parameter for specifying the image filter size used in the image processing method may be input to the high image quality engine together with the input image.

Here, the input data of the learning data of the high image quality engine according to the present embodiment is a low image quality image acquired by the same model as the photographing apparatus 100 and the same settings as the photographing apparatus 100. Further, the output data of the learning data of the high image quality engine is a high image quality image acquired by setting and image processing related to shooting conditions that require more man-hours than that of the same model. Specifically, the output data includes, for example, a high-quality image (superimposed image) obtained by performing superposition processing such as addition averaging on an image (original image) group acquired by shooting a plurality of times. can do.

Here, the motion contrast data of OCTA will be described as an example for a high-quality image and a low-quality image. The motion contrast data is data used in OCTA or the like that repeatedly shoots the same part of a shooting target and detects a temporal change of the shooting target between the shootings. Further, as described above, among the calculated motion contrast data (an example of three-dimensional medical image data), the front image is generated by using the data in a desired range in the depth direction of the imaging target to generate the OCTA. An En-Face image (OCTA front image) can be generated. In the following, the number of times that OCT data is repeatedly photographed at substantially the same position (substantially the same location) is referred to as NOR (Number Of Repeat).

Regarding the learning data according to the present embodiment, two different methods as examples of generating a high-quality image and a low-quality image by superposition processing will be described with reference to FIGS. 5 (a) and 5 (b). The method for generating a high-quality image and a low-quality image is not limited to these, and any known generation method may be used.

A first method according to an example of generating a high-quality image and a low-quality image will be described with reference to FIG. 5 (a). In the first method, as an example of a high-quality image, a motion contrast image generated from OCT data obtained by repeatedly shooting substantially the same position of a shooting target is used. In FIG. 5A, the motion contrast image Im510 shows a three-dimensional motion contrast image (three-dimensional motion contrast data). Further, the motion contrast image Im511 shows a two-dimensional motion contrast image (two-dimensional motion contrast data) constituting the three-dimensional motion contrast image.

The tomographic images Im501-1 to Im501-3 show OCT tomographic images (B scan images) for generating the motion contrast image Im511. Here, NOR corresponds to the number of OCT tomographic images in the tomographic images Im501-1 to Im501-3 in FIG. 5A, and NOR is 3 in the example of the figure. The tomographic images Im501-1 to Im501-3 are taken at predetermined time intervals (Δt). The substantially same position means one line in the front direction (XY) of the eye to be inspected, and corresponds to the position of the motion contrast image Im511 in FIG. 5A. The front direction is an example of a direction that intersects the depth direction.

Since the motion contrast data is data in which a change over time is detected, it is necessary to set NOR at least twice in order to generate this data. For example, when NOR is 2, one motion contrast data is generated. When the NOR is 3, if the motion contrast data is generated using only the OCT data of the adjacent time intervals (first and second times, second and third times), two motion contrast data are generated. When the motion contrast data is generated by using the OCT data at distant time intervals (first and third times), a total of three motion contrast data are generated. That is, if the NOR is increased three times, four times, ..., The number of motion contrast data at substantially the same position also increases. A high-quality motion contrast image can be generated by aligning a plurality of motion contrast images acquired by repeatedly shooting substantially the same position and performing superposition processing such as addition averaging. Therefore, in order to generate a high-quality motion contrast image, the NOR can be set to at least 3 times, and in order to obtain a higher-quality motion contrast image, for example, the NOR can be set to 5 times or more.

On the other hand, as an example of a low-quality image corresponding to this, a motion contrast image before superposition processing such as addition averaging can be used. In this case, the low-quality image can be used as a reference image for performing superposition processing such as addition averaging for generating a high-quality image, for example. When the superposition process is performed, if the position and shape of the target image are deformed with respect to the reference image to perform alignment, there is almost no spatial positional deviation between the reference image and the image after the superposition process. .. Therefore, it is possible to easily make a pair of a low-quality image and a high-quality image. It should be noted that the target image that has undergone the image transformation processing for alignment may be used as a low-quality image instead of the reference image.

By using each of the original image groups (reference image and target image) as input data and the corresponding superimposed image as output data, a plurality of pair groups can be generated. For example, when obtaining one superimposed image from 15 original image groups, a pair of the first original image and the superimposed image in the original image group, and the second original image in the original image group. It is possible to generate a pair with a superposed image. In this way, when one superposed image is obtained from the fifteen original image groups, it is possible to generate a group of 15 pairs of one image in the original image group and the superposed image. It should be noted that three-dimensional high-quality data can be generated by repeatedly photographing substantially the same position in the main scanning (X) direction and scanning while shifting the image in the sub-scanning (Y) direction.

Next, a second method according to an example of generating a high-quality image and a low-quality image will be described with reference to FIG. 5 (b). In the second method, a high-quality image is generated by superimposing a motion contrast image obtained by photographing substantially the same area of an image target a plurality of times. The substantially same area refers to an area such as 3 × 3 mm or 10 × 10 mm in the front direction (XY) of the eye to be inspected, and by photographing the substantially same area of the imaged object multiple times, a tomography is performed. It is possible to acquire a three-dimensional motion contrast image (three-dimensional motion contrast data) including the depth direction of the image. When the same area is photographed a plurality of times and the superposition processing is performed, the NOR can be set to 2 or 3 times in order to shorten the imaging per time.

Further, in order to generate high-quality 3D motion contrast data, at least two or more 3D data in the same region are acquired. FIG. 5B shows an example of a plurality of three-dimensional motion contrast images. The motion contrast images Im520 to Im540 are three-dimensional motion contrast images similar to the motion contrast image Im510 described with reference to FIG. 5A. Using these two or more 3D motion contrast images, alignment processing is performed in the front direction (XY) and depth direction (Z), and after excluding data that is an artifact in each data, averaging processing is performed. I do. This makes it possible to generate one high quality 3D motion contrast image with the artifacts excluded.

On the other hand, the low-quality image corresponding to this can be used as reference data when performing superposition processing such as addition averaging. As described in the first method, since there is almost no spatial positional deviation between the reference image and the image after addition averaging, a pair of a low-quality image and a high-quality image can be easily obtained. It should be noted that an arbitrary three-dimensional motion contrast image generated from the target data subjected to the image transformation processing for alignment instead of the reference data may be used as a low-quality image.

In the first method, the burden on the subject is small because the shooting itself is completed only once. However, as the number of NORs increases, the shooting time for each shot becomes longer. In addition, good images may not always be obtained if artifacts such as eye turbidity or eyelashes are included during shooting. In the second method, the burden on the subject increases a little because the images are taken multiple times. However, the time required for one shooting is short, and even if an artifact is included in one shooting, if the artifact is not captured in another shooting, a clear image with few artifacts can be finally obtained. In view of these characteristics, when collecting data, any method can be selected according to the situation of the subject.

In the present embodiment, a low-quality image used as training data and a motion contrast image as high-quality images have been described as examples, but the image used as training data is not limited to this. Since the OCT data is acquired in order to generate the motion contrast data, it is possible to similarly generate a low-quality image and a high-quality image by using the OCT data. Further, although the description of the tracking process is omitted in the present embodiment, since the image is taken at substantially the same position or the substantially same area of the eye to be inspected, it is possible to take an image while tracking the eye to be inspected. The tracking process may be performed by any known method.

If a pair of three-dimensional high-quality data and low-quality data can be acquired, any pair of two-dimensional images can be generated from these. For example, a high-quality OCTA plane image can be generated by projecting or integrating the generated high-quality three-dimensional motion contrast image in a desired depth range and generating an arbitrary OCTA front image. Further, the low-quality image corresponding to this can be an arbitrary OCTA front image generated from the reference data when performing the superposition processing such as addition averaging. Also in this case, since there is almost no spatial positional deviation between the reference image and the image after addition averaging, a pair of a low-quality image and a high-quality image can be easily obtained. It should be noted that an arbitrary motion contrast front image generated from the target data subjected to the image transformation processing for alignment instead of the reference data may be used as a low-quality image.

An example of a pair of two-dimensional images used as such training data will be described in more detail with reference to FIGS. 6 (a) and 6 (b). For example, when the image used for the training data is an OCTA front image, the OCTA front image can be generated by projecting or integrating the three-dimensional volume data related to the motion contrast in a desired depth range as described above. can. Here, the depth range is a range in the Z direction shown in FIGS. 5 (a) and 5 (b).

FIG. 6A shows an example of an OCTA front image. As the OCTA front image used for the training data, OCTA front images generated in different depth ranges such as the surface layer (image Im610), the deep layer (image Im620), the outer layer (image Im630), and the choroidal vascular network (image Im640) are used. Can be done. The type of the OCTA front image is not limited to this, and the types may be increased by generating an OCTA front image in which different depth ranges are set by changing the reference layer and the offset value. When learning, the OCTA front images having different depths may be learned separately, or a plurality of images having different depth ranges may be combined and learned (for example, divided into the surface layer side and the deep layer side). Alternatively, the OCTA front image of the entire depth range may be trained together. When the En-Face image of the brightness generated from the OCT data is used as the training data, a plurality of En-Face images generated from an arbitrary depth range can be used as in the OCTA front image.

For example, consider the case where the high image quality engine includes a trained model obtained by using training data including a plurality of OCTA front images corresponding to different depth ranges of the eye to be inspected. At this time, the acquisition unit 101-1 can acquire the OCTA front image corresponding to a part of the depth range in the long depth range including the different depth range as the first image. That is, the OCTA front image corresponding to a depth range different from the plurality of depth ranges corresponding to the plurality of OCTA front images included in the learning data can be used as the input image at the time of high image quality processing. Of course, the OCTA front image having the same depth range as that at the time of learning may be used as an input image at the time of high image quality processing. Further, a part of the depth range may be set in response to the operator pressing an arbitrary button on the UI, or may be set automatically. The above-mentioned contents are not limited to the OCTA front image, and can be applied to, for example, an En-Face image having brightness.

When the image to be processed in the trained model is a tomographic image, training is performed using an OCT tomographic image which is a B scan image or a tomographic image of motion contrast data as training data. This will be described with reference to FIG. 6 (b). In FIG. 6B, the images Im651 to Im653 are OCT tomographic images (luminance tomographic images). The image is different in FIG. 6 (b) because it shows a tomographic image of a place where the position in the sub-scanning (Y) direction is different. In the tomographic image, learning may be performed together without worrying about the difference in the position in the sub-scanning direction. However, in the case of images taken at different locations (for example, the center of the macula or the center of the optic nerve head), learning may be performed separately for each region, or together without worrying about the imaging site. You may try to learn. Since the image feature amount is significantly different between the OCT tomographic image and the tomographic image of the motion contrast data, it is better to perform the learning separately.

As the high-quality image used as the output data of the training data, for example, a superimposed image can be used as described above. The superposed image obtained by superimposing is a high-quality image suitable for image diagnosis because the pixels commonly drawn in the original image group are emphasized. In this case, the generated high-quality image becomes a high-contrast image in which the difference between the low-luminance region and the high-luminance region is clear as a result of emphasizing the pixels commonly drawn. Further, for example, in a superposed image, random noise generated at each shooting can be reduced, or a region that was not well depicted in the original image at a certain point in time can be interpolated by another original image group.

Further, when the superimposed image is used as the output data of the training data, a low-quality image used as the input data of the training data can be generated from the superimposed image. In this case, for example, the superimposed image is once downsampled to lower resolution, and the low resolution image is upsampled by a known method (nearest neighbor method, bilinear method, etc.) of the training data. It can be input data. It is also possible to configure a high image quality engine that improves the sense of resolution by performing learning using such a pair of images as learning data.

Further, when it is necessary to configure the input data of the machine learning model with a plurality of images, a required number of original image groups can be selected from the original image groups and used as input data. For example, in the case of obtaining one superimposed image from a group of 15 original images, if two images are required as input data of a machine learning model, a pair group of 105 (15C2 = 105) can be generated. be.

Of the pairs that make up the training data, the pairs that do not contribute to high image quality can be removed from the training data. For example, if the high-quality image that is the output data that constitutes the pair of training data has an image quality that is not suitable for image diagnosis, the image output by the high-quality engine trained using the training data is also suitable for image diagnosis. There is a possibility that the image quality will not be good. Therefore, by removing the pair whose output data has an image quality unsuitable for image diagnosis from the learning data, it is possible to reduce the possibility that the high image quality engine produces an image with an image quality unsuitable for image diagnosis.

Further, when the average brightness and the brightness distribution of the paired image groups are significantly different, the image quality improvement engine learned using the training data is not suitable for image diagnosis having a brightness distribution significantly different from the low image quality image. May be output. Therefore, pairs of input data and output data having significantly different average luminance and luminance distribution can be removed from the training data.

Furthermore, if the structure and position of the shooting target drawn on the pair of images are significantly different, the image quality improvement engine learned using the training data will shoot at a structure and position that is significantly different from the low-quality image. There is a possibility of outputting an image that is not suitable for diagnostic imaging. Therefore, it is possible to remove the pair of input data and output data whose structure and position of the imaged object to be drawn differ greatly from the training data. Further, the high image quality engine can be configured so that the high image quality image output by itself is not used as learning data from the viewpoint of maintaining quality.

By using the high image quality engine that has been learned in this way, the high image quality processing unit 301 can increase the contrast and noise by superimposing when the medical image acquired in one shooting is input. It is possible to output a high-quality image as if it had been reduced. Therefore, the high image quality processing unit 301 can generate a high image quality image suitable for image diagnosis based on the low image quality image which is an input image.

Although an example of using a superposed image as the output data of the training data has been described here, the output data of the training data of the high image quality engine is not limited to this. The output data of the training data may be a high-quality image corresponding to the input data, for example, an image with noise reduced to be suitable for diagnosis, an image with contrast correction, an image with high resolution, and more manpower. Images taken under many shooting conditions may be used. Further, an image obtained by subjecting a low-quality image used as input data to image processing using statistical processing such as maximum a posteriori estimation (MAP estimation) processing can also be used as output data of training data. As a method for generating a high-quality image, any known method may be used.

Further, as the high image quality engine, a plurality of high image quality engines that independently perform various high image quality processing such as noise reduction, contrast correction, and high resolution may be prepared. Further, one high image quality engine that performs at least two high image quality processing may be prepared. In these cases, as the output data of the learning data, a high-quality image corresponding to the desired processing may be used. For example, for a high image quality engine that performs individual processing, a high image quality image that has undergone individual processing such as noise reduction processing may be used as output data of learning data. Further, for a high image quality engine that performs a plurality of high image quality processing, for example, a high image quality image that has been subjected to noise reduction processing, contrast correction processing, or the like may be used as the output data of the learning data.

(Medical image area detection engine)
Next, the area detection engine for medical images used by the area detection unit 302 will be described using a contrast front image as an example. The region detection unit 302 according to the present embodiment detects a region classified into a perfused region and a non-perfused region in order to confirm, for example, the presence or absence of blood flow in the contrast front image. Hereinafter, the non-perfused region will be referred to as NPA (Non Perfusion Area). Further, the region detection unit 302 according to the present embodiment may detect, for example, a foveal avascular region (FAZ) or an optic nerve head (ONH) as a region. As the information of the detected area, the label of the area can be associated with the pixels in the image. Deep learning that associates labels with all pixels in an image in this way is called semantic segmentation.

Generally, in deep learning, the work of creating output data (correct answer data) of learning data is called annotation. Regarding the training data related to the trained model for region detection, output data can be created by giving a label indicating the region to be classified for each pixel position of the input data or the corresponding image by annotation. .. An image having information indicating the label of each region generated in this way as a pixel value is called an region label image. All annotation work may be performed manually, or some parts may be performed automatically. It is also possible to improve the efficiency of annotation work by performing deep learning at any time when a certain work is completed and modifying the label while referring to the segmentation result by the region detection engine in the middle stage. When creating the area label image, not only the input data but also other information may be used. For example, when creating an area label image to be output data of training data in which the input data is a high-quality image, an area label image is created by referring to, for example, using an image before the image quality is improved. May be good.

As a region detection engine, a trained model in which machine learning is performed using training data as a pair group of input data which is a predetermined number of medical images and output data which is a region label image corresponding to the medical images is used. be able to. In the area detection engine trained in this way, it is possible to output the certainty (reliability, probability) of the area label for each pixel. Therefore, the area detection unit 302 can output, for example, a label having a higher probability than the other labels among the labels with respect to the probability of the label output from the area detection engine as the area detection result. Further, among the labels, the label having a probability higher than the threshold value can be output as the detection result. At this time, if there are a plurality of labels having a probability higher than the threshold value, all of them may be output, or labels having a higher probability than the other labels may be output as the detection result. Further, the region detection unit 302 may determine the detection result by using the machine learning model from the probability of each label obtained by using the trained model. The machine learning algorithm used in this case may be a different type of machine learning algorithm than the machine learning algorithm used to obtain the probability of the label, for example, a neural network, a support vector machine, AdaBoost, a Basian network, or It may be a random forest or the like.

A common model may be used as the high image quality engine and the area detection engine. In this case, for example, a known U-Net model or the like can be used, and by making the high image quality engine and the area detection engine the same, each inference processing can be performed only by replacing the parameters.

Further, as the input data for area detection, the above-mentioned high-quality image (superimposed image) may be used, or a high-quality image to which a high-quality engine is applied to the input medical image is used as input data. You may use it. In this way, by performing learning according to the input data, it is possible to configure an optimum region detection engine for each input data.

The region detection engine does not necessarily have to be an artificial intelligence engine based on machine learning, and may be configured by using a rule-based algorithm or the like based on the structure of the eye to be inspected, including various conventional filter processing and the like. .. Therefore, the region detection engine used by the region detection unit 302 may, for example, extract a FAZ or NPA region by combining a filter process such as a known Gaussian filter, or responds to an instruction by an operator via a GUI or the like. Regions may be extracted. Further, the region detection engine may detect a region in a motion contrast image by classifying it into a perfused region and a non-perfused region by threshold processing using the structure of the eye to be inspected or a predetermined threshold.

(Image processing procedure for medical images)
Next, a series of image processing procedures by the image processing apparatus 101 according to the present embodiment will be described with reference to FIG. 7. FIG. 7 is a flow chart showing a schematic flow of image processing according to the present embodiment.

When the image processing according to the present embodiment is started, the acquisition unit 101-1 acquires a medical image in step S701. The acquisition unit 101-1 may acquire a medical image from the photographing apparatus 100 or an external device, or may acquire a medical image generated by using the signal data acquired from these. In the present embodiment, the acquisition unit 101-1 acquires, for example, a contrast front image as a medical image. The medical image is not limited to this, and may be a tomographic image, an En-Face image of luminance, an SLO image, an analysis image (analysis map) obtained by analyzing the medical image of the subject, or the like.

In step S702, the high image quality processing unit 301 applies the high image quality processing, which is the first image processing, to the acquired medical image by using at least one high image quality engine described above. Acquire an image-quality medical image. The high image quality processing unit 301 according to the present embodiment uses the high image quality engine to acquire the high image quality OCTA front image from the OCTA front image.

Next, in step S703, the region detection unit 302 performs region detection processing on the medical image acquired in step S701 or the high-quality medical image acquired in step S702 using the region detection engine described above. Apply. The region detection unit 302 detects at least two regions in the medical image or the high-quality medical image by the region detection process, and acquires information indicating each region. For example, the area detection unit 302 may acquire an area label image in which a label indicating each area with respect to the detected area is used as information of each pixel, or a label indicating each area in the information of each pixel for a medical image. May be added. The area detection unit 302 only needs to be able to output information indicating the detected area, and the information indicating the detected area is not limited to the above-mentioned format. For example, the area detection unit 302 may be label information or the like associated with pixel information of a medical image. Further, as described above, a value indicating the certainty (reliability, probability) of each attribute output from the region detection engine may be associated with each pixel of the medical image and added to the pixel information. In the present embodiment, a region detection engine is used to detect a perfused region and a non-perfused region (NPA) in an OCTA front image or a high-quality OCTA front image. It should be noted that the pixel positions of the detected regions in the medical image or the high-quality medical image can correspond to each other in the medical image and the high-quality medical image.

When the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the high-quality medical image based on the output information. The ROI may be set for at least one region among the regions detected by the region detection unit 302. In the present embodiment, the ROI setting unit 303 sets the ROI for the NPA in the OCTA front image with high image quality. As described above, the ROI may be set by the area detection unit 302.

Finally, in step S704, the blend processing unit 304 or the BC adjustment unit 305 widens the difference between the pixel value of the ROI and the pixel value in the region other than the ROI with respect to the ROI in the high-quality medical image. The second image processing is applied. At this time, the blend processing unit 304 or the BC adjustment unit 305 sets the ROI pixel value lower than the ROI pixel value before the second image processing with respect to the ROI in the high-quality medical image. Apply the image processing of 2. For example, the blend processing unit 304 has a second image process for the ROI in the high-quality medical image, that is, the medical image acquired in step S701 and the high-quality medical image acquired in step S702. Blending process. The blending process may be performed using any known method, and the blending ratio may be a predetermined ratio or may be set according to the instruction of the operator. By performing the blending process, the image before the overcorrection and the image with the overcorrection are blended in the area where the overcorrection is caused by the high image quality processing using the trained model, and the overcorrection is performed. The difference between the pixel value in the area where is occurring and the pixel value in the other area is widened. In other words, by performing the blending process, the pixel value in the region where the overcorrection occurs becomes low. In such a process, the pixel value can be lowered only in the region where the overcorrection occurs without lowering the pixel value of the entire image. Therefore, for example, overcorrection in a region such as NPA or FAZ can be suppressed.

Further, the BC adjustment unit 305 performs BC adjustment processing as a second image processing on the ROI in the medical image with high image quality. The BC adjustment process may be performed using any known method, for example, the brightness in ROI may be set to a predetermined value, or the brightness may be increased or decreased by a predetermined value. Further, in the ROI, the contrast may be adjusted by using, for example, a tone curve or a gamma curve. Further, in the BC adjustment process, at least one of the brightness and the contrast in the ROI may be set to a value according to the instruction of the operator, or may be increased or decreased by the value. For example, the BC adjustment unit 305 may determine the correction value of the brightness or the setting value of the tone curve or the like used for the contrast adjustment according to the instruction of the operator. By performing the BC adjustment processing, in the region where the overcorrection by the high image quality processing using the trained model occurs, the pixel value of the region where the overcorrection occurs and the pixel value of the other region At least one of brightness and contrast is adjusted so that the difference is widened. In other words, by performing the BC adjustment process, the pixel value in the region where the overcorrection occurs becomes low. In such a process, the pixel value can be lowered only in the region where the overcorrection occurs without lowering the pixel value of the entire image. Therefore, for example, overcorrection in a region such as NPA or FAZ can be suppressed.

In the present embodiment, the blending processing unit 304 performs blending processing of the OCTA front image and the high-quality OCTA front image with respect to the NPA set as the ROI in the high-quality OCTA front image. Further, the BC adjustment unit 305 performs BC adjustment processing of the high image quality OCTA front image with respect to the NPA set as the ROI in the high image quality OCTA front image. As described above, the processing by the blend processing unit 304 and the processing by the BC adjustment unit 305 may be performed only by one or both. The high-quality image subjected to the second image processing by the blend processing unit 304 and the BC adjustment unit 305 is displayed on the output unit 103 by the display control unit 101-5, or is output to an external device or the like by the output unit 103. , May be stored in the storage unit 101-3 or the external storage device 102.

By performing the image processing according to the present embodiment in this way, the region where the overcorrection occurs due to the image processing of the medical image using the trained model is detected, and the overcorrection is reduced with respect to the detected region. Processing can be applied. For example, the NPA in the OCTA front image can be detected, and a blending process or a BC adjustment process for reducing overcorrection can be performed on the NPA in the high-quality OCTA front image acquired by using the trained model.

As mentioned above, NPA is particularly important in diagnosing fundus images. By identifying the NPA, it is possible to determine whether there is no blood flow where the blood vessel should be, or whether there is some blood flow where there should be no blood vessel (new blood vessel, etc.). In particular, with respect to FAZ and NPA, by leaving the original state before the image quality is improved to some extent, it is possible to support the operator (examiner)'s judgment regarding the distinction between noise and blood vessels in diagnostic imaging.

On the other hand, it is possible to improve the visibility by making the NPA more like NPA by positively removing noise and the like by setting the brightness and contrast to a desired state. In this case as well, it is possible to support the operator's judgment regarding the distinction between noise and blood vessels in the diagnostic imaging.

In general, since the diagnosis is performed in a complex manner including other tests, the second image processing method may be set in advance or may be appropriately selected. Further, the second image processing method may be changed depending on the area to be classified. Furthermore, it is possible to change the setting depending on the eye to be inspected. Further, the depth of the OCTA front image, that is, the setting can be changed between the shallow layer and the deep layer. These settings may be stored for each user or subject.

As described above, the image processing apparatus 101 according to the present embodiment includes a high image quality processing unit 301, an area detection unit 302, a blend processing unit 304, and a BC adjustment unit 305. The high image quality processing unit 301 uses a high image quality model (high image quality engine) obtained by learning using the medical image of the subject as learning data, and is higher than the first image of the medical image of the subject. Image quality processing is performed to obtain a high-quality second image of the medical image of the subject. The region detection unit 302 detects a target region in the medical image (first image or second image) of the subject. At least one of the blend processing unit 304 and the BC adjustment unit 305 is such that the difference between the pixel value of the target area and the pixel value of the area other than the target area is widened with respect to the target area in the second image, and the target area is widened. Image processing is performed so that the pixel value of is lower than the pixel value of the target area before image processing.

The blend processing unit 304 performs image processing on the first image and the second image so that the difference between the pixel value in the target area and the pixel value in the area other than the target area is widened and the pixel value in the target area is lower. Performs a blending process to blend with the image of. Further, as image processing, the BC adjustment unit 305 adjusts the brightness and contrast so that the difference between the pixel value in the target area and the pixel value in the area other than the target area is widened and the pixel value in the target area is lower. A BC adjustment process for correcting at least one is performed. The medical image of the subject may be, for example, a motion contrast image of the eye to be inspected, and the target region may include, for example, at least one of a non-perfused region, a foveal blood vessel region, and an optic disc region. Further, the region detection unit 302 can detect the target region by using the trained model obtained by learning using the medical image of the subject as the learning data.

With such a configuration, the image processing apparatus according to the present embodiment detects a region such as NPA in which overcorrection occurs by image processing of a medical image using a trained model, and the overcorrection is made for the detected region. Image processing can be performed to reduce the problem. In such a process, the pixel value can be lowered only in the region where the overcorrection occurs without lowering the pixel value of the entire image. Therefore, it is possible to reduce overcorrection due to image processing of medical images using the trained model. This makes it possible to support the operator's judgment regarding the distinction between noise and blood vessels in diagnostic imaging.

Here, various modifications are possible with respect to the series of image processing according to the present embodiment. Hereinafter, specific examples of a series of image processing according to the present embodiment will be described in detail with reference to FIGS. 8 to 13. For each example, the same processing as the above-mentioned image processing shown in FIG. 7 will be omitted.

<First example of image processing method>
A first example of the image processing method will be described with reference to FIG. In this example, an image processing method for suppressing overcorrection by a high image quality engine for FAZ and NPA will be described.

Since the processes in steps S801 and S802 are the same as those in steps S701 and S702, the description thereof will be omitted. When the high-quality medical image is acquired in step S802, the process proceeds to step S803.

In step S803, the area detection unit 302 detects the FAZ and NPA areas in the medical image (input image) acquired in step S801 by using the area detection engine. FAZ and NPA may be detected separately or collectively as one region. Further, only one of the regions may be detected, or regions other than FAZ and NPA may be detected as perfusion regions. After the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the medical image improved in image quality in step S802 based on the output information. It should be noted that the pixel positions of the detected regions in the medical image can correspond to those in the medical image and the high-quality medical image.

Finally, in step S804, the blend processing unit 304 uses the input image and the high-quality medical image acquired in step S802 to perform blend processing on the detected region in the high-quality medical image. conduct. Here, as the blending process, a known α-blending process or the like may be used. Further, the blend ratio may be a fixed value, or the blend processing unit 304 may change the blend ratio in each pixel according to the reliability (probability, probability) of the attribute information output by the region detection engine. Specifically, for pixels having attribute information in the non-perfused region, the α blend ratio can be set so that the higher the reliability, the higher the blend ratio of the input image.

Further, when the attribute information such as FAZ and NPA is detected separately, the blend ratio of each may be changed according to the distinguished attribute information. Further, the input image may be appropriately combined with respect to the pixels around the region other than the FAZ and NPA, for example, the target region such as FAZ and NPA. This makes it possible to mitigate abrupt changes at the domain boundaries.

The blend ratio may be instructed by the operator via the UI. For example, a GUI such as a slide bar may be used so that the blend ratio corresponding to the suppression strength of overcorrection can be set.

According to this example, with respect to FAZ and NPA, the original state before the image quality is improved can be retained to some extent. Therefore, in diagnostic imaging, it is possible to support the operator's judgment regarding the distinction between noise and blood vessels.

When detecting the target region in the medical image (first image) which is the input image, the region detection unit 302 can use the region detection engine configured by the trained model as the region detection engine. .. In this case, as the training data of the region detection engine, the medical image as the input image is used as the input data, and the region label image having the information indicating the label of each region in the medical image as the input image as the pixel value is used as the output data. be able to. Further, the region detection unit 302 can also use a region detection engine configured by using a rule-based algorithm or the like based on the structure of the subject. Also in this case, the region detection unit 302 does not improve the image quality using the trained model, in other words, the region detection unit 302 detects the target region such as FAZ and NPA from the feature unit in the medical image in which the overcorrection does not occur. Can be detected.

<Second example of image processing method>
A second example of the image processing method will be described with reference to FIG. In this example, an image processing method for emphasizing the high image quality processing for NPA will be described.

When the performance of the region detection engine is high, the region detection unit 302 can more appropriately detect regions such as FAZ and NPA. For example, in FAZ and NPA, it is correct that there are no blood vessels. Therefore, when the performance of the region detection engine is high, an image more suitable for diagnosis can be obtained by further darkening these regions by BC adjustment after processing by the high image quality engine. The processing of this example, which can obtain an image more suitable for diagnosis by using such image processing, will be described in more detail below.

Since the processes in steps S901 and S902 are the same as those in steps S701 and S702, the description thereof will be omitted. When the high-quality medical image is acquired in step S902, the process proceeds to step S903.

In step S903, the area detection unit 302 detects the FAZ and NPA areas in the medical image (input image) acquired in step S901 by using the area detection engine. FAZ and NPA may be detected separately, or may be collectively detected as one region. Further, only one of the regions may be detected, or regions other than FAZ and NPA may be detected as perfusion regions. After the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the medical image that has been improved in image quality in step S902 based on the output information. It should be noted that the pixel positions of the detected regions in the medical image can correspond to those in the medical image and the high-quality medical image.

Finally, in step S904, the BC adjustment unit 305 performs BC adjustment processing on the pixels of the detected region (FAZ and NPA region) in the high-quality medical image acquired by using the high-quality engine in step S902. I do. The BC adjustment may be corrected so as to make the luminance value (brightness) related to the pixel value darker. For example, the brightness value may be simply lowered or set to zero. Further, image processing according to the original pixel value, such as gamma correction, may be applied. The strength of the correction may be changed by FAZ and NPA, respectively, or BC adjustment may be performed according to the reliability as long as the attribute and the reliability are retained as the information indicating the detected area.

The BC adjustment method and strength may be instructed by the operator via the UI. For example, a GUI such as a radio button may be used to select a method, or a GUI such as a slide bar may be used to specify the intensity.

According to this example, the visibility of regions such as NPA and FAZ can be improved. In this case as well, it is possible to assist the doctor's judgment regarding the distinction between noise and blood vessels in diagnostic imaging. Further, the suppression process shown in the first example and the emphasis process shown in the second example may be selected for each attribute of the area. For example, FAZ may be an enhancement process and NPA may be an suppression process. In this case, it is possible to generate an image more suitable for diagnosis according to each region, and it is possible to support the judgment of the doctor.

<Third example of image processing method>
A third example of the image processing method will be described with reference to FIG. In this example, an image process for detecting a region such as FAZ and NPA from a medical image whose image quality has been improved by using a high image quality engine and suppressing overcorrection in those regions will be described.

Since the processing in step S1001 and step S1002 is the same as in step S701 and step S702, the description thereof will be omitted. When the high-quality medical image is acquired in step S1002, the process proceeds to step S1003.

In step S1003, the area detection unit 302 uses the area detection engine to detect the FAZ and NPA areas in the medical image that has been improved in image quality in S1002. FAZ and NPA may be detected separately or collectively as one region. Further, only one of the regions may be detected, or regions other than FAZ and NPA may be detected as perfusion regions. After the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the medical image that has been improved in image quality in step S1002 based on the output information.

Finally, in step S1004, the blend processing unit 304 uses the high-quality medical image and the medical image acquired in step S1001 to perform blend processing on the detected region in the high-quality medical image. conduct. The blending process may be the same as the process described in the first example.

In the case of this example as well, as in the first example, with respect to FAZ and NPA, the original state before the image quality is improved can be left to some extent. Therefore, in diagnostic imaging, it is possible to support the operator's judgment regarding the distinction between noise and blood vessels.

When detecting the target area in the medical image with high image quality, the area detection unit 302 can use the area detection engine configured by the trained model. In this case, as the training data of the region detection engine, a high-quality medical image (second image) is used as input data, and information indicating a label of each region in the medical image (first image) which is an input image. The area label image having the above as the pixel value can be used as the output data. Here, the high-quality medical image used as the input data is a high-quality medical image using a high-quality engine, and is a high-quality image in which overcorrection occurs in a target area such as NPA. be able to. On the other hand, the area label image to be the output data may be an image obtained by labeling the medical image before the image quality is improved by using the image quality improvement engine. In this case, the region detection engine configured by the trained model can output the region label image including the label of the target region where the overcorrection has occurred from the input high-quality image according to the learning tendency. .. By using a high-quality image as an input, it is expected that the region detection engine configured by the trained model can more appropriately extract the features in the image and output a more accurate region label image. Will be done.

<Fourth example of image processing method>
A fourth example of the image processing method will be described with reference to FIG. In this example, an image processing that detects areas such as FAZ and NPA from a medical image that has been improved in image quality by using a high image quality engine and emphasizes the high image quality processing in those areas by using an area detection engine will be described. do.

Since the processing in step S1101 and step S1102 is the same as in step S701 and step S702, the description thereof will be omitted. When the high-quality medical image is acquired in step S1102, the process proceeds to step S1103.

In step S1103, the area detection unit 302 uses the area detection engine to detect the FAZ and NPA areas in the high-quality medical image in step S1102. FAZ and NPA may be detected separately or collectively as one region. Further, only one of the regions may be detected, or regions other than FAZ and NPA may be detected as perfusion regions. After the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the medical image that has been improved in image quality in step S1102 based on the output information.

Finally, in step S1104, the BC adjustment unit 305 performs BC adjustment processing on the pixels in the detected region (FAZ or NPA region) in the high-quality medical image. The BC adjustment process may be the same process as the process described in the second example.

In the case of this example as well, the visibility of regions such as NPA and FAZ can be improved as in the second example. In this case as well, it is possible to assist the doctor's judgment regarding the distinction between noise and blood vessels in the diagnostic imaging. Further, the suppression process shown in the first example and the third example and the emphasis process shown in the fourth example may be selected for each attribute of the area. For example, FAZ may be an enhancement process and NPA may be an suppression process. In this case, it is possible to generate an image more suitable for diagnosis according to each region, and it is possible to support the judgment of the doctor.

<Fifth example of image processing method>
A fifth example of the image processing method will be described with reference to FIG. In this example, the region detection engine is used to detect regions such as FAZ and NPA for the medical image before the image quality is improved, BC adjustment is performed for those regions, and then the image quality enhancement engine is used. Image processing to which high image quality processing is applied will be described.

Since the processing in step S1201 is the same as that in step S701, the description thereof will be omitted. In step S1202, the area detection unit 302 detects the FAZ and NPA areas in the medical image (input image) acquired in step S1201 by using the area detection engine. After the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the input image based on the output information.

Next, in step S1203, the BC adjustment unit 305 performs BC adjustment processing on the pixels in the detected region (FAZ or NPA region) in the input image. The BC adjustment process may be the same process as that described in the second example.

Finally, in step S1204, the high image quality processing unit 301 executed the high image quality processing on the medical image to which the BC adjustment processing was performed in step S1203 using the high image quality engine to improve the image quality. Get a medical image. It should be noted that the medical image after the high image quality processing by the high image quality processing unit 301 may be further subjected to a blending process or a BC adjustment process based on the detected region. The blending process and the BC adjustment process may be the same processes as those described in the first example and the second example, respectively.

As described above, in this example, the region detection unit 302 detects the target region in the first image of the medical image of the subject. Further, the BC adjusting unit 305 increases the difference between the pixel value of the target area and the pixel value of the area other than the target area with respect to the target area in the first image of the medical image of the subject so as to widen the difference between the target area and the target area. The BC adjustment process is performed so that the pixel value is lower than the pixel value of the target area before the BC adjustment process. Further, the high image quality processing unit 301 uses the high image quality model obtained by learning using the medical image of the subject as learning data, and performs high image quality processing on the first image to which the BC adjustment processing has been performed. To obtain a high-quality second image of the medical image of the subject.

In the case of this example, by performing the BC adjustment processing, the medical image in which the pixel value of the region where the overcorrection occurs is lowered is subjected to the high image quality processing. In such a process, the pixel value can be lowered only in the region where the overcorrection occurs without lowering the pixel value of the entire image. Therefore, for example, overcorrection in a region such as NPA or FAZ can be suppressed, and visibility in a region such as NPA or FAZ can be improved. In this case as well, it is possible to assist the doctor's judgment regarding the distinction between noise and blood vessels in the diagnostic imaging.

<Sixth example of image processing method>
A sixth example of the image processing method will be described with reference to FIG. In this example, the area detection is performed by using the area detection engine for the medical image which has been improved in image quality by using the image quality improvement engine. After that, for the medical image before the image quality is improved, BC adjustment is performed based on the area detected by the above-mentioned area detection, and then the image processing using the image quality improvement engine is further applied. explain.

Since the processes in steps S1301 and S1302 are the same as those in steps S701 and S702, the description thereof will be omitted. When the high-quality medical image is acquired in step S1302, the process proceeds to step S1303.

In step S1303, the area detection unit 302 uses the area detection engine to perform area detection processing on the image whose image quality has been improved in step S1302. The detected region may be FAZ, NPA, or the like, as in the first to fifth examples. In this way, it can be expected that the detection performance can be improved by applying the region detection engine to the medical image with high image quality. After the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the input image based on the output information. It should be noted that the pixel positions of the detected regions in the high-quality medical image can be assumed to correspond to the medical image and the high-quality medical image.

Next, in step S1304, the BC adjustment unit 305 performs a BC adjustment process based on the region detected in step S1303 on the medical image (input image) acquired in step S1301. The BC adjustment process may be the same process as that described in the second example.

Finally, in step S1305, the high image quality processing unit 301 performs high image quality processing on the medical image subjected to the BC adjustment processing in step S1303 using the high image quality engine to improve the image quality. Get a medical image. It should be noted that the blending process and BC adjustment based on the area classification may be further added to the image after the high image quality processing by the high image quality processing unit 301 in step S1305. The blending process and BC adjustment may be the same processes as those described in the first example and the second example, respectively.

As described above, in this example, in this example, the high image quality processing unit 301 uses the high image quality model obtained by learning using the medical image of the subject as training data, and uses the medical image of the subject as the first image. An image in which the first image of the medical image of the subject is improved in quality is acquired from the image of 1. The region detection unit 302 detects a target region in an image obtained by improving the image quality of the first image of the medical image of the subject. The BC adjustment unit 305 increases the difference between the pixel value of the target area and the pixel value of the area other than the target area with respect to the target area in the first image of the medical image of the subject, and the pixel value of the target area. BC adjustment processing is performed so that is lower than the pixel value of the target area before BC adjustment processing. Further, the high image quality processing unit 301 performs high image quality processing on the first image to which the BC adjustment processing has been performed by using the high image quality model, and the high image quality second image of the medical image of the subject is subjected to the high image quality processing. Get the image of. The area detection unit 302 detects the target area by using a high-quality image of the first image of the medical image of the subject whose BC adjustment has not been performed by the BC adjustment unit 305.

In the case of this example, by performing the BC adjustment processing, the medical image in which the pixel value of the region where the overcorrection occurs is lowered is subjected to the high image quality processing. In such a process, the pixel value can be lowered only in the region where the overcorrection occurs without lowering the pixel value of the entire image. Therefore, for example, overcorrection in a region such as NPA or FAZ can be suppressed, and visibility in a region such as NPA or FAZ can be improved. In this case as well, it is possible to assist the doctor's judgment regarding the distinction between noise and blood vessels in the diagnostic imaging.

As described above, various modifications are possible for image processing using the high image quality engine and the region detection engine. These described examples are subject to numerous modifications and are not intended to be a limiting interpretation of the present disclosure. For example, when the follow-up of a medical image is performed, the blending process and the BC adjustment process based on the detected region may be adjusted to the same conditions, or these conditions may be stored and managed for each subject.

[Second Embodiment]
An image processing system including an image processing apparatus according to a second embodiment of the present invention will be described in detail with reference to FIG. FIG. 14 is a flow chart showing a flow of a series of image processing according to the present embodiment. Since the configuration of the image processing system according to the present embodiment is the same as the configuration of the image processing system according to the first embodiment, the same reference number will be used and the description thereof will be omitted. However, in the image quality improving unit according to the present embodiment, the area detection unit 302, the ROI setting unit 303, the blend processing unit 304, and the BC adjustment unit 305 may not be provided. Hereinafter, the image processing system according to the present embodiment will be described focusing on the differences from the image processing system according to the first embodiment.

In the first embodiment, a method for obtaining a more suitable high-quality image by combining a high-quality image engine using deep learning and a region detection engine and partially performing additional image correction has been described. .. On the other hand, in the present embodiment, a configuration for performing high image quality processing using a high image quality engine constructed by applying these methods will be described.

The machine learning algorithm of the high image quality engine according to the present embodiment may be the same as the machine learning algorithm of the high image quality engine according to the first embodiment. However, as the output data of the learning data of the high image quality engine according to the present embodiment, a high image quality image finally generated by any of the methods described in the first embodiment is used. These processing flows are applied to a predetermined number of medical images to output high-quality images, and a learning data set is constructed by pairing each of them. It can be expected that the high image quality engine obtained by learning using such learning data can generate a high image quality image in which overcorrection is suppressed according to the learning tendency. In addition, it can be expected that the image processing load at the time of inference can be reduced by performing machine learning collectively including the result of partial correction.

The processing procedure of the image processing apparatus 101 of the present embodiment will be described with reference to FIG. First, in step S1401, the acquisition unit 101-1 acquires a medical image. The process of step S1401 may be the same process as the process of step S701.

Next, in step S1402, the high image quality processing unit 301 improves the image quality of the medical image acquired in step S1401 by using the high image quality engine obtained by learning using the learning data as described above. Performs processing and acquires a high-quality image. The high-quality image acquired in this way becomes a high-quality image in which overcorrection is suppressed according to the learning tendency of the high-quality engine.

As described above, the image processing apparatus according to the present embodiment includes an acquisition unit 101-1 and an image quality improvement processing unit 301. The acquisition unit 101-1 acquires the first image of the medical image of the subject. The high image quality processing unit 301 performs high image quality processing using the first trained model obtained by learning using the medical image of the subject as training data, and blending processing or brightness in the target area of the medical image of the subject. Using the second trained model obtained by training using the medical image of the subject to which at least one of the correction processing of the contrast and the contrast has been performed as training data, the image quality improvement processing is performed on the first image. A second image of the medical image of the subject is obtained. Also in this case, it is possible to generate a high-quality image in which overcorrection is suppressed.

It should be noted that more accurate analysis can be performed by performing various image analyzes using the medical images whose image quality has been improved by any of the methods according to the first embodiment and the second embodiment. Further, in the disease screening by the artificial intelligence engine, the image with high image quality can be used as the input image, and it can be expected that more accurate processing can be performed.

[Modification 1]
In the first embodiment, as the training data when the region detection unit 302 detects the target region using the trained model, a plurality of front images corresponding to a plurality of depth ranges can be used. For example, as input data of training data of a region detection engine using a trained model, a plurality of OCTA front images corresponding to a plurality of depth ranges can be used. As the output data of the training data, the area label image corresponding to the OCTA front image used as the input data may be used. By using a common trained model obtained by learning such training data, the region detection unit 302 corresponds to an arbitrary depth range, for example, a depth range selected in response to an instruction from the examiner. A two-dimensional target area can be detected from the front image of the sample. The depth range may include, for example, a surface layer, a deep layer, an outer layer, a choroidal vascular network, a reference layer, and a different depth range in which the offset value is changed. Further, the front image is not limited to the OCTA front image, and may be an En-Face image.

Further, a plurality of trained models corresponding to the depth range may be prepared by using the training data for each depth range. In this case, the region detection unit 302 responds to an instruction from the examiner among a plurality of trained models obtained by learning using a plurality of front images of the subject corresponding to a plurality of depth ranges as learning data. A trained model corresponding to the selected depth range may be selected, and the selected trained model may be used to detect a two-dimensional target area from the front image of the subject. Further, the region detection unit 302 selects a trained model corresponding to the depth range of the medical image used for the region detection process from among such a plurality of trained models, and uses the selected trained model to receive a subject. A two-dimensional target area may be detected from the front image of the sample.

Further, as learning data, a plurality of front images having different depth ranges may be used in combination (for example, a plurality of front images are separated on the surface layer side and the deep layer side). In this case as well, the region detection unit 302 selects a trained model corresponding to the depth range selected according to the instruction from the examiner or the depth range of the medical image used for the region detection process, and the selected trained model. Can be used to detect a two-dimensional target area from the front image of the subject.

Further, a three-dimensional image may be used as input data of the training data of the region detection engine using the trained model. For example, a three-dimensional tomographic image or a three-dimensional motion contrast image is used as the input data of the training data, and a three-dimensional area label image in which the three-dimensional image is labeled is used as the output data of the training data. be able to. In this case, the region detection unit 302 can detect a three-dimensional target region from the three-dimensional image of the subject by using the trained model obtained by learning using the three-dimensional image of the subject as learning data.

Similarly, the high image quality engine may also be trained using the training data using the three-dimensional image as the input data and the high image quality three-dimensional image as the output data. In this case, the high image quality processing unit 301 improves the image quality from the three-dimensional medical image of the subject by using the high image quality engine obtained by learning using the three-dimensional medical image of the subject as learning data. A three-dimensional image can be acquired. Further, the blend processing unit 304 and the BC adjustment unit 305 can be configured to perform various processing on the three-dimensional target area in the three-dimensional medical image. In this case, by processing the three-dimensional medical image, the same effect as that of the first embodiment can be obtained. Similarly, a three-dimensional image may be used as the learning data of the high image quality engine according to the second embodiment. In this case, by processing the three-dimensional medical image, the same effect as that of the second embodiment can be obtained.

[Modification 2]
Learning is performed to adjust (tune) the trained model (trained model for high image quality, trained model for region detection) used for high image quality processing and area detection processing for each subject, and the subject You may generate a dedicated trained model. For example, transfer of a general-purpose trained model for generating high-quality medical images or a general-purpose trained model for detecting regions using medical images acquired in the subject's past examinations. It is possible to perform training and generate a trained model dedicated to the subject. By associating the trained model dedicated to the subject with the ID of the subject and storing it in an external device such as a storage unit 101-3 or a server, the image processing device 101 can perform the current inspection of the subject. It is possible to identify and use a trained model dedicated to the subject based on the ID of the subject. By using the trained model dedicated to the subject, it is possible to improve the accuracy of the high image quality processing and the area detection processing.

[Modification 3]
The high image quality engine may be prepared for each type of various images as input data. For example, a high image quality model for an anterior eye image, a high image quality model for an SLO image, a high image quality model for a tomographic image, a high image quality model for an OCTA front image, and the like may be prepared. Further, for the OCTA front image and the En-Face image, a high image quality model may be prepared for each depth range for generating an image. For example, a high image quality model for the surface layer, a high image quality model for the deep layer, and the like may be prepared. Further, the high image quality model may be one in which learning is performed on an image for each imaging site (for example, the center of the macula and the center of the optic nerve head), or may be one in which learning is performed regardless of the imaging site. ..

At this time, for example, using a high image quality model obtained by learning the fundus OCTA front image as training data, the fundus OCTA front image is improved in image quality, and further, the anterior eye OCTA front image is learned as training data. The high image quality model may be used to improve the image quality of the frontal image of the front eye OCTA. Further, the high image quality model may be one that has been learned regardless of the imaging region. Here, for example, the fundus OCTA frontal image and the anterior eye OCTA frontal image may have relatively similar distributions of blood vessels to be imaged. As described above, in a plurality of types of medical images in which the appearances of the objects to be imaged are relatively similar to each other, the feature quantities may be relatively similar to each other. Therefore, for example, by using a high image quality model obtained by learning the fundus OCTA front image as learning data, it is possible not only to improve the image quality of the fundus OCTA front image but also to improve the image quality of the anterior eye OCTA front image. May be done. Further, for example, by using a high image quality model obtained by learning the anterior eye OCTA front image as learning data, not only the anterior eye OCTA front image can be improved, but also the fundus OCTA front image can be improved. It may be configured. That is, using a high-quality model obtained by learning at least one type of frontal image of the fundus OCTA frontal image and the anterior eye OCTA frontal image as training data, the fundus OCTA frontal image and the anterior eye OCTA frontal image are At least one type of front image may be configured to have high image quality.

Here, consider a case where the anterior eye can also be photographed in the OCT device capable of photographing the fundus. At this time, for example, the fundus OCTA frontal image may be applied to the OCTA En-Face image in the fundus photography mode, and the anterior eye OCTA frontal image may be applied in the anterior ocular segment imaging mode. At this time, when the high image quality button is pressed, for example, in the fundus photography mode, in the display area of the En-Face image of OCTA, among the low image quality fundus OCTA front image and the high image quality fundus OCTA front image. One display may be configured to change to the other display. When the high image quality button is pressed, for example, in the anterior segment imaging mode, a low-quality front eye OCTA front image and a high-quality front eye OCTA front image are displayed in the OCTA En-Face image display area. The display of one of the above may be changed to the display of the other.

In the OCT device capable of photographing the fundus, the anterior eye adapter may be configured to be attachable when the anterior eye can also be imaged. Further, the optical system of the OCT device may be configured to be movable at a distance of about the axial length of the eye to be inspected without using the anterior eye adapter. At this time, the focus position of the OCT device may be configured to be largely changeable to the emmetropic side to the extent that an image is formed on the front eye.

Further, for example, a fundus OCT tomographic image may be applied to the tomographic image in the fundus photography mode, and an anterior eye OCT tomographic image may be applied to the tomographic image in the anterior ocular segment imaging mode. Further, the above-mentioned high image quality processing of the fundus OCTA front image and the anterior ocular OCTA front image can be applied as, for example, high image quality processing of the fundus OCT tomographic image and the anterior ocular OCT tomographic image. At this time, when the high image quality button is pressed, for example, in the fundus photography mode, one of the low image quality fundus OCT tomographic image and the high image quality fundus OCT tomographic image is displayed in the tomographic image display area. It may be configured to change to the other display. When the high image quality button is pressed, for example, in the anterior segment imaging mode, one of a low-quality anterior OCT tomographic image and a high-quality anterior OCT tomographic image is displayed in the tomographic image display area. The display of is changed to the other display.

Further, for example, the fundus OCTA tomographic image may be applied to the tomographic image in the fundus photography mode, and the anterior eye OCTA tomographic image may be applied to the tomographic image in the anterior ocular segment imaging mode. Further, the above-mentioned high image quality processing of the fundus OCTA front image and the anterior ocular OCTA front image can be applied as, for example, high image quality processing of the fundus OCTA tomographic image and the anterior ocular OCTA tomographic image. At this time, for example, in the fundus photography mode, in the tomographic image display area, the information indicating the blood vessel region (for example, motion contrast data above the threshold value) in the fundus OCTA tomographic image is superimposed on the fundus OCT tomographic image at the corresponding position. It may be configured to be displayed. Further, for example, in the anterior segment imaging mode, information indicating a vascular region in the anterior ocular OCTA tomographic image may be superimposed and displayed on the anterior ocular OCT tomographic image at a corresponding position in the tomographic image display region. ..

In this way, for example, when it is considered that the feature quantities (the appearance of the imaged object) of a plurality of types of medical images are relatively similar to each other, at least one type of the plurality of types of medical images may be used. Using a high image quality model obtained by learning a medical image as training data, at least one type of medical image of a plurality of types of medical images may be configured to have high image quality. Thereby, for example, a common trained model (common high image quality improvement model) can be used to enable execution of high image quality improvement of a plurality of types of medical images.

The display screen of the fundus photography mode and the display screen of the anterior eye part photography mode may have the same display layout or may have a display layout corresponding to each photography mode. Various conditions such as imaging conditions and analysis conditions may be the same or different between the fundus imaging mode and the anterior segment imaging mode.

Here, the target image for high image quality processing may be, for example, a plurality of OCTA front images (OCTA En-Face images, motion contrast En-Face images) (corresponding to a plurality of depth ranges). Further, the target image for high image quality processing may be, for example, one OCTA front image corresponding to one depth range. Further, the target image of the high image quality processing is, for example, a front image of luminance (En-Face image of luminance), an OCT tomographic image which is a B scan image, or a tomographic image of motion contrast data (instead of the OCTA front image). OCTA tomographic image) may be used. Further, the target image of the high image quality processing is not only the OCTA front image, but also various medical applications such as the front image of brightness, the OCT tomographic image which is a B scan image, and the tomographic image of motion contrast data (OCTA torpedo image). It may be an image. That is, the target image for the high image quality processing may be, for example, at least one of various medical images displayed on the display screen of the output unit 103. At this time, for example, since the feature amount of the image may be different for each type of image, a trained model for high image quality corresponding to each type of the target image for high image quality processing may be used. For example, when the high image quality button is pressed in response to an instruction from the examiner, the OCTA front image is not only processed for high image quality by using the trained model for high image quality corresponding to the OCTA front image. The OCT tomographic image may also be configured to be processed for high image quality by using a trained model for high image quality corresponding to the OCT tomographic image. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, the high image quality OCTA front image generated by using the trained model for high image quality corresponding to the OCTA front image is displayed. It may be configured to be changed to display a high-quality OCT tomographic image generated by using a trained model for high image quality corresponding to the OCT tomographic image. At this time, the line indicating the position of the OCT tomographic image may be configured to be superimposed and displayed on the OCTA front image. Further, the line may be configured to be movable on the OCTA front image according to an instruction from the examiner. Further, when the display of the high image quality button is in the active state, after the above line is moved, the high image quality OCT tomographic image obtained by performing high image quality processing on the OCT tomographic image corresponding to the position of the current line is performed. It may be configured to be modified to display an image. Further, by displaying the image quality enhancement button for each target image of the image quality enhancement process, the image quality enhancement process may be independently enabled for each image.

Further, the information indicating the blood vessel region (for example, motion contrast data equal to or higher than the threshold value) in the OCTA tomographic image may be superimposed and displayed on the OCT tomographic image which is the B scan image at the corresponding position. At this time, for example, when the image quality of the OCTA tomographic image is improved, the image quality of the OCTA tomographic image at the corresponding position may be improved. Then, the information indicating the blood vessel region in the OCTA tomographic image obtained by improving the image quality may be superimposed and displayed on the OCTA tomographic image obtained by improving the image quality. The information indicating the blood vessel region may be any information as long as it is identifiable information such as color. Further, the superimposed display and non-display of the information indicating the blood vessel region may be configured to be changeable according to an instruction from the examiner. Further, when the line indicating the position of the OCT tomographic image is moved on the OCTA front image, the display of the OCT tomographic image may be updated according to the position of the line. At this time, since the OCTA tomographic image at the corresponding position is also updated, the superimposed display of the information indicating the blood vessel region obtained from the OCTA tomographic image may be updated. Thereby, for example, the three-dimensional distribution and state of the blood vessel region can be effectively confirmed while easily confirming the positional relationship between the blood vessel region and the region of interest at an arbitrary position. Further, the image quality of the OCTA tomographic image may be improved by an addition averaging process or the like of a plurality of OCTA tomographic images acquired at the corresponding positions instead of using the trained model for the image quality improvement. .. Further, the OCT tomographic image may be a pseudo OCT tomographic image reconstructed as a cross section at an arbitrary position in the OCT volume data. Further, the OCTA tomographic image may be a pseudo OCTA tomographic image reconstructed as a cross section at an arbitrary position in the OCTA volume data. It should be noted that the arbitrary position may be at least one arbitrary position, and may be configured to be changeable according to an instruction from the examiner. At this time, a plurality of pseudo tomographic images corresponding to a plurality of positions may be configured to be reconstructed.

Only one tomographic image (for example, OCT tomographic image or OCTA tomographic image) may be displayed, or a plurality of tomographic images may be displayed. When a plurality of tomographic images are displayed, the tomographic images acquired at positions in different sub-scanning directions may be displayed, or the plurality of tomographic images obtained by, for example, cross-scanning may be displayed in high quality. When displaying, images in different scanning directions may be displayed respectively. Further, when displaying a plurality of tomographic images obtained by, for example, a radial scan with high image quality, a plurality of partially selected tomographic images (for example, two tomographic images at positions symmetrical with respect to a reference line) are displayed. ) May be displayed respectively. Furthermore, a plurality of tomographic images are displayed on the follow-up display screen (follow-up display screen), and instructions for improving the image quality and analysis results (for example, the thickness of a specific layer, etc.) are displayed by the same method as the above method. ) May be displayed. At this time, the plurality of tomographic images displayed may be a plurality of tomographic images obtained at different dates and times at a predetermined site of the eye to be inspected, or a plurality of tomographic images obtained at different times on the same inspection day. May be good. Further, the tomographic image may be subjected to high image quality processing based on the information stored in the database by the same method as the above method.

Similarly, when displaying an SLO image with high image quality, for example, the SLO image displayed on the same display screen may be displayed with high image quality. Further, when the front image of the brightness is displayed with high image quality, for example, the front image of the brightness displayed on the same display screen may be displayed with high image quality. Further, a plurality of SLO images and frontal images of luminance are displayed on the display screen for follow-up observation, and instructions for improving the image quality and analysis results (for example, the thickness of a specific layer) are displayed by the same method as the above method. May be done. Further, the high image quality processing may be executed on the SLO image or the front image of the brightness based on the information stored in the database by the same method as the above method. The display of the tomographic image, the SLO image, and the frontal image of the luminance is an example, and these images may be displayed in any manner depending on the desired configuration. Further, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the brightness front image may be displayed with high image quality by one instruction.

With such a configuration, the display control unit 101-5 can display the high-quality image obtained by the high-quality processing on the output unit 103. If at least one of a plurality of conditions relating to the display of a high-quality image, the display of the analysis result, the depth range of the displayed front image, etc. is selected, the selection is made even if the display screen is changed. It may be configured to maintain the specified conditions. The display control unit 101-5 may control the display of various high-quality images, the above lines, information indicating the blood vessel region, and the like.

Further, the high image quality model may be used for at least one frame of the live moving image on the preview screen displayed on the output unit 103 by the display control unit 101-5. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the trained model corresponding to each live moving image may be used. For example, as for the anterior eye image used for the alignment process, an image having a high image quality using a high image quality model for the anterior eye image may be used. Similarly, for various images used for the detection process of a predetermined region in various images, an image whose image quality has been improved by using a high image quality model for each image may be used.

At this time, for example, when the high image quality button is pressed in response to an instruction from the examiner, a plurality of live moving images of different types (for example, anterior eye image, SLO image, tomographic image) are displayed (for example). At the same time), it may be configured to be changed to display a high-quality moving image obtained by performing high-quality processing. At this time, the display of the high-quality moving image may be a continuous display of the high-quality image obtained by performing high-quality processing on each frame. Further, for example, since the feature amount of the image may be different for each type of image, a trained model for high image quality corresponding to each type of the target image for high image quality processing may be used. For example, when the high image quality button is pressed in response to an instruction from the examiner, not only the high image quality processing of the anterior eye image is performed using the high image quality model corresponding to the anterior eye image, but also the SLO image is supported. The SLO image may also be configured to perform high image quality processing using the high image quality improvement model. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, the display is changed to the high image quality front eye image generated by using the high image quality model corresponding to the front eye image. Not only that, it may be configured to be changed to display a high-quality SLO image generated by using a high-quality model corresponding to the SLO image. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, not only the high image quality processing of the SLO image is performed using the high image quality model corresponding to the SLO image, but also the tomographic image is supported. The tomographic image may also be configured to be processed for high image quality by using the high image quality model. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, the display is simply changed to the high image quality SLO image generated by using the high image quality model corresponding to the SLO image. Instead, it may be configured to be modified to display a high quality tomographic image generated using a high image quality model corresponding to the tomographic image. At this time, the line indicating the position of the tomographic image may be configured to be superimposed and displayed on the SLO image. Further, the line may be configured to be movable on the SLO image according to an instruction from the examiner. Further, when the display of the high image quality button is in the active state, after the above line is moved, the tomographic image corresponding to the position of the current line is processed to have high image quality to obtain a high image quality tomographic image. It may be configured to change to display. Further, by displaying the image quality enhancement button for each target image of the image quality enhancement process, the image quality enhancement process may be independently enabled for each image.

As a result, for example, even if it is a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of shooting. Therefore, for example, when the operator corrects the alignment position while checking the preview screen, it is possible to reduce the failure of re-shooting and the like, so that the accuracy and efficiency of the diagnosis can be improved. Further, the image processing device 101 has described above so that a partial region such as an artifact region obtained by segmentation processing or the like is photographed (rescanned) again during or at the end of imaging in response to an instruction regarding the start of imaging. The scanning means may be driven and controlled. Depending on the movement of the eye to be inspected, it may not be possible to take a good picture by one rescan. Therefore, the drive may be controlled so that the rescan is repeated a predetermined number of times. At this time, even during a predetermined number of rescans, the rescan may be terminated in response to an instruction from the operator (for example, after the shooting cancel button is pressed). At this time, it may be configured to save the shooting data until the rescan is completed according to the instruction from the operator. For example, a confirmation dialog may be displayed after the shooting cancel button is pressed, and the shooting data may be saved or the shooting data may be discarded according to an instruction from the operator. Also, for example, after pressing the shooting cancel button, the next rescan is not executed (although the current rescan is executed until it is completed), and it waits until there is an instruction (input) from the operator in the confirmation dialog. It may be configured as follows. Further, for example, when the information indicating the certainty of the object detection result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, shooting start, etc. may be automatically performed. good. Further, for example, when the information indicating the certainty of the object detection result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, the start of imaging, etc. can be executed according to the instruction from the examiner. It may be configured to change to a new state (release the execution prohibited state).

Here, during auto-alignment, it is possible that the imaged object such as the retina of the eye 200 to be inspected has not yet been successfully imaged. Therefore, since there is a large difference between the medical image input to the trained model and the medical image used as the training data, there is a possibility that a high-quality image cannot be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B scan image) exceeds the threshold value, the display of the high-quality moving image (continuous display of high-quality frames) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image exceeds the threshold value, the image quality improvement button may be configured to be changed to a state (active state) that can be specified by the examiner. The high image quality button is a button for designating the execution of the high image quality processing. Of course, the high image quality button may be a button for instructing the display of a high image quality image.

Further, a different high image quality model may be prepared for each shooting mode having a different scan pattern or the like, and a trained model for high image quality corresponding to the selected shooting mode may be selected. Further, one high image quality model obtained by learning learning data including various medical images obtained in different imaging modes may be used.

Here, in an ophthalmic apparatus, for example, an OCT apparatus, the scan pattern of the luminous flux used for the measurement and the imaging portion differ depending on the imaging mode. Therefore, for the trained model that uses the tomographic image as input data, a trained model is prepared for each shooting mode, and the trained model corresponding to the shooting mode selected according to the instruction of the operator is selected. It may be configured. In this case, the imaging mode may include, for example, a retinal imaging mode, an anterior ocular segment imaging mode, a vitreous imaging mode, a macula imaging mode, an optic disc imaging mode, an OCTA imaging mode, and the like. Further, the scan pattern may include 3D scan, radial scan, cross scan, circle scan, raster scan, Lissajous scan (scan along the Lissajous curve) and the like. In the OCTA imaging mode, the imaging control unit 101-2 controls the scanning unit described above so that the measurement light is scanned a plurality of times in the same region (same position) of the eye to be inspected. Even in the OCTA shooting mode, for example, raster scan, radial scan, cross scan, circle scan, Lissajous scan and the like can be set as the scan pattern. Further, with respect to the trained model using the tomographic image as input data, it is possible to perform training by using the tomographic image corresponding to the cross section in different directions as the training data. For example, learning may be performed using a tomographic image of a cross section in the xz direction, a tomographic image of a cross section in the yz direction, or the like as training data.

It should be noted that the operator determines whether or not it is necessary to execute the high image quality processing by the high image quality model (or to display the high image quality image obtained by the high image quality processing) with respect to the high image quality button provided on the display screen. It may be performed according to an instruction, or may be performed according to a setting stored in advance in the storage unit 101-3. It should be noted that the fact that the high image quality processing using the trained model (high image quality improvement model) may be displayed in the active state of the high image quality improvement button, or that fact may be displayed on the display screen as a message. good. Further, the execution of the high image quality processing may maintain the execution state at the time of the previous examination of the ophthalmic apparatus, or may maintain the execution state at the time of the previous examination for each subject.

Further, the moving image to which various trained models such as a high image quality improvement model can be applied is not limited to the live moving image, and may be, for example, a moving image stored (stored) in the storage unit 101-3. At this time, for example, the moving image obtained by aligning each frame of the tomographic moving image of the fundus stored (stored) in the storage unit 101-3 may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous body, first, a reference frame based on a condition such as the presence of the vitreous body on the frame as much as possible may be selected. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) sequentially generated by the high-quality engine may be continuously displayed for each at least one frame of the moving image.

As the above-mentioned alignment method between frames, the same method may be applied to the alignment method in the X direction and the alignment method in the Z direction (depth direction), and all different methods may be applied. May be applied. Further, the alignment in the same direction may be performed a plurality of times by different methods, and for example, a precise alignment may be performed after performing a rough alignment. Further, as a method of alignment, for example, a plurality of alignments obtained by segmenting a tomographic image (coarse in the Z direction) using a retinal layer boundary obtained by performing segmentation processing of a tomographic image (B scan image) are performed. Alignment using the correlation information (similarity) between the region and the reference image (precise in the X and Z directions), and the one-dimensional projection image generated for each tomographic image (B scan image) was used (in the X direction). ) Alignment, alignment (in the X direction) using a two-dimensional front image, and the like. Further, it may be configured so that the alignment is roughly performed in pixel units and then the precise alignment is performed in sub-pixel units.

Further, the high image quality model may be updated by additional learning using the value of the ratio set (changed) according to the instruction from the examiner as the learning data. For example, if the examiner tends to set a high ratio of the input image to the high-quality image when the input image is relatively dark, the trained model will be additionally trained so as to have such a tendency. This allows, for example, to be customized as a trained model that can obtain a percentage of synthesis that suits the examiner's tastes. At this time, a button for deciding whether or not to use the set (changed) ratio value as the learning data for additional learning according to the instruction from the examiner may be displayed on the display screen. Further, the ratio determined by using the trained model may be set as the default value, and then the ratio value may be changed from the default value according to the instruction from the examiner. Further, the high image quality model may be a trained model obtained by additional learning of learning data including at least one high image quality image generated by using the high image quality model. At this time, whether or not to use the high-quality image as learning data for additional learning may be configured to be selectable according to an instruction from the examiner.

[Modification 4]
The image feature acquisition unit 101-44, the extraction unit 101-462, and the region detection unit 302 may generate a label image using a trained model for image segmentation and perform image segmentation processing. Here, the label image means a label image in which a region is labeled for each pixel of the tomographic image. Specifically, it is an image in which any area is divided by a group of identifiable pixel values (hereinafter, label value) among the area groups drawn in the acquired image. Here, the specified arbitrary region includes a region of interest, a volume of interest (VOI: Volume Of Interest), and the like.

By specifying the coordinate group of the pixel having an arbitrary label value from the image, the coordinate group of the pixel that depicts the corresponding region such as the retinal layer in the image can be specified. Specifically, for example, when the label value indicating the ganglion cell layer constituting the retina is 1, a coordinate group having a pixel value of 1 is specified among the pixel groups of the image, and the coordinate group corresponds to the coordinate group from the image. Extract the pixel group to be used. This makes it possible to identify the region of the ganglion cell layer in the image.

The image segmentation process may include a process of reducing or enlarging the label image. At this time, the image complement processing method used for reducing or enlarging the label image shall use the nearest neighbor method or the like so as not to erroneously generate an undefined label value or a label value that should not exist at the corresponding coordinates. ..

The image segmentation process is a process for specifying a region called ROI (Region Of Interest) or VOI, such as an organ or a lesion depicted in an image, for use in image diagnosis or image analysis. For example, according to the image segmentation process, it is possible to identify a region group of a group of layers constituting the retina from an image acquired by OCT imaging with the back eye portion as an imaging target. If the area to be specified is not drawn in the image, the number of specified areas is 0. Further, as long as a plurality of region groups to be specified are depicted in the image, the number of the specified regions may be a plurality, or even one region surrounding the region group so as to include the region group. good.

The specified area group is output as information that can be used in other processing. Specifically, for example, the coordinate group of the pixel group constituting each of the specified area groups can be output as a numerical data group. Further, for example, a coordinate group indicating a rectangular area, an ellipsoidal area, a rectangular parallelepiped area, an ellipsoidal area, etc. including each of the specified area groups can be output as a numerical data group. Further, for example, a coordinate group indicating a straight line, a curve, a plane, a curved surface, or the like corresponding to the boundary of the specified region group can be output as a numerical data group. Further, for example, a label image showing the specified region group can be output.

Here, as a machine learning model for image segmentation, for example, a convolutional neural network (CNN) can be used. The configuration of the CNN used in this modification is a U-net having a function of an encoder composed of a plurality of layers including a plurality of downsampling layers and a function of a decoder composed of a plurality of layers including a plurality of upsampling layers. It can be a type machine learning model. In the U-net type machine learning model, position information (spatial information) that is ambiguous in a plurality of layers configured as an encoder is displayed in the same-dimensional layer (layers corresponding to each other) in a plurality of layers configured as a decoder. ) (For example, using a skip connection).

Further, as an example of changing the configuration of the CNN, for example, a batch normalization layer or an activation layer using a normalized linear function (Rectifier Liner Unit) may be incorporated after the convolution layer. Through these steps of CNN, the features of the captured image can be extracted.

As the machine learning model according to this modification, for example, a CNN (U-net type machine learning model), a model combining CNN and LSTM, an FCN (Full Convolutional Network), a SegNet, or the like can be used. .. Further, a machine learning model or the like that performs object recognition can also be used according to a desired configuration. As a machine learning model for performing object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, a machine learning model that recognizes an object in units of regions can also be used. As a machine learning model that recognizes an object in a region unit, YOLO (You Only Look None) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used.

Further, as the training data of the machine learning model for image segmentation, the tomographic image acquired by OCT is used as input data, and the label image in which the area is labeled for each pixel of the tomographic image is used as output data. Label images include, for example, internal limiting membrane (ILM), nerve fiber layer (NFL), ganglion cell layer (GCL), photoreceptor inner segment outer segment junction (ISOS), retinal pigment epithelial layer (RPE), Bruch. Labeled images with labels such as membrane (BM) and choroid can be used. Other regions include, for example, the vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner plexiform layer (IPL), inner nuclear layer (INL), corneal membrane, anterior chamber, iris, and the like. And an image with a label such as a crystalline lens may be used.

Further, the input data of the machine learning model for image segmentation is not limited to the tomographic image. It may be an anterior eye image, an SLO image, an OCTA image, or the like. In this case, the training data can be input data of various images and output data of a label image in which a region name or the like is labeled for each pixel of the various images. For example, when the input data of the training data is an SLO image, the output data may be an image labeled with a peripheral portion of the optic disc, Disc, Cup, or the like.

The label image used as the output data may be an image in which each region is labeled in the tomographic image by a doctor or the like, or an image in which each region is labeled by the rule-based region detection process. There may be. However, if machine learning is performed using a label image that is not properly labeled as the output data of the training data, the image obtained by using the trained model trained using the training data is also properly labeled. There is a possibility that the label image will not be used. Therefore, by removing the pair containing such a label image from the training data, it is possible to reduce the possibility that an inappropriate label image is generated by using the trained model. Here, the rule-based region detection process refers to a detection process that utilizes a known regularity such as the regularity of the shape of the retina.

The image feature acquisition unit 101-44, the extraction unit 101-462, and the region detection unit 302 perform image segmentation processing using such a trained model for image segmentation to obtain a specific region for various images. It can be expected to detect at high speed and with high accuracy. A trained model for image segmentation may also be prepared for each type of various images as input data. Further, for the OCTA front image and the En-Face image, a trained model may be prepared for each depth range for generating an image. Further, the trained model for image segmentation may also be one in which learning is performed on an image for each imaging site (for example, the center of the macula and the center of the optic nerve head), or the training is performed regardless of the imaging site. You may.

Further, for the trained model for image segmentation, additional learning may be performed using the data manually modified according to the instruction of the operator as training data. Further, the determination of the necessity of additional learning and the determination of whether or not to transmit the data to the server may be performed by the same method. In these cases as well, it can be expected that the accuracy of each process can be improved and the process can be performed according to the tendency of the examiner's preference.

Further, when the image processing apparatus 101 detects a partial region of the eye to be inspected 200 (for example, a region of interest, an artifact region, an abnormal region, etc.) using the trained model, a predetermined image is determined for each detected partial region. It can also be processed. As an example, the case of detecting at least two partial regions of the vitreous region, the retinal region, and the choroid region will be described. In this case, when performing image processing such as contrast adjustment on at least two detected partial regions, adjustments suitable for each region can be performed by using different image processing parameters. By displaying the image adjusted suitable for each area, the operator can more appropriately diagnose the disease or the like in each partial area. The configuration using different image processing parameters for each detected partial region is similarly applied to the partial region of the eye 200 to be inspected obtained by detecting the partial region of the eye 200 to be inspected without using the trained model. You may.

[Modification 5]
The display control unit 101-5 in the various embodiments and modifications described above may display analysis results such as the layer thickness of a desired layer and various blood vessel densities on the report screen of the display screen after taking a tomographic image. .. In addition, the papilla of the optic nerve, the macula, the vascular region, the capillary region, the arterial region, the venous region, the nerve fiber bundle, the vitreous region, the macula region, the choroid region, the sclera region, the lamina cribrosa region, the retinal layer boundary, the retina. Parameter values (distribution) for the site of interest, including at least one of the layer boundary edges, photoreceptors, blood cells, blood vessel walls, blood vessel inner wall boundaries, blood vessel lateral boundaries, ganglion cells, corneal region, macula region, Schlemm's canal, etc. May be displayed as an analysis result. Here, the site of interest may be, for example, a vortex vein or the like, which is an outlet of a blood vessel in the Haller layer (an example of a blood vessel in a depth range of a part of the choroidal region) to the outside of the eye. At this time, the parameters related to the site of interest include, for example, the number of vortex veins (for example, the number for each region), the distance from the optic disc to each vortex vein, the angle at which each vortex vein centered on the optic nerve head is located, and the like. May be. This makes it possible to accurately diagnose, for example, various diseases related to Pachychoroid (thickened choroid) (for example, choroidal neovascularization). Further, for example, by analyzing a medical image to which various artifact reduction processes are applied, the above-mentioned various analysis results can be displayed as accurate analysis results. The artifact is, for example, a false image region generated by light absorption by a blood vessel region or the like, a projection artifact, a band-shaped artifact in a front image generated in the main scanning direction of the measured light depending on the state of the eye to be inspected (movement, blinking, etc.), or the like. There may be. Further, the artifact may be, for example, any image loss region that randomly occurs at each shooting on the medical image of the predetermined portion of the subject. Further, the display control unit 101-5 may display the value (distribution) of the parameter relating to the region including at least one of various artifacts (photo-loss regions) as described above in the output unit 103 as an analysis result. Further, the value (distribution) of the parameter relating to the region including at least one such as drusen, new blood vessel, vitiligo (hard vitiligo), and abnormal site such as pseudo-drusen may be displayed as an analysis result. Further, the comparison result obtained by comparing the standard value or standard range obtained by using the standard database with the analysis result may be displayed.

Further, the analysis result may be displayed in an analysis map, a sector showing statistical values corresponding to each divided area, or the like. The analysis result may be generated by using a trained model (analysis result generation engine, trained model for analysis result generation) obtained by learning the analysis result of the medical image as training data. .. At this time, the trained model is trained using training data including a medical image and an analysis result of the medical image, and training data including a medical image and an analysis result of a medical image of a type different from the medical image. It may be obtained by.

Further, the training data for performing image analysis may include a label image generated by using the trained model for image segmentation processing and an analysis result of a medical image using the label image. In this case, the image processing apparatus 101 may function as an example of an analysis result generation unit that generates an analysis result of a tomographic image from the result of image segmentation processing by using, for example, a trained model for generating an analysis result. can. Further, the trained model is a training data including input data including a set of a plurality of medical images of different types of predetermined parts, such as an En-Face image of luminance and an En-Face image of motion contrast (OCTA En-Face image). It may be obtained by learning using.

Further, the analysis result obtained by using the high-quality image generated by using the high-quality model may be displayed. In this case, the input data included in the training data may be a high-quality image generated by using the trained model for high image quality, or may be a set of a low-quality image and a high-quality image. May be good. The training data may be an image in which at least a part of an image whose image quality has been improved by using the trained model has been manually or automatically modified.

Further, the training data includes, for example, at least the analysis value (for example, average value, median value, etc.) obtained by analyzing the analysis area, the table including the analysis value, the analysis map, the position of the analysis area such as the sector in the image, and the like. The information including one may be the data labeled (annotated) with the input data as the correct answer data (for supervised learning). It should be noted that the analysis result obtained by using the trained model for generating the analysis result may be displayed according to the instruction from the operator.

In addition, the display control unit 101-5 in the above-described embodiments and modifications may display various diagnostic results such as diabetic retinopathy, glaucoma, and age-related macular degeneration on the report screen of the display screen. At this time, for example, by analyzing the medical image to which the various artifact reduction processes as described above are applied, it is possible to display an accurate diagnostic result. Further, the diagnosis result may be displayed on the image at the position of the specified abnormal part or the like, or the state of the abnormal part or the like may be displayed by characters or the like. Further, the classification result of the abnormal part or the like (for example, Cartin classification) may be displayed as the diagnosis result. Further, as the classification result, for example, information indicating the certainty of each abnormal part (for example, a numerical value indicating a ratio) may be displayed. In addition, information necessary for the doctor to confirm the diagnosis may be displayed as a diagnosis result. As the necessary information, for example, advice such as additional shooting can be considered. For example, when an abnormal site is detected in the blood vessel region in the OCTA image, it may be displayed that fluorescence imaging using a contrast medium capable of observing the blood vessel in more detail than OCTA is performed. In addition, the diagnosis result may be information on the future medical treatment policy of the subject. In addition, the diagnosis result is, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Medical support information, etc.) and grounds for denying the diagnosis name (negative medical support information, etc.) may be included in the information. At this time, for example, a diagnosis result that is more probable than the diagnosis result such as a diagnosis name input in response to an instruction from the examiner may be displayed as medical support information. Further, when a plurality of types of medical images are used, for example, the types of medical images that can be the basis of the diagnosis result may be displayed in an identifiable manner. The basis of the diagnosis result is a map (attention map, activation map) that visualizes the features extracted by the trained model, for example, a color map (heat map) that shows the features in color. good. At this time, for example, the heat map may be superimposed and displayed on the medical image used as the input data. The heat map is, for example, a method for visualizing a region (a region having a large gradient) that contributes greatly to the output value of the predicted (estimated) class, such as Grad-CAM (Gradient-weighted Class Activity Mapping) or Guided Grade. -Can be obtained using CAM or the like.

The diagnosis result may be generated by using a trained model (diagnosis result generation engine, trained model for generation of diagnosis result) obtained by learning the diagnosis result of the medical image as training data. .. Further, the trained model is based on training using training data including a medical image and a diagnosis result of the medical image, and training data including a medical image and a diagnosis result of a medical image of a type different from the medical image. It may be obtained.

Further, the training data may include a label image generated by using a region recognition engine or a trained model for segmentation processing, and a diagnostic result of a medical image using the label image. In this case, the image processing apparatus 101 may function as an example of a diagnostic result generation unit that generates a diagnostic result of a tomographic image from the result of image segmentation processing by using, for example, a trained model for generating a diagnostic result. can.

Further, it may be configured to display the diagnostic result obtained by using the high-quality image generated by using the high-quality engine. In this case, the input data included in the learning data may be a high-quality image generated by using the high-quality image engine, or may be a set of a low-quality image and a high-quality image. The training data may be an image in which at least a part of an image whose image quality has been improved by using the trained model has been manually or automatically modified.

In addition, the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Information including at least one such as (general medical support information, etc.) and grounds for denying the diagnosis name (negative medical support information), etc. are labeled (annotated) as correct answer data (for supervised learning). Data may be used. In addition, according to the instruction from the examiner, the diagnosis result obtained by using the trained model for generating the diagnosis result may be displayed.

A trained model may be prepared for each information used as input data or for each type of information, and the diagnosis result may be acquired using the trained model. In this case, the information output from each trained model may be statistically processed to determine the final diagnostic result. For example, the ratio of the information output from each trained model may be added for each type of information, and the information having a higher total ratio than the other information may be determined as the final diagnosis result. Note that the statistical processing is not limited to the calculation of the total, and may be the calculation of the average value, the median value, or the like. Further, for example, among the information output from each trained model, the information having a higher ratio than the other information (the information having the highest ratio) may be used to determine the diagnosis result. Similarly, the diagnosis result may be determined using the information of the ratio of the information output from each trained model that is equal to or higher than the threshold value.

Further, it may be configured so that the quality of the determined diagnosis result can be determined (approved) according to the instruction (selection) of the operator. Further, the diagnosis result may be determined from the information output from each trained model according to the instruction (selection) of the operator. At this time, for example, the display control unit 101-5 may display the information output from each trained model and the ratio thereof side by side on the output unit 103. Then, the operator may be configured to determine the selected information as a diagnosis result, for example, by selecting information having a higher ratio than other information. Further, the diagnosis result may be determined by using the machine learning model from the information output from each trained model. In this case, the machine learning algorithm may be a different type of machine learning algorithm from the machine learning algorithm used to generate the diagnostic result, for example, a neural network, a support vector machine, an adaboost, a Basian network, or a random. A forest or the like may be used.

The learning of the various trained models described above may be not only supervised learning (learning with labeled learning data) but also semi-supervised learning. In semi-supervised learning, for example, after multiple classifiers (classifiers) perform supervised learning, unlabeled learning data is identified (classified), and the identification result (classification result) is determined according to the reliability. It is a method of automatically labeling (annotating) (for example, an identification result whose certainty is equal to or higher than the threshold value) and learning with the labeled learning data. Semi-supervised learning may be, for example, co-training (or Multiview). At this time, the trained model for generating the diagnosis result uses, for example, a first discriminator that identifies a medical image of a normal subject and a second discriminator that identifies a medical image containing a specific lesion. It may be a trained model obtained by semi-supervised learning (for example, co-training). It should be noted that the purpose is not limited to diagnostic purposes, but may be, for example, photography support or the like. In this case, the second classifier may, for example, identify a medical image including a partial region such as a region of interest or an artifact region.

Further, the display control unit 101-5 according to the various embodiments and modifications described above may display an object recognition result of a partial region such as a region of interest, an artifact region, and an abnormal region as described above on the report screen of the display screen. The object detection result) and the segmentation result may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, colors and the like may be superimposed and displayed on an object in an image. The object recognition result and segmentation result are trained models (object recognition engine, for object recognition) obtained by learning learning data that labels (annotifies) medical images with information indicating object recognition and segmentation as correct answer data. It may be generated using a trained model, a segmentation engine, a trained model for segmentation). The above-mentioned analysis result generation and diagnosis result generation may be obtained by using the above-mentioned object recognition result and segmentation result. For example, analysis result generation or diagnosis result generation processing may be performed on a region of interest obtained by object recognition or segmentation processing.

Further, when detecting an abnormal portion, the image processing apparatus 101 may use a hostile generation network (GAN: Generative Adversarial Networks) or a variational autoencoder (VAE: Variational Auto-Encoder). For example, a DCGAN (Deep Convolutional GAN) consisting of a generator obtained by learning the generation of a medical image and a classifier obtained by learning the discrimination between a new medical image generated by the generator and a real medical image. Can be used as a machine learning model.

When DCGAN is used, for example, the classifier encodes the input medical image into a latent variable, and the generator generates a new medical image based on the latent variable. After that, the difference between the input medical image and the generated new medical image can be extracted (detected) as an abnormal part. When VAE is used, for example, the input medical image is encoded by an encoder to be a latent variable, and the latent variable is decoded by a decoder to generate a new medical image. After that, the difference between the input medical image and the generated new medical image can be extracted as an abnormal part.

Further, the image processing apparatus 101 may detect an abnormal portion by using a convolutional autoencoder (CAE). When CAE is used, the same medical image is learned as input data and output data at the time of learning. As a result, when a medical image having an abnormal part is input to CAE at the time of estimation, a medical image having no abnormal part is output according to a learning tendency. After that, the difference between the medical image input to the CAE and the medical image output from the CAE can be extracted as an abnormal portion.

In these cases, the image processing apparatus 101 obtains information on the difference between the medical image obtained by using the hostile generation network or the auto-encoder and the medical image input to the hostile generation network or the auto-encoder, and information on the abnormal portion. Can be generated as. As a result, the image processing apparatus 101 can be expected to detect the abnormal portion at high speed and with high accuracy. For example, even if it is difficult to collect many medical images including abnormal parts as learning data in order to improve the detection accuracy of abnormal parts, a relatively large number of normal medical images of subjects that are easy to collect are used as learning data. Can be used. Therefore, for example, learning for accurately detecting an abnormal portion can be performed efficiently. Here, the autoencoder includes VAE, CAE, and the like. Further, at least a part of the generation part of the hostile generation network may be composed of VAE. As a result, for example, it is possible to generate a relatively clear image while reducing the phenomenon of generating similar data. For example, the image processing apparatus 101 obtains information regarding the difference between a medical image obtained from various medical images using a hostile generation network or an auto encoder and a medical image input to the hostile generation network or the auto encoder. , Can be generated as information about the abnormal part. Further, for example, the display control unit 101-5 sets the medical image obtained from various medical images by using the hostile generation network or the autoencoder, and the medical image input to the hostile generation network or the autoencoder. The information regarding the difference can be displayed on the output unit 103 as information regarding the abnormal portion.

Further, in particular, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a set of a plurality of medical images of different types of the predetermined part of the subject. good. At this time, as the input data included in the training data, for example, input data in which a motion contrast front image and a luminance front image (or a luminance tomographic image) of the fundus are set can be considered. Further, as the input data included in the training data, for example, input data in which a tomographic image (B scan image) of the fundus and a color fundus image (or a fluorescent fundus image) are set can be considered. Further, the plurality of medical images of different types may be anything as long as they are acquired by different modality, different optical systems, different principles, or the like.

Further, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a plurality of medical images of different parts of the subject. At this time, as the input data included in the training data, for example, input data in which a tomographic image of the fundus (B scan image) and a tomographic image of the anterior segment of the eye (B scan image) are considered as a set can be considered. Further, as the input data included in the training data, for example, input data in which a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic nerve head of the fundus are set as a set, etc. Is also possible.

The input data included in the learning data may be different parts of the subject and a plurality of different types of medical images. At this time, the input data included in the training data may be, for example, input data in which a tomographic image of the anterior segment of the eye and a color fundus image are set. Further, the trained model described above may be a trained model obtained by learning from training data including input data including a set of a plurality of medical images having different shooting angles of view of a predetermined portion of the subject. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a predetermined portion into a plurality of regions, such as a panoramic image. At this time, by using a wide angle of view image such as a panoramic image as training data, there is a possibility that the feature amount of the image can be accurately acquired because the amount of information is larger than that of the narrow angle of view image. The result of can be improved. Further, the input data included in the learning data may be input data in which a plurality of medical images of different dates and times of a predetermined part of the subject are set.

Further, the display screen on which at least one of the above-mentioned analysis result, diagnosis result, object recognition result, and segmentation result is displayed is not limited to the report screen. Such a display screen is, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). It may be displayed in. For example, by displaying at least one result obtained by using the above-mentioned trained model on the shooting confirmation screen, the operator can confirm the accurate result even immediately after shooting.

Further, for example, when a specific object is recognized, a frame surrounding the recognized object may be configured to be superimposed and displayed on the live moving image. At this time, if the information indicating the certainty of the object recognition result (for example, a numerical value indicating the ratio) exceeds the threshold value, the color of the frame surrounding the object may be changed or highlighted. good. This allows the examiner to easily identify the object on the live video.

In addition, in the generation of the correct answer data used for learning the various trained models described above, the trained model for generating the correct answer data for generating the correct answer data such as labeling (annotation) may be used. At this time, the trained model for generating correct answer data may be obtained by (sequentially) additional learning of the correct answer data obtained by labeling (annotation) by the examiner. That is, the trained model for generating correct answer data may be obtained by additional training of training data in which the data before labeling is used as input data and the data after labeling is used as output data. Further, in a plurality of continuous frames such as a moving image, the result of the frame judged to have low accuracy is corrected in consideration of the results of object recognition and segmentation of the preceding and following frames. May be good. At this time, according to the instruction from the examiner, the corrected result may be additionally learned as correct answer data. Further, for example, for a medical image with low accuracy of the result, the feature amount is an example of a map (attention map, activation map) in which the examiner visualizes the feature amount extracted by the trained model on the medical image. It may be configured to additionally learn an image labeled (annotated) as input data while checking a color map (heat map) showing the above in color. For example, in a heat map on a layer just before outputting the result in the trained model, if the part to be noted is different from the intention of the examiner, the part to be noted by the examiner is labeled (annotation). You may additionally learn the medical image. Thereby, for example, the trained model is a partial region on the medical image, and the feature amount of the partial region having a relatively large influence on the output result of the trained model is prioritized over the other regions ( Additional learning can be done (with weighting).

Here, the various trained models described above can be obtained by machine learning using the training data. Machine learning includes, for example, deep learning consisting of a multi-layered neural network. Further, for example, a convolutional neural network can be used for at least a part of the multi-layered neural network. Further, a technique related to an autoencoder (self-encoder) may be used for at least a part of a multi-layered neural network. Further, a technique related to backpropagation (error backpropagation method) may be used for learning. Further, for learning, a method (dropout) in which each unit (each neuron or each node) is randomly inactivated may be used. Further, for learning, a method (batch normalization) may be used in which the data transmitted to each layer of the multi-layer neural network is normalized before the activation function (for example, the ReLu function) is applied. However, the machine learning is not limited to deep learning, and any learning using a model capable of extracting (expressing) the features of learning data such as images by learning may be used. Here, the machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. The trained model is a model in which a machine learning model by an arbitrary machine learning algorithm is trained (learned) in advance using appropriate learning data. However, the trained model does not require further training, and additional training can be performed. The learning data is composed of a pair of input data and output data (correct answer data). Here, the learning data may be referred to as teacher data, or the correct answer data may be referred to as teacher data.

The GPU can perform efficient operations by processing more data in parallel. Therefore, when learning is performed a plurality of times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in this modification, a GPU is used in addition to the CPU for processing by the image processing device 101, which is an example of the learning unit (not shown). Specifically, when a learning program including a learning model is executed, learning is performed by the CPU and the GPU collaborating to perform calculations. In addition, the processing of the learning unit may be performed only by the CPU or the GPU. Further, the processing unit (estimation unit) that executes the processing using the various trained models described above may also use the GPU in the same manner as the learning unit. Further, the learning unit may include an error detection unit and an update unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error back propagation method is a method of adjusting the coupling weighting coefficient and the like between the nodes of each neural network so that the above error becomes small.

Further, as the machine learning model used for the above-mentioned object recognition, segmentation, high image quality, etc., the function of the encoder consisting of a plurality of layers including a plurality of downsampling layers and the function of a plurality of layers including a plurality of upsampling layers A U-net type machine learning model having a decoder function consisting of is applicable. In the U-net type machine learning model, position information (spatial information) that is ambiguous in a plurality of layers configured as an encoder is displayed in the same-dimensional layer (layers corresponding to each other) in a plurality of layers configured as a decoder. ) (For example, using a skip connection).

Further, as the machine learning model used for the above-mentioned object recognition, segmentation, high image quality, etc., for example, FCN (Full Convolutional Network), SegNet, or the like can be used. Further, a machine learning model that recognizes an object in a region unit according to a desired configuration may be used. As a machine learning model for performing object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Further, as a machine learning model for recognizing an object in a region unit, YOLO (You Only Look None) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used.

Further, the machine learning model may be, for example, a capsule network (Capsule Network; CapsNet). Here, in a general neural network, each unit (each neuron or each node) is configured to output a scalar value, for example, a spatial positional relationship (relative position) between features in an image. It is configured to reduce spatial information about. Thereby, for example, learning can be performed so as to reduce the influence of local distortion, translation, and the like of the image. On the other hand, in the capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is retained. This makes it possible to perform learning in which, for example, the spatial positional relationship between features in an image is taken into consideration.

[Modification 6]
In the preview screens in the various embodiments and modifications described above, the various trained models described above may be configured to be used for at least one frame of the live moving image. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the trained model corresponding to each live moving image may be used. As a result, for example, even if it is a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of shooting. Therefore, for example, it is possible to reduce the failure of re-imaging, and thus it is possible to improve the accuracy and efficiency of diagnosis.

The plurality of live moving images may be, for example, a moving image of the anterior eye portion for alignment in the XYZ direction, and a frontal moving image of the fundus for focusing adjustment or OCT focus adjustment of the fundus observation optical system. Further, the plurality of live moving images may be, for example, a tomographic moving image of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measured optical path length and the reference optical path length). When such a preview image is displayed, the above-mentioned various adjustments are performed so that the region detected by using the above-mentioned trained model for object recognition and the above-mentioned trained model for segmentation satisfies a predetermined condition. The image processing device 101 may be configured as described above. For example, a value (for example, a contrast value or an intensity value) related to a predetermined retinal layer such as a vitreous region or an RPE detected by using a trained model for object recognition or a trained model for segmentation exceeds a threshold value (for example, a contrast value or an intensity value). Alternatively, it may be configured to perform various adjustments such as OCT focus adjustment so as to reach a peak value). Further, for example, the OCT so that a predetermined retinal layer such as a vitreous region or RPE detected by using a trained model for object recognition or a trained model for segmentation is at a predetermined position in the depth direction. Coherence gate adjustments may be configured to be made.

In these cases, the image processing device 101 can generate a high-quality moving image by performing high-quality processing on the moving image using the trained model. Further, the photographing control unit 101-2 of the reference mirror 221 so that the partial area such as the attention portion obtained by the segmentation process or the like is at a predetermined position in the display area while the high-quality moving image is displayed. The optical member for changing the shooting range can be driven and controlled. In such a case, the photographing control unit 101-2 can automatically perform the alignment process so that the desired area becomes a predetermined position in the display area based on the highly accurate information. The optical member for changing the photographing range may be, for example, an optical member for adjusting the coherence gate position, and specifically, a reference mirror 221 for reflecting reference light. Further, the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measured optical path length and the reference optical path length, and the optical member changes, for example, the optical path length of the measurement light (not shown). It may be a mirror or the like. The optical member for changing the photographing range may be, for example, the stage unit 100-2. Further, in response to an instruction regarding the start of shooting by the shooting control unit 101-2, scanning is performed so that a partial region such as an artifact region obtained by segmentation processing or the like is shot (rescanned) again during or at the end of shooting. The means may be driven and controlled. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, various adjustments, shooting start, etc. may be automatically performed. good. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, the start of imaging, etc. can be executed according to the instruction from the examiner. It may be configured to change to a new state (release the execution prohibited state).

Further, the moving image to which the various trained models described above can be applied is not limited to the live moving image, and may be, for example, a moving image stored (stored) in the storage unit 101-3. At this time, for example, the moving image obtained by aligning each frame of the tomographic moving image of the fundus stored (stored) in the storage unit 101-3 may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous body, first, a reference frame based on a condition such as the presence of the vitreous body on the frame as much as possible may be selected. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) sequentially generated by the trained model for high image quality may be continuously displayed for each at least one frame of the moving image.

As the above-mentioned alignment method between frames, the same method may be applied to the alignment method in the X direction and the alignment method in the Z direction (depth direction), or all different methods may be applied. May be applied. Further, the alignment in the same direction may be performed a plurality of times by different methods, and for example, a precise alignment may be performed after performing a rough alignment. Further, as a method of alignment, for example, a plurality of alignments obtained by segmenting a tomographic image (coarse in the Z direction) using a retinal layer boundary obtained by performing segmentation processing of a tomographic image (B scan image) are performed. Alignment using the correlation information (similarity) between the region and the reference image (precise in the X and Z directions), and the one-dimensional projection image generated for each tomographic image (B scan image) was used (in the X direction). ) Alignment, alignment (in the X direction) using a two-dimensional front image, and the like. Further, it may be configured so that the alignment is roughly performed in pixel units and then the precise alignment is performed in sub-pixel units.

Here, during various adjustments, it is possible that the imaged object such as the retina of the eye to be inspected has not yet been successfully imaged. Therefore, since there is a large difference between the medical image input to the trained model and the medical image used as the training data, there is a possibility that a high-quality image cannot be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the display of the high-quality moving image (continuous display of high-quality frames) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the image quality enhancement button may be configured to be changed to a state (active state) that can be specified by the examiner.

Further, for example, a trained model for high image quality is prepared for each shooting mode having a different scan pattern, and the trained model for high image quality corresponding to the selected shooting mode is selected. May be done. Further, one trained model for high image quality obtained by learning training data including various medical images obtained in different imaging modes may be used.

[Modification 7]
In the above-described embodiment and modification, when various trained models are executing additional learning, it may be difficult to output (estimate / predict) using the trained model itself during additional learning. be. Therefore, it may be configured to prohibit the input of medical images other than the training data to the trained model during the execution of additional learning. Further, another trained model that is the same as the trained model before the execution of the additional training may be prepared as another preliminary trained model. At this time, during the execution of the additional learning, it may be configured so that the input of the medical image other than the training data to the preliminary trained model can be executed. Then, after the additional learning is completed, the trained model after the additional learning is executed may be evaluated, and if there is no problem, the preliminary trained model may be replaced with the trained model after the additional learning is executed. Also, if there is a problem, a preliminary trained model may be used.

As an evaluation of the trained model after the execution of additional learning, for example, a trained model for classification for classifying a high-quality image obtained by the trained model for high image quality from another type of image is used. It may be used. The trained model for classification uses, for example, a plurality of images including a high-quality image and a low-quality image obtained by the trained model for high image quality as input data, and the types of these images are labeled (annotation). It may be a trained model obtained by training training data including the obtained data as correct answer data. At this time, the image type of the input data at the time of estimation (prediction) is displayed together with the information indicating the certainty of each type of image included in the correct answer data at the time of learning (for example, a numerical value indicating a ratio). May be good. In addition to the above images, the input data of the trained model for classification includes overlay processing of a plurality of low-quality images (for example, averaging processing of a plurality of low-quality images obtained by alignment) and the like. It may include high-quality images such as those with high contrast and noise reduction. Further, as the evaluation of the trained model after the execution of the additional learning, for example, the trained model after the execution of the additional learning and the trained model before the execution of the additional learning (preliminary trained model) are used and the same. A plurality of high-quality images obtained from the above images may be compared, or the analysis results of the plurality of high-quality images may be compared. At this time, for example, the comparison result of the plurality of high-quality images (an example of change due to additional learning) or the comparison result of the analysis result of the plurality of high-quality images (an example of change due to additional learning) is within a predetermined range. It may be determined whether or not, and the determination result may be displayed.

In addition, the trained model obtained by learning for each imaging site may be selectively used. Specifically, the first trained model obtained by using the learning data including the first imaged part (for example, the anterior eye part, the posterior eye part, etc.) and the second imaged part different from the first imaged part. It is possible to prepare a second trained model obtained by using the training data including the imaged portion, and a plurality of trained models including the trained model. Then, the image processing device 101 may have a selection means for selecting one of the plurality of trained models. At this time, the image processing device 101 may have a control means for executing additional learning on the selected trained model. The control means searches for data in which the captured part corresponding to the selected trained model and the captured image of the captured part are paired in response to an instruction from the examiner, and the data obtained by the search is learned data. Can be executed as additional training for the selected trained model. The imaged portion corresponding to the selected trained model may be acquired from the information in the header of the data or manually input by the examiner. Further, the data can be searched for, for example, from a server of an external facility such as a hospital or a research institute via a network. As a result, additional learning can be efficiently performed for each imaged part by using the photographed image of the imaged part corresponding to the trained model.

The selection means and the control means may be composed of a software module executed by a processor such as a CPU or an MPU of the image processing device 101. Further, the selection means and the control means may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like.

In addition, when learning data for additional learning is acquired from a server of an external facility such as a hospital or research institute via a network, it is necessary to reduce reliability deterioration due to falsification or system trouble during additional learning. Is useful. Therefore, the legitimacy of the learning data for additional learning may be detected by confirming the consistency by digital signature or hashing. This makes it possible to protect the learning data for additional learning. At this time, if the validity of the training data for additional learning cannot be detected as a result of confirming the consistency by digital signature or hashing, a warning to that effect is given and additional learning is performed using the training data. Make it not exist. The server may be in any form, for example, a cloud server, a fog server, an edge server, or the like, regardless of its installation location. In addition, when the network in the facility, the site including the facility, the area including multiple facilities, etc. is configured to enable wireless communication, for example, it is assigned only to the facility, the site, the area, etc. The reliability of the network may be improved by configuring so as to use radio waves in a dedicated wavelength band. Further, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and simultaneous connection of a large number of people.

Further, the data protection by confirming the consistency as described above can be applied not only to the learning data for additional learning but also to the data including the medical image. Further, the image management system may be configured so that the transaction of data including medical images between the servers of a plurality of facilities is managed by a decentralized network. Further, the image management system may be configured to connect a plurality of blocks in which the transaction history and the hash value of the previous block are recorded together in chronological order. As a technique for confirming consistency, even if cryptography that is difficult to calculate even using a quantum computer such as a quantum gate method (for example, lattice cryptography, quantum cryptography by quantum key distribution, etc.) is used. good. Here, the image management system may be a device and a system that receives and stores an image taken by a photographing device or an image processed image. In addition, the image management system may send an image in response to a request from the connected device, perform image processing on the saved image, or request another device to perform image processing. can. The image management system can include, for example, an image storage communication system (PACS). In addition, the image management system includes a database that can store various information such as information on the subject and shooting time associated with the received image. In addition, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with saved images in response to requests from other devices. ..

When additional learning is performed on various trained models, GPU can be used to perform high-speed processing. Since the GPU can perform efficient operations by processing more data in parallel, it is possible to perform processing on the GPU when learning is performed multiple times using a learning model such as deep learning. It is valid. The GPU, CPU, and the like may cooperate in the additional learning process.

[Modification 8]
In the various embodiments and modifications described above, the instruction from the examiner may be an instruction by voice or the like in addition to the instruction by manual operation (for example, the instruction using the user interface or the like). At this time, for example, a machine learning model including a voice recognition model (speech recognition engine, trained model for voice recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by character input or the like using a keyboard, a touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (character recognition engine, trained model for character recognition) obtained by machine learning may be used. Further, the instruction from the examiner may be an instruction by a gesture or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, trained model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be the line-of-sight detection result of the examiner on the display screen of the output unit 103. The line-of-sight detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from the periphery of the display screen in the output unit 103. At this time, the object recognition engine as described above may be used for the pupil detection from the moving image. Further, the instruction from the examiner may be an instruction by an electroencephalogram, a weak electric signal flowing through the body, or the like.

In such a case, for example, as the training data, character data or voice data (waveform data) indicating an instruction for displaying the result by processing of various trained models as described above is used as input data, and various trained data have been trained. It may be learning data in which the execution instruction for actually displaying the result or the like obtained by the processing of the model on the output unit 103 is the correct answer data. Further, the learning data may be, for example, learning data in which an execution command for whether or not to automatically set shooting parameters and an execution command for changing the button for the command to the active state are correct data. good. Further, as the learning data, for example, an execution command for whether or not to perform high image quality processing in order to obtain a high image quality image, an execution command for changing the button for the command to the active state, and the like are used as correct answer data. It may be training data. The learning data may be any data as long as the instruction content and the execution command content indicated by the character data, the voice data, or the like correspond to each other. Further, the voice data may be converted into character data by using an acoustic model, a language model, or the like. Further, the waveform data obtained by the plurality of microphones may be used to perform a process of reducing the noise data superimposed on the voice data. Further, the instruction by characters or voice and the instruction by a mouse or a touch panel may be configured to be selectable according to the instruction from the examiner. Further, the on / off of the instruction by characters or voice may be selectably configured according to the instruction from the examiner.

Here, machine learning includes deep learning as described above, and RNN can be used, for example, for at least a part of a multi-layer neural network. Here, as an example of the machine learning model according to this modification, RNN, which is a neural network that handles time-series information, will be described with reference to FIGS. 15 (a) and 15 (b). Further, a Long short-term memory (hereinafter referred to as LSTM), which is a kind of RNN, will be described with reference to FIGS. 16 (a) and 16 (b).

FIG. 15A shows the structure of the RNN, which is a machine learning model. The RNN 1520 has a loop structure in the network, data x ^t 1510 is input at time t, and data h ^t 1530 is output. Since the RNN1520 has a loop function in the network, the current state can be inherited to the next state, so that time-series information can be handled. FIG. 15B shows an example of input / output of the parameter vector at time t. The data x ^t 1510 contains N pieces of data (Params1 to ParamsN). Further, the data h ^t 1530 output from the RNN 1520 includes N pieces of data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information during error back propagation, LSTM may be used. The LSTM can learn long-term information by including a forgetting gate, an input gate, and an output gate. Here, FIG. 16A shows the structure of the LSTM. In LSTM1640, the information that the network takes over at the next time t is the internal state ct ^-1 of the network called the cell and the output data h ^t-1 . The lowercase letters (c, h, x) in the figure represent vectors.

Next, FIG. 16 (b) shows the details of LSTM1640. In FIG. 16B, the oblivion gate network FG, the input gate network IG, and the output gate network OG are shown, each of which is a sigmoid layer. Therefore, a vector in which each element has a value of 0 to 1 is output. The oblivion gate network FG determines how much past information is retained, and the input gate network IG determines which value to update. Further, in FIG. 16B, the cell update candidate network CU is shown, and the cell update candidate network CU is the activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gate network OG selects the element of the cell candidate and selects how much information is to be transmitted at the next time.

Since the above-mentioned LSTM model is a basic model, it is not limited to the network shown here. You may change the coupling between the networks. QRNN (Quasi Recurrent Neural Network) may be used instead of LSTM. Further, the machine learning model is not limited to the neural network, and boosting, a support vector machine, or the like may be used. Further, when the instruction from the examiner is input by characters, voice, or the like, a technique related to natural language processing (for example, Sequence to Sequence) may be applied. At this time, as a technique related to natural language processing, for example, a model that is output for each input sentence may be applied. Further, the various trained models described above are not limited to the instructions from the examiner, and may be applied to the output to the examiner. Further, a dialogue engine (dialogue model, trained model for dialogue) that responds to the examiner with output by characters, voice, or the like may be applied.

Further, as a technique related to natural language processing, a trained model obtained by pre-learning document data by unsupervised learning may be used. Further, as a technique related to natural language processing, a trained model obtained by transfer learning (or fine tuning) of a trained model obtained by pre-learning may be used. Further, as a technique related to natural language processing, for example, BERT (Bidirectional Encoder Representations from Transfermers) may be applied. Further, as a technique related to natural language processing, a model capable of extracting (expressing) a context (feature amount) by itself by predicting a specific word in a sentence from both the left and right contexts may be applied. Further, as a technique related to natural language processing, a model capable of determining the relationship (continuity) between two sequences (sentences) in the input time series data may be applied. Further, as a technique related to natural language processing, a Transformer Encoder is used for the hidden layer, and a model in which a sequence of vectors is input and output may be applied.

Here, the instruction from the examiner to which this modification is applicable is for changing the display of various images and analysis results as described in the various embodiments and modifications described above, and for generating an En-Face image. Selection of depth range, selection of whether to use as training data for additional learning, selection of trained model, output (display, transmission, etc.) and storage of results obtained using various trained models, etc. Any instruction may be used as long as it is at least one instruction. Further, the instruction from the examiner to which this modification is applicable may be an instruction before photography as well as an instruction after photography, for example, an instruction regarding various adjustments and an instruction regarding setting of various imaging conditions. , It may be an instruction regarding the start of shooting. Further, the instruction from the examiner to which this modification is applicable may be an instruction regarding a change (screen transition) of the display screen.

The machine learning model may be a machine learning model that combines a machine learning model related to images such as CNN and a machine learning model related to time series data such as RNN. In such a machine learning model, for example, it is possible to learn the relationship between the feature amount related to an image and the feature amount related to time series data. When the input layer side of the machine learning model is CNN and the output layer side is RNN, for example, the medical image is used as the input data, and the text related to the medical image (for example, the presence or absence of a lesion, the type of lesion, and the recommendation of the next examination). Etc.) may be used as output data for training. As a result, for example, medical information related to medical images is automatically explained in sentences, so that even an examiner with little medical experience can easily grasp medical information related to medical images. When the input layer side of the machine learning model is RNN and the output layer side is CNN, for example, texts related to medical treatment such as lesions, findings, and diagnoses are used as input data, and medical images corresponding to the texts related to the medical treatment are output. Learning may be performed using the learning data as data. This makes it possible, for example, to easily search for a medical image related to a case that the examiner wants to confirm.

In addition, even if a machine translation engine (machine translation model, trained model for machine translation) that machine translates sentences such as characters and voices into any language is used for instructions from the examiner and output to the examiner. good. In addition, any language may be configured to be selectable according to an instruction from the examiner. Further, any language may be configured to be automatically selectable by using a trained model that automatically recognizes the type of language. Further, the automatically selected language type may be configured to be modifiable according to an instruction from the examiner. For example, the above-mentioned techniques related to natural language processing (for example, Sequence to Sequence) may be applied to the machine translation engine. For example, after the sentence input to the machine translation engine is machine-translated, the machine-translated sentence may be input to the character recognition engine or the like. Further, for example, the sentences output from the various trained models described above may be input to the machine translation engine, and the sentences output from the machine translation engine may be output.

Further, the various trained models described above may be used in combination. For example, the character corresponding to the instruction from the examiner is input to the character recognition engine, and the voice obtained from the input character is input to another type of machine learning engine (for example, a machine translation engine). May be done. Further, for example, characters output from other types of machine learning engines may be input to the character recognition engine, and the voice obtained from the input characters may be output. Further, for example, the voice corresponding to the instruction from the examiner is input to the voice recognition engine, and the characters obtained from the input voice are input to another type of machine learning engine (for example, a machine translation engine). It may be configured in. Further, for example, the voice output from another type of machine learning engine may be input to the voice recognition engine, and the characters obtained from the input voice may be displayed on the output unit 103. At this time, it may be configured so that the output to the examiner can be selected from the output by characters and the output by voice according to the instruction from the examiner. Further, it may be configured so that the instruction from the examiner can be selected from the input by characters and the input by voice according to the instruction from the examiner. Further, the various configurations described above may be adopted by selection according to an instruction from the examiner.

[Modification 9]
A label image, a high-quality image, or the like related to the image acquired by the main shooting may be stored in the storage unit 101-3 according to an instruction from the operator. At this time, for example, after an instruction from the operator for saving a high-quality image, when registering the file name, as a recommended file name, any part of the file name (for example, the first part, or In the last part), a file name containing information (for example, characters) indicating that the image is generated by processing using a trained model for high image quality (high image quality processing) is given by the operator. It may be displayed in an editable state according to the instruction. Similarly, for the label image and the like, the file name including the information which is the image generated by the process using the trained model may be displayed.

Further, when displaying a high-quality image on the output unit 103 on various display screens such as a report screen, the displayed image is a high-quality image generated by processing using a high-quality model. The display shown may be displayed together with a high-quality image. In this case, the operator can easily identify from the display that the displayed high-quality image is not the image itself acquired by shooting, so that misdiagnosis can be reduced or the diagnosis efficiency can be improved. be able to. The display indicating that the image is a high-quality image generated by the process using the high-quality model is any mode as long as the input image and the high-quality image generated by the process can be distinguished. It may be one. Further, it is shown that not only the processing using the high image quality model but also the processing using various trained models as described above is the result generated by the processing using the trained model of that kind. The display may be displayed with the result. For example, when displaying the analysis result of the segmentation result using the trained model for image segmentation processing, the display indicating that the analysis result is based on the result using the trained model for image segmentation is analyzed. It may be displayed with the result.

At this time, the display screen such as the report screen may be stored in the storage unit 101-3 as image data in response to an instruction from the operator. For example, a report screen is stored in the storage unit 101-3 as one image in which a high-quality image or the like and a display indicating that these images are images generated by processing using a trained model are arranged side by side. May be good.

In addition, regarding the display indicating that the image is a high-quality image generated by processing using the high-quality model, the output unit shows what kind of training data the high-quality model has learned. It may be displayed in 103. The display may include an explanation of the types of input data and correct answer data of learning data, and an arbitrary display regarding correct answer data such as an imaging site included in the input data and correct answer data. It should be noted that even in the case of processing using the various trained models described above, such as image segmentation processing, the output unit 103 displays a display indicating what kind of training data the trained model of that type is trained by. It may be displayed.

Further, information (for example, characters) indicating that the image is generated by processing using the trained model may be displayed or stored in a state of being superimposed on the image or the like. At this time, the portion superimposed on the image may be any region (for example, the edge of the image) that does not overlap with the region in which the region of interest to be photographed is displayed. Further, the non-overlapping areas may be determined and superimposed on the determined areas. It should be noted that not only the processing using the high image quality model but also the image obtained by the processing using the various trained models described above such as the image segmentation processing may be processed in the same manner.

In addition, if the initial display screen of the report screen is set by default so that the high image quality processing button or the like is in the active state (high image quality processing is on), it is set to high according to the instruction from the examiner. The report image corresponding to the report screen including the image quality image may be configured to be transmitted to the server. Also, if the button is set to the active state by default, at the end of the inspection (for example, when the shooting confirmation screen or preview screen is changed to the report screen according to the instruction from the inspector). In addition, the report image corresponding to the report screen including the high-quality image may be configured to be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, whether or not the analysis map is superimposed, whether or not the image is a high-quality image, and the display screen for follow-up observation. The report image generated based on (settings related to at least one such as whether or not) may be configured to be sent to the server. It should be noted that the same processing may be performed even when the button represents the switching of the image segmentation processing.

[Modification 10]
In the above-described embodiments and modifications, among the various trained models described above, images obtained by the first type of trained model (for example, high-quality images, images showing analysis results such as analysis maps, etc.). An image showing the area recognition result, an image showing the segmentation result) may be input to the trained model of the second type different from the first type. At this time, it may be configured to generate the result (for example, analysis result, diagnosis result, area recognition result, segmentation result) by the processing of the second kind of trained model.

Further, among the various trained models as described above, the first type is used by using the result of the processing of the first type of trained model (for example, analysis result, diagnosis result, area recognition result, segmentation result). An image to be input to a second type of trained model different from the first type may be generated from the image input to the trained model of. At this time, the generated image is likely to be an image suitable as an image to be processed using the second type of trained model. Therefore, an image obtained by inputting the generated image into the second type of trained model (for example, a high-quality image, an image showing an analysis result such as an analysis map, an image showing an area recognition result, and a segmentation result are displayed. The accuracy of the image shown) can be improved.

By inputting a common image into the first type of trained model and the second type of trained model, the generation (or display) of each processing result using these trained models can be generated. It may be configured to run. At this time, for example, it may be configured to collectively (in conjunction with) generate (or display) each processing result using these trained models in response to an instruction from the examiner.

Also, the type of image to be input (for example, high-quality image, area recognition result, object recognition result, segmentation result, similar case image), and the type of processing result to be generated (or displayed) (for example, high-quality image, area recognition result). , Diagnosis result, analysis result, object recognition result, segmentation result, similar case image), input type and output type (for example, character, voice, language) can be selected according to the instruction from the examiner. May be done. Further, the input type may be configured to be automatically selectable by using a trained model that automatically recognizes the input type. Further, the output type may be configured to be automatically selectable so as to correspond to the input type (for example, to be the same type). Further, the automatically selected type may be configured to be modifiable according to an instruction from the examiner. At this time, at least one trained model may be configured to be selected according to the selected type. At this time, when a plurality of trained models are selected, how to combine the plurality of trained models (for example, the order in which data is input) may be determined according to the selected type. It should be noted that, for example, the type of the image to be input and the type of the processing result to be generated (or displayed) may be configured to be differently selectable, or if they are the same, they may be selected differently. It may be configured to output prompting information to the examiner.

Also, each trained model may be executed anywhere. For example, some of the plurality of trained models may be used in a cloud server, and the others may be configured to be used in another server such as a fog server or an edge server. In addition, when the network in the facility, the site including the facility, the area including multiple facilities, etc. is configured to enable wireless communication, for example, it is assigned only to the facility, the site, the area, etc. The reliability of the network may be improved by configuring so as to use radio waves in a dedicated wavelength band. Further, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and simultaneous connection of a large number of people. With these, for example, surgery such as vitreous body, cataract, glaucoma, corneal refraction correction, external eye, and treatment such as laser photocoagulation can be supported in real time even if it is remote. At this time, for example, information obtained by a fog server, an edge server, or the like that wirelessly receives at least one of various medical images obtained by these devices related to surgery or treatment using at least one of various trained models. May be configured to wirelessly transmit to a device for surgery or treatment. Further, for example, the information wirelessly received by the device related to surgery or treatment may be the amount of movement (vector) of the optical system or optical member as described above. In this case, the device related to surgery or treatment is automatically controlled. It may be configured to be. Further, for example, for the purpose of supporting the operation by the examiner, it may be configured as an automatic control (semi-automatic control) with the permission of the examiner.

Further, a similar case image search using an external database stored in a server or the like may be performed using the analysis result, the diagnosis result, etc. obtained by the processing of the trained model as described above as a search key. Further, a similar case image search using an external database stored in a server or the like may be performed using an object recognition result, a segmentation result, or the like obtained by processing various trained models as described above as a search key. If the plurality of medical images stored in the database are already managed by machine learning or the like with the feature amount of each of the plurality of medical images attached as incidental information, the medical image itself may be used. A similar case image search engine (similar case image search model, trained model for similar case image search) as a search key may be used. For example, the image processing apparatus 101 searches various medical images for similar case images related to the medical image by using the trained model for similar case image search (different from the trained model for high image quality). It can be performed.

Further, for example, the display control unit 101-5 can display the similar case image obtained from various medical images using the trained model for searching the similar case image on the output unit 103. At this time, the similar case image is, for example, an image having a feature amount similar to the feature amount of the medical image input to the trained model. Further, the similar case image is, for example, an image of a feature amount similar to the feature amount of the partial area such as the abnormal part when the medical image input to the trained model includes a partial area such as an abnormal part. .. Therefore, for example, not only can the learning for accurately searching for similar case images be performed efficiently, but also when the medical image contains an abnormal part, the examiner efficiently diagnoses the abnormal part. be able to. Further, a plurality of similar case images may be searched, and a plurality of similar case images may be displayed so that the order in which the feature amounts are similar can be identified. In addition, a trained model for searching similar case images is additionally learned using learning data including an image selected according to an instruction from the examiner and a feature amount of the image among a plurality of similar case images. It may be configured to be.

Further, the training data of various trained models is not limited to the data obtained by using the ophthalmic apparatus itself that actually performs the imaging, and the data obtained by using the same type of ophthalmic apparatus or the same type according to the desired configuration. It may be data obtained by using an ophthalmic apparatus or the like.

The various trained models according to the above-described embodiments and modifications can be provided in the image processing device 101. The trained model may be configured by, for example, a CPU, a software module executed by a processor such as an MPU, GPU, FPGA, or the like, or may be configured by a circuit or the like that performs a specific function such as an ASIC. Further, these trained models may be provided in a device of another server connected to the image processing device 101 or the like. In this case, the image processing device 101 can use the trained model by connecting to a server or the like provided with the trained model via an arbitrary network such as the Internet. Here, the server provided with the trained model may be, for example, a cloud server, a fog server, an edge server, or the like. In addition, when the network in the facility, the site including the facility, the area including multiple facilities, etc. is configured to enable wireless communication, for example, it is assigned only to the facility, the site, the area, etc. The reliability of the network may be improved by configuring so as to use radio waves in a dedicated wavelength band. Further, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and simultaneous connection of a large number of people.

[Modification 11]
The medical images processed by the image processing apparatus 101 according to the various embodiments and modifications described above include images acquired using any modality (imaging apparatus, imaging method). The medical image to be processed may include a medical image acquired by an arbitrary imaging device or the like, or an image created by a medical image processing device or a medical image processing method.

Further, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. In addition, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black-and-white image or a color image. Further, the medical image may be an image showing the structure (morphology) of a predetermined part or an image showing the function thereof. The image showing the function includes, for example, an OCTA image, a Doppler OCT image, an fMRI image, and an image showing blood flow dynamics (blood flow volume, blood flow velocity, etc.) such as an ultrasonic Doppler image. The predetermined site of the subject may be determined according to the subject to be imaged, and the human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, organs such as liver, head, chest, etc. Includes any part such as legs and arms.

Further, the medical image may be a tomographic image of the subject or a frontal image. The frontal image is, for example, an SLO image of the fundus or anterior segment of the eye, a fundus image taken by fluorescence, and data acquired by OCT (three-dimensional OCT data) in at least a part of the range in the depth direction of the object to be imaged. Includes En-Face images generated using. The En-Face image is an OCTA En-Face image (motion contrast front image) generated by using at least a part of the data in the depth direction of the shooting target for the three-dimensional OCTA data (three-dimensional motion contrast data). ) May be. Further, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Here, the motion contrast data is data showing a change between a plurality of volume data obtained by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data showing changes between a plurality of tomographic images obtained at substantially the same position at different positions. The motion contrast front image is also referred to as an OCTA front image (OCTA En-Face image) relating to OCTA angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also referred to as OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals. , Can be determined by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. When controlling the scanning means so that the measurement light is scanned multiple times at substantially the same position, the time interval (time interval) between one scan (one B scan) and the next scan (next B scan). ) May be modified (determined). Thereby, for example, even if the blood flow velocity may differ depending on the state of the blood vessel, the blood vessel region can be visualized with high accuracy. At this time, for example, the time interval may be changed so as to be instructed by the examiner. Further, for example, any motion contrast image may be configured to be selectable from a plurality of motion contrast images corresponding to a plurality of preset time intervals according to an instruction from the examiner. Further, for example, the time interval when the motion contrast data is acquired and the motion contrast data may be associated with each other and stored in the storage unit 101-3. Further, for example, the display control unit 101-5 may display the time interval when the motion contrast data is acquired and the motion contrast image corresponding to the motion contrast data on the output unit 103. Further, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. At this time, for example, using a machine learning model, the time interval may be determined (output) from the motion contrast image. In such a machine learning model, for example, a plurality of motion contrast images corresponding to a plurality of time intervals are used as input data, and the difference from the plurality of time intervals to the time interval when a desired motion contrast image is acquired is correctly answered. Learning as data It can be obtained by learning the data.

The En-Face image is, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions. At this time, the front image is at least a part of the depth range of the volume data (three-dimensional tomographic image) obtained by using optical interference, and is the data corresponding to the depth range determined based on the two reference planes. Is projected or integrated on a two-dimensional plane. The En-Face image is a frontal image generated by projecting the data corresponding to the depth range determined based on the detected retinal layer among the volume data onto a two-dimensional plane. As a method of projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on the two-dimensional plane. Techniques can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Further, the depth range related to the En-Face image is, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries relating to the detected retinal layer. May be good. Further, the depth range related to the En-Face image may be, for example, a range changed (offset) according to the instruction of the operator from the range between the two layer boundaries regarding the detected retinal layer. good.

The photographing device is a device for photographing an image used for diagnosis. The photographing device detects, for example, a device that obtains an image of a predetermined part by irradiating a predetermined part of the subject with radiation such as light or X-rays, electromagnetic waves, ultrasonic waves, or the like, or radiation emitted from a subject. This includes a device for obtaining an image of a predetermined part. More specifically, the imaging devices according to the various embodiments and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, and a fundus. Includes cameras, endoscopes, etc. It should be noted that the configurations according to each of the above-described embodiments and modifications can be applied to these imaging devices. In this case, the movement of the subject corresponding to the above-mentioned movement of the eye to be predicted may be, for example, the movement of the face or body, the movement of the heart (heartbeat), or the like.

The OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Further, the Fourier domain OCT apparatus may include a spectral domain OCT (SD-OCT) apparatus and a wavelength sweep type OCT (SS-OCT) apparatus. Further, the OCT device may include a Line-OCT device (or SS-Line-OCT device) using line light. Further, the OCT device may include a Full Field-OCT device (or SS-Full Field-OCT device) using area light. Further, the OCT device may include a Doppler-OCT device. Further, the SLO device and the OCT device may include a wave surface compensating SLO (AO-SLO) device using a wave surface compensating optical system, a wave surface compensating OCT (AO-OCT) device, and the like. Further, the SLO device and the OCT device may include a polarized SLO (PS-SLO) device, a polarized OCT (PS-OCT) device, and the like for visualizing information on polarization phase difference and polarization elimination. Further, the SLO device and the OCT device may include a pathological microscope SLO device, a pathological microscope OCT device, and the like. Further, the SLO device and the OCT device may include a handheld type SLO device, a handheld type OCT device, and the like. Further, the SLO device and the OCT device may include a catheter SLO device, a catheter OCT device and the like. Further, the SLO device and the OCT device may include a head-mounted SLO device, a head-mounted OCT device, and the like. Further, the SLO device and the OCT device may include a binocular type SLO device, a binocular type OCT device, and the like. Further, the SLO device and the OCT device may be capable of changing the shooting angle of view by a configuration capable of optical scaling. Further, the SLO device can acquire a color image or a fluorescent image by using each of the RGB light sources and having a configuration in which one light receiving element receives light in a time-division manner or a configuration in which a plurality of light receiving elements simultaneously receive light. May be good.

Further, in the above-described embodiment and modification, the image processing device 101 is configured as a part of the OCT device, but the image processing device 101 may be configured as a separate body from the OCT device. In this case, the image processing device 101 may be connected to the photographing device 100 or the like of the OCT device via the Internet or the like. Further, the configuration of the OCT apparatus is not limited to the above configuration, and a part of the configuration included in the OCT apparatus may be a configuration in which, for example, an SLO photographing unit or the like is separate from the OCT apparatus.

In the trained models for voice recognition, character recognition, gesture recognition, etc., the inclination between the input continuous time-series data values is extracted as a part of the feature amount and used for the estimation process. it is conceivable that. It is expected that such a trained model can perform accurate estimation by using the influence of a specific numerical value over time in the estimation process. Further, even in the trained models for high image quality, region recognition, segmentation processing, image analysis, and diagnosis result generation according to the above-described embodiments and modifications, the magnitude and brightness of the tomographic image are different. It can be considered that the order, inclination, position, distribution, continuity, etc. of the dark part are extracted as a part of the feature amount and used for the estimation process.

(Other embodiments)
The present invention supplies software (programs) that realize one or more functions of the various embodiments and modifications described above to a system or device via a network or storage medium, and the computer of the system or device provides the program. It can also be realized by the process of reading and executing. A computer may have one or more processors or circuits and may include multiple separate computers or a network of separate processors or circuits for reading and executing computer-executable instructions.

At this time, the processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Although the present invention has been described above with reference to the embodiments and modifications, the present invention is not limited to the above embodiments and modifications. The present invention also includes inventions modified to the extent not contrary to the gist of the present invention, and inventions equivalent to the present invention. In addition, the above-described embodiments and modifications can be appropriately combined as long as they do not contradict the gist of the present invention.

101: Image processing device, 101-47: High image quality improving unit, 200: Eye to be inspected (subject), 301: High image quality processing unit (high image quality improving unit) 302: Area recognition unit, 304: Blend processing unit (image) Processing unit), 305: BC adjustment unit (image processing unit)

Claims (29)

Using the trained model obtained by learning using the medical image of the subject as training data, the first image of the medical image of the subject is subjected to high image quality processing, and the second image of the medical image of the subject is subjected to high image quality processing. High image quality section to acquire images and
A detection unit that detects the target area in the medical image of the subject,
An image in which image processing is performed so that the difference between the pixel value of the target area and the pixel value of the area other than the target area is widened and the pixel value of the target area is lower than the target area in the second image. Processing unit and
An image processing device. The image processing according to claim 1, further comprising a process of blending the first image and the second image so that the difference between the pixel value of the target area and the pixel value of the area other than the target area is widened. Image processing equipment. The detection unit detects a target area using the first image, and determines the target area.
The image processing apparatus according to claim 1 or 2, wherein the image processing unit performs the image processing on the target area in the second image corresponding to the detected target area in the first image. The image processing apparatus according to claim 1 or 2, wherein the detection unit detects a target area using the second image. A detection unit that detects the target area in the medical image of the subject,
The difference between the pixel value of the target area and the pixel value of the area other than the target area is widened and the pixel value of the target area is lower than the target area in the first image of the medical image of the subject. An image processing unit that performs image processing and
Using the trained model obtained by learning using the medical image of the subject as training data, the first image subjected to the image processing is subjected to high image quality processing, and the second image of the medical image of the subject is subjected to high image quality processing. High image quality section to acquire the image of
An image processing device. The image processing apparatus according to claim 5, wherein the detection unit detects a target area using the first image. The image processing apparatus according to claim 5, wherein the detection unit detects a target area using an image obtained by processing the first image without image processing to improve the image quality. The image processing apparatus according to any one of claims 1 to 7, wherein the detection unit detects a target area by using a trained model obtained by learning using a medical image of a subject as learning data. The medical image is a frontal image,
The detection unit detects a two-dimensional target area by using a trained model obtained by learning using a plurality of front images of a subject corresponding to a plurality of depth ranges as training data, according to claims 1 to 8. The image processing apparatus according to any one of the items. The medical image is a frontal image,
The detection unit has a depth selected according to an instruction from the examiner among a plurality of trained models obtained by learning using a plurality of front images of a subject corresponding to a plurality of depth ranges as learning data. The image processing apparatus according to any one of claims 1 to 8, wherein a trained model corresponding to a range is selected, and the selected trained model is used to detect a two-dimensional target area. The medical image is a three-dimensional medical image,
The one according to any one of claims 1 to 8, wherein the detection unit detects a three-dimensional target area by using a learned model obtained by learning using a three-dimensional medical image of a subject as learning data. Image processing equipment. The image according to any one of claims 1 to 3, 5, and 6, wherein the detection unit detects a target region by a rule-based process based on the structure of the subject using the first image. Processing equipment. One of claims 1 to 12, wherein the image processing includes a process of correcting at least one of brightness and contrast so that a difference between a pixel value in a target area and a pixel value in a region other than the target area is widened. The image processing apparatus according to. The medical image of the subject is a motion contrast image of the eye to be inspected.
The image processing apparatus according to any one of claims 1 to 13, wherein the target region includes at least one of a non-perfused region, a foveal blood vessel region, and an optic disc region. The acquisition unit that acquires the first image of the medical image of the subject, and
High image quality processing using the first trained model obtained by training using the medical image of the subject as training data, blending processing in the target area of the medical image of the subject, or correction of at least one of brightness and contrast. Using the second trained model obtained by learning using the processed medical image of the subject as training data, the first image is subjected to high image quality processing, and the medical image of the subject is processed. The high image quality section that acquires the second image of
An image processing device comprising. The image processing apparatus according to any one of claims 1 to 15, further comprising a display control unit for displaying a live moving image of a subject on the display unit. The display control unit is a high-quality image generated by using a trained model obtained by learning learning data including an image relating to the subject, and is a high-quality image obtained by inputting a medical image of the subject. The image processing apparatus according to claim 16, wherein an image is displayed on the display unit. The display control unit displays the front eye image generated as the high-quality image on the display unit as the live moving image, and is an SLO image generated as the high-quality image, which is generated as the high-quality image. The SLO image on which the line indicating the position of the tomographic image is superimposed is displayed on the display unit as the live moving image, and the tomographic image corresponding to the position of the line on the SLO image is used as the live moving image. The image processing apparatus according to claim 17, which is displayed on the display unit. The display control unit superimposes and displays information indicating a blood vessel region in the tomographic image corresponding to the position of the line, which is a tomographic image generated as the high-quality image, on the tomographic image corresponding to the position of the line. The image processing apparatus according to claim 18. The display control unit is an analysis result generated by using a trained model for generating an analysis result obtained by learning training data including an image related to a subject, and is obtained by inputting an image related to the subject. The image processing apparatus according to any one of claims 16 to 19, wherein the analysis result is displayed on the display unit. The display control unit is a diagnostic result generated by using a trained model for generating a diagnostic result obtained by learning training data including an image related to the subject, and is obtained by inputting an image related to the subject. The image processing apparatus according to any one of claims 16 to 20, wherein the diagnosis result is displayed on the display unit. The display control unit is an image generated by using a hostile generation network or an auto encoder, and is an image obtained by inputting an image relating to a subject and the image input to the hostile generation network or the auto encoder. The image processing apparatus according to any one of claims 16 to 21, wherein information regarding a difference from an image relating to a subject is displayed on the display unit as information regarding an abnormal site. The display control unit is a similar case image searched by using a trained model for similar case image search obtained by learning learning data including an image related to the subject, and inputs an image related to the subject. The image processing apparatus according to any one of claims 16 to 22, wherein the obtained similar case image is displayed on the display unit. The display control unit is an object recognition result or segmentation result generated by using a trained model for object recognition or a trained model for segmentation obtained by learning training data including an image of a subject. The image processing apparatus according to any one of claims 16 to 23, wherein an object recognition result or a segmentation result obtained by inputting an image relating to a subject is displayed on the display unit. The operator's instruction for acquiring the second image uses at least one trained model of a trained model for character recognition, a trained model for voice recognition, and a trained model for gesture recognition. The image processing apparatus according to any one of claims 1 to 24, which is the information obtained. Using the trained model obtained by learning using the medical image of the subject as training data, the first image of the medical image of the subject is subjected to high image quality processing, and the second image of the medical image of the subject is subjected to high image quality processing. To get an image and
Detecting the target area in the medical image of the subject and
Image processing is performed so that the difference between the pixel value of the target area and the pixel value of the area other than the target area is widened and the pixel value of the target area is lower than the target area in the second image. When,
Image processing methods, including. Detecting the target area in the medical image of the subject and
The difference between the pixel value of the target area and the pixel value of the area other than the target area is widened and the pixel value of the target area is lower than the target area in the first image of the medical image of the subject. Performing image processing and
Using the trained model obtained by learning using the medical image of the subject as training data, the first image subjected to the image processing is subjected to high image quality processing, and the second image of the medical image of the subject is subjected to high image quality processing. To get an image of
Image processing methods, including. Acquiring the first image of the subject's medical image and
Learning using the medical image of the subject as training data, which has been subjected to high image quality processing using the first trained model obtained by learning using the medical image of the subject as training data and image processing in the target area. Using the second trained model obtained in the above, the first image is subjected to high image quality processing to obtain a second image of the medical image of the subject.
Image processing methods, including. A program that, when executed by a computer, causes the computer to perform each step of the image processing method according to any one of claims 26 to 28.

Images