JP7344847B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

As an apparatus for obtaining a tomographic image of a subject, an apparatus using optical coherence tomography (OCT) (OCT apparatus) is known. By using a medical tomographic imaging device such as an OCT device, it is possible to observe the internal state of the retinal layer three-dimensionally. useful for diagnosis. OCT, which has been used in clinical practice in recent years, is roughly divided into two methods, for example, SD-OCT (Spectral Domain OCT) and SS-OCT (Swept Source OCT), as a method for acquiring images at high speed. In SD-OCT, a broadband light source is used and an interferogram is obtained with a spectrometer. In contrast, SS-OCT uses a high-speed wavelength swept light source as a light source to measure spectral interference with a single-channel photodetector.

Recently, among both types of OCT, OCT angiography (OCTA), which images blood vessels without using a contrast agent, has attracted attention. In OCTA, motion contrast data is generated from an OCT image obtained by OCT. Here, motion contrast data is data obtained by repeatedly photographing the same cross-section of a measurement object using OCT and detecting temporal changes in the measurement object between the images, such as the phase, vector, and intensity of a complex OCT signal. It is calculated from the difference, ratio, correlation, etc. over time.

Furthermore, when displaying an OCTA image, it is becoming common to display it as a two-dimensional OCTA frontal image by projecting three-dimensional motion contrast data calculated from the acquired three-dimensional OCT image onto a two-dimensional plane. be. In this regard, Patent Document 1 discloses a technique for generating a two-dimensional front image by specifying a range in the depth direction of motion contrast data to be projected. Further, Patent Document 2 discloses a technique for generating a high-resolution image from a low-resolution image using an artificial intelligence engine obtained by machine learning using low-resolution images and high-resolution images.

JP 2017-6179 Publication Unexamined Japanese Patent Publication No. 2018-5841

However, there may be room for improvement in various aspects regarding analysis processing of data obtained using OCT (OCT data) or motion contrast data. For example, when performing analysis processing of OCT data or motion contrast data, there is a possibility that over-correction may occur, especially when an artificial intelligence engine is used to correct an image.

Therefore, one of the objects of the present invention is to provide an image processing apparatus, an image processing method, and a program that reduce overcorrection caused by image processing of medical images using a trained model.

An image processing apparatus according to at least one embodiment of the present invention processes a first image that is a medical image of a subject using a trained model obtained by learning using a medical image of a subject as learning data. an image quality improvement unit that acquires a second image that is a medical image of the subject by performing image quality enhancement processing; a detection unit that detects a target area in the medical image of the subject; and a detection unit that detects a target area in the medical image of the subject; an image processing unit that performs image processing on the target region in such a way that the difference between the pixel value of the target region and the pixel value of a region other than the target region is widened and the pixel value of the target region is lowered; a display control unit that displays a high-quality image obtained by inputting a medical image of the subject as input data of a trained model obtained by learning training data including images related to the subject as a live video image on a display unit; The display control unit displays the front image obtained as the high-quality image on which a line indicating the position of the tomographic image obtained as the high-quality image is superimposed and displayed as the live video image. The tomographic image corresponding to the position of the line on the front image is displayed on the display section as the live moving image.

According to at least one embodiment of the present invention, overcorrection due to image processing of medical images using a trained model can be reduced.

FIG. 1 is a block diagram showing a schematic configuration of an OCT apparatus according to a first embodiment. FIG. 1 is a diagram illustrating a schematic configuration of an imaging device according to a first embodiment. FIG. 2 is a block diagram showing a schematic configuration of an image quality improvement unit according to the first embodiment. An example of the configuration of a neural network related to a high image quality engine is shown. An example of learning data regarding high image quality processing is shown. An example of an input image related to high image quality processing is shown. FIG. 2 is a flow diagram schematically showing the flow of image processing according to the first embodiment. FIG. 2 is a flow diagram illustrating an example of the flow of image processing according to the first embodiment. FIG. 2 is a flow diagram illustrating an example of the flow of image processing according to the first embodiment. FIG. 2 is a flow diagram illustrating an example of the flow of image processing according to the first embodiment. FIG. 2 is a flow diagram illustrating an example of the flow of image processing according to the first embodiment. FIG. 2 is a flow diagram illustrating an example of the flow of image processing according to the first embodiment. FIG. 2 is a flow diagram illustrating an example of the flow of image processing according to the first embodiment. FIG. 7 is a flow diagram showing a schematic flow of image processing according to the second embodiment. An example of a machine learning model according to modification 8 is shown. An example of a machine learning model according to modification 8 is shown.

Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of components, etc. described in the following embodiments are arbitrary and can be changed depending on the configuration of the device to which the present invention is applied or various conditions. Additionally, in the drawings, the same reference numerals are used between the drawings to indicate elements that are identical or functionally similar. In addition, hereinafter, the ocular axis direction will be described as Z, the fundus plane horizontal direction as X, and the fundus plane vertical direction as Y.

Note that in the following, a machine learning model refers to a learning model using a machine learning algorithm. Specific algorithms for machine learning include the nearest neighbor method, the Naive Bayes method, decision trees, and support vector machines. Another example is deep learning, which uses neural networks to generate feature quantities and connection weighting coefficients for learning by itself. Appropriately, any of the above algorithms that can be used can be applied to the following embodiments and modifications. Further, the teaching data refers to learning data, and is composed of a pair of input data and output data. In addition, correct data refers to output data of learning data (teacher data).

Note that a trained model is a machine learning model that follows any machine learning algorithm such as deep learning, and has been trained (learning) in advance using appropriate teaching data (learning data). . However, although a trained model is obtained in advance using appropriate training data, it does not mean that no further training is performed, and additional training can be performed. Additional learning can also occur after the device is installed at the site of use.

(First embodiment)
Hereinafter, an image processing system according to a first embodiment of the present invention will be described in detail with reference to FIGS. 1 to 13. In this embodiment, as an example of an image processing system, an OCT apparatus including an image processing apparatus that processes a tomographic image of a subject obtained by OCT will be described.

(Configuration of image processing device)
The configuration of the image processing apparatus 101 and connections with other devices according to this embodiment will be described with reference to FIG. 1. The image processing device 101 is connected to a photographing device 100, an external storage device 102, an output section 103, and an input section 104. Note that these connections may be wired connections or wireless connections. Further, these connections may be connections via a network. For example, the image processing device 101 may be connected to the photographing device 100 via a network such as the Internet. Further, for example, the external storage device 102 may be placed on a network such as the Internet, and configured such that data can be shared by a plurality of image processing devices.

The image processing device 101 is provided with an acquisition section 101-1, an imaging control section 101-2, an image processing section 101-4, a display control section 101-5, and a storage section 101-3, which are functional blocks. There is. The image processing device 101 can be configured using a general computer including a processor, memory, etc., but may also be configured as a dedicated computer for the OCT device. Here, the image processing apparatus 101 may be a built-in (internal) computer of the OCT apparatus, or may be a separate (external) computer to which the OCT apparatus is communicably connected. Furthermore, the image processing device 101 may be, for example, a personal computer, such as a desktop PC, a notebook PC, or a tablet PC (portable information terminal). Note that the processor may be a CPU (Central Processing Unit). Further, the processor may be, for example, an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or an FPGA (Field-Programmable Gate Array).

Each function of the image processing device 101 may be realized by a processor such as a CPU or an MPU executing a software module stored in the storage unit 101-3. Note that the processor may be, for example, a GPU, FPGA, or the like. Furthermore, each function may be configured by a circuit, such as an ASIC, that performs a specific function. For example, the image processing unit 101-4 may be implemented using dedicated hardware such as an ASIC, and the display control unit 101-5 may be implemented using a dedicated processor such as a GPU that is different from the CPU. The storage unit 101-3 may be configured by any storage medium such as an optical disk such as a hard disk or a memory.

The acquisition unit 101-1 acquires signal data such as SLO images, tomographic images, and anterior eye segment images obtained by photographing the subject using the imaging device 100, images such as SLO images, tomographic images, and anterior eye observation images, and patient information. This is a functional block that obtains etc. Further, the acquisition unit 101-1 can generate images such as an SLO image, a tomographic image, and an anterior eye observation image using the acquired signal data. The acquisition unit 101-1 is provided with a tomographic image generation unit 101-11 and a motion contrast data generation unit 101-12. Note that the acquisition unit 101-1 may acquire various data including images from an external device (not shown) such as a server connected to the image processing device 101. Here, the image processing device 101 and an external device (not shown) may be connected via any network such as the Internet. Further, the acquisition unit 101-1 may acquire various data stored in the storage unit 101-3.

The tomographic image generation unit 101-11 performs signal processing on the acquired tomographic image signal data (interference signal) to generate a tomographic image, and stores the generated tomographic image in the storage unit 101-3. Note that any known method may be used to generate a tomographic image from the interference signal.

The motion contrast data generation unit 101-12 generates motion contrast data based on the plurality of tomographic images at approximately the same position (mutually corresponding regions of the subject) generated by the tomographic image generation unit 101-11. A method of generating motion contrast data will be described below.

First, the tomographic image generation unit 101-11 generates a plurality of tomographic images from a plurality of interference signals obtained by photographing substantially the same position of the subject multiple times. Here, a scanning group of measurement light when photographing substantially the same position of a subject multiple times in order to generate motion contrast data is referred to as one cluster. Furthermore, a plurality of tomographic images corresponding to a plurality of interference signals obtained by photographing the same position of the subject multiple times are referred to as one cluster (one group) of tomographic images. More specifically, the tomographic image generation unit 101-11 performs wave number transformation, fast Fourier transform (FFT), and absolute value transformation ( By performing the amplitude acquisition), a tomographic image for one cluster is generated. Note that the method of generating a tomographic image is not limited to this, and any known method of performing other image processing may be used.

Next, an alignment unit 101-41 of the image processing unit 101-4, which will be described later, aligns the tomographic images belonging to the same cluster and performs superimposition processing. Thereafter, an image feature acquisition unit 101-44 of the image processing unit 101-4, which will be described later, acquires layer boundary data from the superimposed tomographic image. In this embodiment, a variable shape model is used as a layer boundary acquisition method, but any known layer boundary acquisition method may be used. Note that the layer boundary acquisition process is not essential; for example, if a motion contrast image is generated only in three dimensions and a two-dimensional motion contrast image projected in the depth direction is not generated, the layer boundary acquisition process can be omitted. .

The motion contrast data generation unit 101-12 calculates motion contrast data between adjacent tomographic images within the same cluster. In this embodiment, the motion contrast data generation unit 101-12 calculates a decorrelation value Mxz as motion contrast data based on the following equation (1).

Here, Axz indicates the amplitude (of complex number data after FFT processing) at the position (x, z) of the tomographic image data A, and Bxz indicates the amplitude at the same position (x, z) of the tomographic data B. 0≦Mxz≦1, and the larger the difference between both amplitude values, the closer the value is to 1. The motion contrast data generation unit 101-12 performs a decorrelation calculation process as shown in equation (1) between arbitrary temporally adjacent tomographic images (belonging to the same cluster). Note that the tomographic images on which the decorrelation calculation process is performed do not necessarily have to be temporally adjacent as long as they are images within a predetermined period. The motion contrast data generation unit 101-12 generates, as a final motion contrast image, an image whose pixel value is the average of the values of the obtained (number of tomographic images per cluster - 1) pieces of motion contrast data.

Note that although the motion contrast data is calculated here based on the amplitude of the complex number data after FFT processing, the method of calculating the motion contrast data is not limited to the above. For example, motion contrast data may be calculated based on phase information of complex number data, or motion contrast data may be calculated based on both amplitude and phase information. Alternatively, motion contrast data may be calculated based on the real part or imaginary part of complex number data.

Further, in this embodiment, a decorrelation value is calculated as motion contrast data, but the calculation method of motion contrast data is not limited to this, and any known calculation method may be used. For example, motion contrast data may be calculated based on the difference between two values, or may be calculated based on a ratio between two values. Further, the motion contrast data can also be obtained, for example, as a dispersion value between two tomographic images or their corresponding interference signals, or a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value). Any known method may be used for these calculations.

In addition, when controlling the scanning means so that the measurement light scans approximately the same position multiple times, the time interval (time interval) between one scan (one B scan) and the next scan (next B scan) ) may be changed (determined). Thereby, for example, even if the blood flow velocity varies depending on the condition of the blood vessel, the blood vessel region can be visualized with high accuracy.

At this time, the time interval may be configured to be changeable, for example, according to an instruction from an operator (examiner). Further, for example, one of the motion contrast images may be selectable from a plurality of motion contrast images corresponding to a plurality of preset time intervals in response to an instruction from the operator. Further, for example, the storage unit 101-3 may be configured to be able to associate the time interval at which the motion contrast data is acquired with the motion contrast data and store it in the storage unit 101-3. Further, for example, the display control unit 101-5 may cause the output unit 103 to display the time interval at which the motion contrast data was acquired and the motion contrast image corresponding to the motion contrast data. Further, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. At this time, the time interval may be determined (output) from the motion contrast image using, for example, a machine learning model. Such a machine learning model, for example, uses multiple motion contrast images corresponding to multiple time intervals as input data, and calculates the difference between the multiple time intervals and the time interval when the desired motion contrast image was acquired as the correct answer. It can be obtained by learning learning data used as data.

Further, here, the final pixel value of the motion contrast image is obtained by calculating the average value of the plurality of acquired decorrelation values, but the final pixel value is not limited to this. For example, an image having the median value or maximum value of the plurality of acquired decorrelation values as a pixel value may be generated as the final motion contrast image.

The photographing control unit 101-2 is a functional block that performs photographing control on the photographing apparatus 100. For example, the photographing control section 101-2 can control the driving of a light source, a scanning section, a focusing lens drive device, etc. included in the photographing apparatus 100. The photographing control section 101-2 can control alignment operations of a stage section 100-2 of the photographing apparatus 100, which will be described later. Further, the photographing control unit 101-2 can also instruct the photographing apparatus 100 regarding the setting of photographing parameters, and the start or end of photographing.

The storage unit 101-3 can store programs for realizing various application software including an operating system (OS), device drivers for peripheral devices, and programs for performing processing, etc., which will be described later. Furthermore, the storage unit 101-3 can also store information acquired by the acquisition unit 101-1, various images processed by the image processing unit 101-4, and the like. For example, the storage unit 101-3 stores medical images such as tomographic images acquired by the acquisition unit 101-1, or stores images whose quality has been improved by an image quality improvement unit 101-47, which will be described later. be able to.

The image processing unit 101-4 is a functional block that performs image processing on various images such as tomographic images, motion contrast images, SLO images, and anterior eye observation images. The image processing unit 101-4 includes an alignment unit 101-41, a composition unit 101-42, a correction unit 101-43, an image feature acquisition unit 101-44, a projection unit 101-45, an analysis unit 101-46, and a height adjustment unit 101-42. An image quality improvement section 101-47 is provided.

The alignment unit 101-41 is a functional block that performs alignment processing between images. For example, the alignment unit 101-41 can align tomographic images belonging to the same cluster and perform superimposition processing. Note that the alignment process and superimposition process may be performed using any known method.

The composition unit 101-42 is a functional block that composes one image from each of a plurality of two-dimensional images. The synthesis section 101-42 includes, for example, a synthesis method specification section 101-421, a same modality image synthesis section 101-422, and a different modality image synthesis section 101-423.

The synthesis method specification unit 101-421 specifies the type of images to be synthesized (for example, tomographic image/motion contrast image/tomographic image and motion contrast image) and the synthesis processing method (for example, superimposition/stitching/juxtaposed display). do. The same modality image synthesis unit 101-422 performs, for example, synthesis processing between tomographic images or between motion contrast images. The different modality image synthesis unit 101-423 performs synthesis processing between images obtained by different types of modalities, such as between a tomographic image and a motion contrast image, for example.

The correction unit 101-43 is a functional block that performs processing to suppress projection artifacts occurring within a motion contrast image. Here, projection artifact is a phenomenon in which motion contrast within the superficial retinal blood vessels is reflected in the deep layers (deep retina, outer retina, and choroid), resulting in high decorrelation values in deep regions where no blood vessels actually exist. Point. For example, the correction unit 101-43 performs processing to reduce projection artifacts in the generated motion contrast data. Therefore, the correction unit 101-43 corresponds to an example of a processing unit that performs processing to reduce projection artifacts on the generated motion contrast data.

The image feature acquisition unit 101-44 acquires layer boundary data from a tomographic image, a motion contrast tomographic image, or the like. Specifically, the image feature acquisition unit 101-44 performs image segmentation processing on tomographic images and the like, extracts the layer structure in the tomogram of the eye to be examined, and acquires layer boundary data such as boundary positions. Note that any known method may be used for the image segmentation process.

The projection unit 101-45 is a functional block that projects or integrates a tomographic image or a motion contrast image in a set depth range to generate a brightness front image (brightness En-Face image) or an OCTA front image. Note that the depth range can be set using two reference planes based on instructions from the operator or layer boundary data acquired by the image feature acquisition unit 101-44. The set depth range may be any depth range. For example, the depth ranges of the retinal surface layer and the retinal outer layer can be set, and two types of combined OCTA frontal images can be generated by the combining unit 101-42.

Here, as a method of projecting data corresponding to a depth range set based on two reference planes onto a two-dimensional plane, for example, the representative value of the data within the depth range is converted into a pixel value on a two-dimensional plane. A method can be used to The representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Specifically, as the projection method, for example, Maximum Intensity Projection (MIP) or Average Intensity Projection (AIP) can be selected.

Further, the depth range related to the frontal image may be, for example, a range that includes a predetermined number of pixels in a deeper direction or a shallower direction with one of the two layer boundaries regarding the detected retinal layer as a reference. . Furthermore, the depth range related to the En-Face image may be, for example, a range changed (offset) from the range between the two layer boundaries regarding the detected retinal layers in accordance with the operator's instructions. good. Here, the depth range for generating the front image can be changed by the operator selecting from a predetermined depth range set displayed in a selection list (not shown) or the like. Additionally, the operator can change the type and offset position of the layer boundary used to specify the projection range from the user interface (UI), or move the layer boundary data superimposed on the tomographic image by operating the input unit 104. You can also change the projection range by

The generated brightness front image and OCTA front image can be output by the output unit 103. Note that the motion contrast image output to the output unit 103 is not limited to the OCTA frontal image, but may also be a three-dimensional motion contrast image rendered three-dimensionally or a motion contrast tomographic image corresponding to a tomographic image. good. Note that when the generated motion contrast image or the like is displayed by the output unit 103, the above-described projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a UI such as a context menu. For example, a motion contrast image after projection artifact suppression processing may be displayed as a three-dimensional image.

The analysis unit 101-46 is a functional block that performs analysis processing of various images such as tomographic images and motion contrast images. The analysis section 101-46 is provided with an emphasis section 101-461, an extraction section 101-462, a measurement section 101-463, and a comparison section 101-464.

The emphasizing unit 101-461 can perform an emphasizing process to emphasize data in an arbitrary area in an image, for example, in response to instructions from an operator. The extraction unit 101-462 can extract characteristic parts, regions, etc. from the image used by the analysis unit 101-46. For example, the extraction unit 101-462 can acquire the layer boundaries of the retina and choroid, the front and rear boundaries of the lamina cribrosa, the position of the fovea and the center of the optic disc, etc. from the tomographic image. Furthermore, the extraction unit 101-462 can extract a blood vessel region from the OCTA frontal image. The measurement unit 101-463 calculates measurement values to be analyzed from various images, for example. For example, the measurement unit 101-463 calculates the thickness of a specific layer, or uses the extracted blood vessel region or blood vessel center line data obtained by thinning the blood vessel region to obtain measured values such as blood vessel density. can be calculated. Furthermore, the comparison unit 101-464 can compare multiple images such as multiple tomographic images and multiple motion contrast images. Furthermore, the comparison unit 101-464 can also compare the analysis results of a plurality of images, such as measurement values calculated by the measurement unit 101-463.

The image quality enhancement unit 101-47 is a functional block that enhances the image quality of various images. Therefore, the image quality improvement unit 101-47 is an example of an image quality improvement unit that improves the image quality of medical images such as tomographic images and OCTA frontal images. Note that details of the configuration and functions of the image quality improvement section 101-47 will be described later.

The display control unit 101-5 can control the display of the output unit 103 connected to the image processing device 101. The display control unit 101-5 can, for example, cause the output unit 103 to display various data such as a tomographic image of the eye to be examined and patient information.

The external storage device 102 can store tomography programs, patient information, image information, and the like. For example, the external storage device 102 stores patient information (patient's name, age, gender, etc.), information about the eye to be examined (left eye, right eye, axial length, etc.), and captured images (tomographic image, SLO image, etc.). and OCTA images, etc.), composite images, imaging parameters, image data and measurement data of past examinations, parameters set by the operator, and the like can be stored in association with each other.

The input unit 104 includes, for example, a mouse, a keyboard, a touch panel, etc., and allows the operator to issue instructions to the image processing device 101 and the photographing device 100 via the input unit 104. The output unit 103 can output data such as various images processed by the image processing device 101. The output unit 103 can be configured with any display or the like, and can display, for example, patient information regarding the subject, various images, etc. based on the control by the display control unit 101-5. In this case, the output unit 103 can function as an example of a display unit that displays various images and information. Note that the output unit 103 may include a touch UI or the like.

Note that a part of the configuration of the OCT device may be configured as a separate device, or may be configured as an integrated device. For example, the output unit 103 may be integrally configured with the input unit 104 as a touch panel display.

(Configuration of imaging device)
The photographing device 100 is a device that photographs the eye to be examined in order to obtain a tomographic image, an SLO image, etc. of the eye to be examined. Hereinafter, the configuration of the measurement optical system and spectrometer in the imaging device 100 according to this embodiment will be described with reference to FIGS. 1 and 2. In this embodiment, it is assumed that an image capturing apparatus including an SD-OCT (Spectral Domain OCT) optical system is used as the image capturing apparatus 100. The present invention is not limited to this, and may be configured using, for example, an imaging device including an optical system such as SS-OCT or TD-OCT (Time Domain OCT).

As shown in FIG. 1, the imaging device 100 is provided with a measurement optical system 100-1, a stage section 100-2, and a base section 100-3. The measurement optical system 100-1 is an optical system for acquiring an anterior segment image, an SLO fundus image of the eye to be examined, and a tomographic image. The stage section 100-2 holds the measurement optical system 100-1 so as to be movable back and forth, right and left, and can move the measurement optical system 100-1 under the control of the imaging control section 101-2. The base portion 100-3 incorporates a spectrometer 230, which will be described later.

The configuration of measurement optical system 100-1 will be described with reference to FIG. 2. In the measurement optical system 100-1, an objective lens 201 is installed facing the eye to be examined 200, and a first dichroic mirror 202 and a second dichroic mirror 203 are placed on its optical axis. The optical path from the objective lens 201 is branched by these dichroic mirrors into an optical path 250 for the OCT optical system, an optical path 251 for the SLO optical system and fixation lamp, and an optical path 252 for anterior ocular observation for each wavelength band. .

In this embodiment, an optical path 252 for anterior eye observation is arranged in the reflection direction of the first dichroic mirror 202, and an optical path 250 for the OCT optical system, an SLO optical system, and a fixation lamp are arranged in the transmission direction of the first dichroic mirror 202. An optical path 251 is arranged. Further, an optical path 250 for the OCT optical system is arranged in the reflection direction of the second dichroic mirror 203, and an optical path 251 for the SLO optical system and the fixation lamp is arranged in the transmission direction of the second dichroic mirror 203. However, the arrangement of each optical path with respect to the dichroic mirror may be reversed.

The optical path 251 for the SLO optical system and fixation lamp includes an SLO scanning unit 204, lenses 205 and 206, a mirror 207, a third dichroic mirror 208, an APD (Avalanche Photodiode) 209, an SLO light source 210, and a fixation lamp 211. It is provided. The mirror 207 is configured using a prism in which a perforated mirror or a hollow mirror is vapor-deposited, and separates the illumination light from the SLO light source 210 and the return light of the illumination light from the eye 200 to be examined. The third dichroic mirror 208 branches the optical path from the third dichroic mirror 208 into an optical path of the SLO light source 210 and an optical path of the fixation lamp 211 for each wavelength band. In this embodiment, the optical path of the SLO light source 210 is arranged in the reflection direction of the third dichroic mirror 208, and the fixation lamp 211 is arranged in the transmission direction of the third dichroic mirror 208. However, the arrangement of each optical path with respect to the dichroic mirror may be reversed.

The SLO scanning unit 204 scans the subject's eye 200 with illumination light emitted from the SLO light source 210, and includes an X scanner that scans in the X direction and a Y scanner that scans in the Y direction. The SLO scanning unit 204 is controlled by the imaging control unit 101-2. In this embodiment, the X scanner is a polygon mirror that performs high-speed scanning, and the Y scanner is a galvanometer mirror. However, the configuration of the SLO scanning unit 204 is not limited to this, and the X scanner and Y scanner may be configured using any deflection mirror depending on the desired configuration.

The lens 205 is driven in the optical axis direction by a motor (not shown) in order to focus the SLO optical system and the fixation lamp 211. Note that the motor for driving the lens 205 is controlled by the photographing control section 101-2.

SLO light source 210 generates light with a wavelength around 780 nm. APD 209 detects the return light from eye 200 to be examined. The fixation lamp 211 generates visible light to encourage the subject to fixate.

The illumination light emitted from the SLO light source 210 is reflected by the third dichroic mirror 208, passes through the mirror 207, passes through the lenses 206 and 205, and reaches the SLO scanning unit 204. scanned. After the return light from the eye 200 to be examined returns along the same path as the illumination light, it is reflected by the mirror 207 and guided to the APD 209 .

The APD 209 generates signal data of the SLO fundus image based on the returned light and outputs it to the image processing device 101. The acquisition unit 101-1 of the image processing device 101 generates an SLO image based on the signal data output from the APD 209. Note that an SLO image of the fundus of the eye to be examined 200 can be acquired by scanning the fundus of the eye to be examined 200 with the illumination light emitted from the SLO light source 210. On the other hand, an SLO image of the anterior segment of the eye to be examined 200 can also be acquired by scanning the anterior segment of the eye to be examined with the illumination light emitted from the SLO light source 210.

The light emitted from the fixation lamp 211 passes through the third dichroic mirror 208 and the mirror 207, passes through the lenses 206 and 205, and forms a predetermined shape at an arbitrary position on the eye 200 by the SLO scanning unit 204. By having the subject gaze at the light emitted from the fixation lamp 211, it is possible to encourage the subject to fixate.

Lenses 212 and 213, a split prism 214, and a CCD 215 for anterior eye observation that detects infrared light are arranged in the optical path 252 for anterior eye observation. The CCD 215 is sensitive to the wavelength of irradiation light for anterior ocular segment observation (not shown), specifically around 970 nm. The data signal output from the CCD 215 is output to the image processing device 101, and the image processing device 101 can generate an anterior segment observation image based on the input signal data.

The split prism 214 is arranged at a position conjugate with the pupil of the eye 200 to be examined. The image processing device 101 can detect the distance of the measuring optical system 100-1 in the Z-axis direction (optical axis direction) with respect to the eye 200 using the split image of the anterior segment based on the light that has passed through the split prism 214.

An OCT optical system is provided in the optical path 250 for the OCT optical system, and the OCT optical system is used to take a tomographic image of the eye 200 to be examined. More specifically, OCT optics are used to obtain interference signals for generating tomographic images. The optical path 250 for the OCT optical system is provided with an XY scanner 216, lenses 217 and 218, and an optical fiber 224.

The XY scanner (OCT scanning unit) 216 is used to scan the eye 200 to be examined with measurement light. The XY scanner 216 is controlled by the imaging control section 101-2. Although the XY scanner 216 is illustrated as a single mirror in FIG. 2, it is actually a galvanometer mirror that scans in two XY axis directions. However, the configuration of the XY scanner 216 is not limited to this, and may be configured using any deflection mirror depending on the desired configuration. For example, the XY scanner 216 may be configured using any deflection means such as a MEMS mirror that can scan measurement light in two-dimensional directions with one piece.

The lens 217 is driven in the optical axis direction by a motor (not shown) in order to focus the measurement light emitted from the optical fiber 224 connected to the optical coupler 219 onto the eye 200 to be examined. By this focusing, the return light of the measurement light from the eye 200 to be examined is simultaneously formed into a spot image and incident on the tip of the optical fiber 224. Note that the motor for driving the lens 217 is controlled by the photographing control section 101-2.

Next, the configuration of the optical path from the OCT light source 220, the reference optical system, and the spectrometer will be explained. The OCT optical system further includes an OCT light source 220, a reference mirror 221, a dispersion compensating glass 222, a lens 223, an optical coupler 219, single mode optical fibers 224 to 227 connected to and integrated with the optical coupler, and a spectrometer 230. It is being The OCT optical system constitutes a Michelson interferometer with these configurations.

The OCT light source 220 is an SLD (Super Luminescent Diode), which is a typical low coherent light source. The center wavelength is 855 nm, and the wavelength bandwidth is approximately 100 nm. Here, the bandwidth is an important parameter because it affects the resolution of the obtained tomographic image in the optical axis direction. As for the type of light source, SLD is selected here, but it is sufficient that it can emit low coherent light, and ASE (Amplified Spontaneous Emission) or the like may be used. Considering that the eye is to be measured, near-infrared light is suitable as the center wavelength. Furthermore, since the center wavelength affects the lateral resolution of the obtained tomographic image, it can be as short as possible. In this embodiment, the center wavelength of the OCT light source 220 is set to 855 nm for both reasons.

The light emitted from the OCT light source 220 passes through the optical fiber 225 and is split into measurement light that enters the optical fiber 224 side via the optical coupler 219 and reference light that enters the optical fiber 226 side. The measurement light is irradiated onto the eye 200 to be observed, which is the object to be observed, through the optical path 250 for the OCT optical system, and reaches the optical coupler 219 as return light through the same optical path due to reflection and scattering by the eye 200 to be examined. On the other hand, the reference light reaches the reference mirror 221 and is reflected through the optical fiber 226, the lens 223, and the dispersion compensating glass 222 inserted to match the wavelength dispersion of the measurement light and the reference light. The reference light reflected by the reference mirror 221 returns along the same optical path and reaches the optical coupler 219.

The measurement light and the reference light are combined by an optical coupler 219 to become interference light. Here, the measurement light and the reference light cause interference when the optical path length of the measurement light and the reference light become almost the same. The reference mirror 221 is held so as to be adjustable in the optical axis direction by a motor and drive mechanism (not shown) that are controlled by the imaging control unit 101-2, and is capable of matching the optical path length of the reference light to the optical path length of the measurement light. be. The interference light is guided to a spectrometer 230 via an optical fiber 227.

Further, polarization adjusting sections 228 and 229 are provided in the optical fibers 224 and 226. Polarization adjustment units 228 and 229 adjust the polarization of the measurement light and the reference light, respectively. The polarization adjustment units 228 and 229 have several portions in which optical fibers are routed around in a loop. In the polarization adjustment units 228 and 229, by rotating this loop-shaped portion around the longitudinal direction of the optical fiber, the optical fiber is twisted, and the polarization states of the measurement light and the reference light can be adjusted and matched. .

The spectrometer 230 is provided with lenses 232 and 234, a diffraction grating 233, and a line sensor 231. The interference light emitted from the optical fiber 227 becomes parallel light through the lens 234 , is separated by the diffraction grating 233 , and is imaged onto the line sensor 231 by the lens 232 .

The interference light is measured by the line sensor 231 as intensity information for each wavelength. Intensity information for each wavelength measured by the line sensor 231 is output to the image processing device 101 as tomographic image signal data (interference signal). The image processing device 101 can generate a tomographic image of the eye 200 using the received signal data.

In this embodiment, a Michelson interferometer is used as the interferometer, but a Mach-Zehnder interferometer may also be used. For example, depending on the difference in light amount between the measurement light and the reference light, a Mach-Zehnder interferometer can be used when the difference in light amount is large, and a Michelson interferometer can be used when the difference in light amount is relatively small.

(High image quality processing)
Hereinafter, high image quality processing for motion contrast data acquired by OCT will be specifically described, but the terms used in the description of the embodiments will be briefly defined. First, three-dimensional volume data information regarding interference signals and brightness tomographic images based thereon will be referred to as OCT data, and three-dimensional volume data information regarding motion contrast will be referred to as OCTA data. Further, two-dimensional information that can be extracted from three-dimensional volume data regarding interference signals and tomographic images of brightness based thereon is referred to as an OCT image, and two-dimensional information that can be extracted from three-dimensional volume data regarding motion contrast is referred to as an OCTA image. In particular, an image generated by projecting or integrating three-dimensional volume data regarding an interference signal and a brightness tomographic image based on the interference signal in a specified depth direction range is referred to as an OCT front image (brightness En-Face). Further, an image generated by projecting or integrating three-dimensional volume data regarding motion contrast is referred to as an OCTA frontal image. Furthermore, a two-dimensional image including data in the depth direction is referred to as a tomographic image.

(High image quality department)
FIG. 3 is a block diagram showing a more detailed configuration of the image quality improvement section 101-47. The image quality improvement section 101-47 is provided with an image quality improvement processing section 301, a region detection section 302, an ROI setting section 303, a blend processing section 304, and a BC adjustment section 305.

The image quality improvement processing unit 301 performs image quality improvement processing on the medical image acquired by the acquisition unit 101-1 using a trained model for image quality improvement (image quality improvement engine, image quality improvement model), which will be described later. and generate high-quality images. The image quality improvement processing unit 301 performs image quality improvement processing on the medical image of the subject (first image) using a trained model obtained by learning using the medical image of the subject as learning data. It can function as an example of a high-quality image unit that acquires a high-quality image (second image) of the subject. The medical images to be processed for high image quality include OCT images, OCTA images, tomographic images, OCT frontal images, OCTA frontal images, SLO images, anterior ocular observation images, and analysis maps obtained by analyzing these images. It may be.

The region detection unit 302 performs region detection processing on a medical image or a high-quality medical image using a trained model for region detection (region detection engine, region detection model) to be described later, and detects at least one region. (for example, a target area). For example, the area detection unit 302 can detect the target area and areas other than the target area. The area detection unit 302 can function as an example of a detection unit that detects a target area in a medical image or a high-quality image of a subject. For example, the region detection unit 302 detects perfusion regions and non-perfusion regions from the OCTA frontal image using a region detection engine. When diagnosing the fundus, by identifying perfused and non-perfused areas, it is possible to detect whether there is no blood flow where there should be blood vessels, or whether some kind of blood flow is observed where there should be no blood vessels (such as new blood vessels). can be judged. Note that the target area is not limited to the perfusion area and the non-perfusion area, but may include any area where over-correction occurs by the image quality engine. For example, the target region may include the foveal avascular region, the optic nerve head, etc. in addition to the perfused region and non-perfused region.

Note that the classification of regions may be managed using attribute information (labels) indicating each region. For example, the area detection unit 302 may generate a label image having attribute information as a pixel value corresponding to each pixel position of the medical image. In addition to the attribute information, a value indicating the likelihood (reliability, probability) of each attribute information output from the region detection engine may be stored in association with each pixel of the medical image.

The ROI setting unit 303 sets a ROI (Region of Interest) based on the region detected by the region detection unit 302. Note that the ROI can be at least one of the regions (target regions) detected by the region detection unit 302. Note that the ROI may be automatically set as a region having predetermined attribute information by the region detection section 302, and in this case, the ROI setting section 303 may be omitted.

For example, the blend processing unit 304 performs blend processing, which is a type of image processing, on the set ROI, which is a region having predetermined attribute information, using a medical image and a high-quality image of the medical image. As the blending process, for example, any known process such as α blending process may be used. The blending processing unit 304 performs a process for a target region in a medical image or a high-quality image so that the difference between the pixel value of the target region and the pixel value of a region other than the target region is widened, and the pixel value of the target region is lowered. It can function as an example of an image processing unit that performs image processing.

The BC adjustment unit 305 performs, for example, a BC (Brightness, Contrast) adjustment process, which is a correction process for at least one of brightness and contrast, on the set ROI, which is a region having predetermined attribute information. Here, the contrast correction processing may include any known contrast correction processing, and may include, for example, correction using a tone curve or gamma curve, and level correction. The BC adjustment unit 305 also adjusts the target area in a medical image or a high-quality image so that the difference between the pixel value of the target area and the pixel value of an area other than the target area increases and the pixel value of the target area becomes lower. It can function as an example of an image processing unit that performs image processing.

Note that only one of the blend processing section 304 and the BC adjustment section 305 may be provided. Further, when both the blend processing unit 304 and the BC adjustment unit 305 are provided, the image processing device 101 performs either blend processing or BC adjustment processing as the image processing to be performed in accordance with instructions from the operator. The configuration may be such that both can be selected. Note that when performing both the blending process and the BC adjustment process, the image processing apparatus 101 displays the images subjected to each image process on the output unit 103 by switching or arranging them according to operations on the UI, etc. may be configured.

(High-quality medical image engine)
The image quality improvement engine used by the image quality improvement processing unit 301 will be described below with reference to FIGS. 4 to 7. In this embodiment, the image quality improvement engine is stored in the storage unit 101-3, but the image quality improvement engine may be provided in an external device connected to the image processing apparatus 101. In this case, the image quality improvement processing unit 301 can perform image quality improvement processing using an image quality improvement engine provided in an external device.

The image quality improvement engine according to this embodiment is a learned model obtained by performing training (learning) on a machine learning algorithm. In this embodiment, for training a machine learning model related to a machine learning algorithm, input data is a low-quality image with specific shooting conditions assumed to be processed, and output data is a high-quality image corresponding to the input data. We use training data consisting of a group of pairs. Note that the specific imaging conditions specifically include a predetermined imaging site, imaging method, imaging angle of view, image size, and the like.

Here, a general trained model will be briefly explained. A trained model is a machine learning model that has been trained (learning) in advance using appropriate learning data for any machine learning algorithm. The learning data consists of one or more pairs of input data and output data (correct data). Note that the formats and combinations of the input data and output data of the pair groups that make up the learning data may be such that one is an image and the other is a numerical value, one is composed of multiple image groups and the other is a character string, or both. may be an image, etc., which is suitable for the desired configuration.

Specifically, for example, learning data (hereinafter referred to as first learning data) configured by a group of pairs of an image acquired by OCT and a radiographed part label corresponding to the image may be mentioned. Note that the imaging site label is a unique numerical value or character string that represents the site. In addition, as an example of other learning data, it is made up of a pair group of a noisy low-quality image obtained by normal OCT imaging and a high-quality image obtained by capturing multiple times using OCT and processing it to improve the image quality. Examples include learning data (hereinafter referred to as second learning data).

At this time, when input data is input to the learned model, output data according to the design of the learned model is output. A trained model outputs output data that is likely to correspond to input data, for example, according to a tendency trained using learning data. In addition, the trained model can, for example, output the likelihood (reliability, probability) corresponding to the input data as a numerical value for each type of output data, according to the tendency trained using the learning data. I can do it.

Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first learning data, the machine learning model outputs the imaged region label of the imaged region captured in the image. or output the probability for each imaging site label. Also, for example, if a noisy, low-quality image obtained by normal OCT imaging is input to a machine learning model trained with the second learning data, the machine learning model will acquire high image quality by capturing multiple times using OCT. Outputs a high-quality image equivalent to the processed image. Note that, from the viewpoint of quality maintenance, the machine learning model can also be configured so that its own output data is not used as learning data.

Further, the machine learning algorithm includes a method related to deep learning such as a convolutional neural network (CNN). In methods related to deep learning, if the parameter settings for the layers and nodes that make up the neural network differ, the extent to which trends trained using learning data can be reproduced in output data may vary. For example, in a deep learning machine learning model using the first learning data, if more appropriate parameters are set, the probability of outputting a correct imaging site label may become higher. Further, for example, in a deep learning machine learning model using the second learning data, if more appropriate parameters are set, a higher quality image may be output.

Specifically, parameters in CNN include, for example, the kernel size of the filter, the number of filters, the stride value, and the dilation value set for the convolutional layer, as well as the number of output nodes of the fully connected layer. etc. can be included. Note that the parameter group and the number of training epochs can be set to values preferable for the usage pattern of the learned model based on the learning data. For example, based on the learning data, it is possible to set a parameter group and the number of epochs that can output correct imaging site labels with a higher probability or output higher quality images.

An example of a method for determining such a parameter group and the number of epochs will be described. First, 70% of the pairs forming the learning data are used for training, and the remaining 30% are randomly set for evaluation. Next, a machine learning model is trained using the training pair group, and at the end of each epoch of training, a training evaluation value is calculated using the evaluation pair group. The training evaluation value is, for example, the average value of a group of values obtained by evaluating the output when the input data forming each pair is input to the machine learning model being trained and the output data corresponding to the input data using a loss function. be. Finally, the parameter group and epoch number when the training evaluation value becomes the smallest are determined as the parameter group and epoch number of the machine learning model. Note that by dividing the pairs that make up the learning data into those for training and evaluation and determining the number of epochs in this way, the machine learning model may overfit the pairs for training. can be prevented.

Here, the image quality improvement engine according to the present embodiment is configured as a module that outputs a high quality image obtained by enhancing the quality of an input low quality image. Here, high image quality in this specification refers to converting an input image into an image with a quality more suitable for image diagnosis. refers to an image that In addition, low-quality images include, for example, two-dimensional images or three-dimensional images obtained by X-ray photography, CT, MRI, OCT, PET, or SPECT, or three-dimensional moving images of continuously captured CT, etc. This photo was taken without any high-quality settings. Specifically, low-quality images include, for example, images obtained by low-dose imaging using an X-ray imaging device or CT, imaging by MRI that does not use a contrast agent, short-time imaging with OCT, etc. Contains OCTA images etc. acquired several times.

In addition, when high-quality images with low noise or high contrast are used for image analysis such as blood vessel analysis processing of images such as OCTA, and region segmentation processing of images such as CT and OCT, low-quality images can be used. In many cases, analysis can be performed more accurately than when using Therefore, high-quality images output by the high-quality engine may be useful not only for image diagnosis but also for image analysis.

Further, the content of image quality suitable for image diagnosis depends on what is desired to be diagnosed by various image diagnoses. Therefore, it is difficult to make a general statement, but for example, image quality suitable for image diagnosis is one that has little noise, high contrast, shows the photographed subject in colors and gradations that are easy to observe, and has a large image size. Including image quality such as high resolution. Furthermore, the image quality can include images in which non-existent objects and gradations that were drawn during the image generation process have been removed from the image.

In the image processing method constituting the image quality improvement method by the image quality improvement processing unit 301 in this embodiment, processing using various machine learning algorithms such as deep learning is performed. In addition to processing using machine learning algorithms, this image processing method uses existing arbitrary methods such as various image filter processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge-based image processing. You may also perform the following processing.

Hereinafter, with reference to FIG. 4, a configuration example of the CNN related to the high image quality engine according to the present embodiment will be described. FIG. 4 shows an example of the configuration of the image quality improvement engine. The configuration shown in FIG. 4 is composed of a plurality of layer groups that perform processing to process and output a group of input values. Note that, as shown in FIG. 4, the types of layers included in the configuration include a convolution layer, a downsampling layer, an upsampling layer, and a merging layer.

The convolution layer is a layer that performs convolution processing on a group of input values according to set parameters such as the kernel size of the filter, the number of filters, the stride value, and the dilation value. Note that the number of dimensions of the kernel size of the filter may also be changed depending on the number of dimensions of the input image. The downsampling layer is a process of thinning out or combining input value groups to make the number of output value groups smaller than the number of input value groups. Specifically, for example, there is Max Pooling processing. The upsampling layer is a process of making the number of output value groups larger than the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. Specifically, for example, there is linear interpolation processing. The compositing layer is a layer that receives a group of values, such as a group of output values of a certain layer or a group of pixel values constituting an image, from a plurality of sources, and performs a process of combining them by concatenating or adding them.

In such a configuration, the group of pixel values forming the input image Im410 is outputted through the convolution processing block and the group of pixel values forming the input image Im410 are combined in the synthesis layer. Thereafter, the combined pixel value group is formed into a high-quality image Im420 in the final convolution layer. Although not shown, examples of changes to the CNN configuration include incorporating a batch normalization layer or an activation layer using a rectifier linear unit after the convolution layer. You may.

Here, the GPU can perform efficient calculations by processing more data in parallel. For this reason, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in this embodiment, a GPU is used in addition to a CPU for processing by the image processing unit 101-4, which is an example of a learning unit. Specifically, when a learning program including a learning model is executed, learning is performed by the CPU and GPU working together to perform calculations. Note that in the processing of the learning section, calculations may be performed only by the CPU or GPU. Further, the image quality improvement processing section 301 may also be implemented using a GPU, similar to the learning section.

Further, the learning section may include an error detection section and an updating section (not shown). The error detection unit obtains an error between output data output from the output layer of the neural network and correct data according to input data input to the input layer. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the correct data. Furthermore, the updating section updates the connection weighting coefficients between the nodes of the neural network, etc., based on the error obtained by the error detection section, so that the error becomes smaller. This updating unit updates the connection weighting coefficients and the like using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting connection weighting coefficients between nodes of each neural network so that the above-mentioned error is reduced.

Note that when using some image processing techniques, such as image processing using CNN, it is necessary to be careful about the image size. Specifically, in order to prevent problems such as the peripheral areas of high-quality images not being sufficiently high-quality, it should be noted that different image sizes may be required for input low-quality images and output high-quality images. Should.

Although not specified in this embodiment for the sake of clear explanation, if a high image quality engine is adopted that requires different image sizes for the input image to the high image quality engine and the output image, the image size may be changed as appropriate. It is assumed that the Specifically, we perform padding on input images, such as images used as learning data for training machine learning models and images input to high-quality engines, and image capturing areas around the input images. Adjust the image size by merging. Note that the area to be padded is filled with a constant pixel value, filled with neighboring pixel values, or mirror padded, depending on the characteristics of the image quality improvement method, so that the image quality can be effectively improved.

Further, the image quality improvement method by the image quality improvement processing unit 301 may be performed using only one image processing method, or may be performed by combining two or more image processing methods. Alternatively, a plurality of image quality improvement method groups may be implemented in parallel to generate a plurality of high quality image groups, and then the highest quality image may be finally selected as the high image quality image. Note that the selection of the highest quality image may be automatically performed using an image quality evaluation index, or by displaying a plurality of high quality image groups on a UI provided in the output unit 103 or the like. It may be performed according to instructions from the examiner (operator).

Note that input images that have not been improved in quality may be more suitable for image diagnosis, and therefore may be added to the final image selection target. Furthermore, parameters may be input to the high-quality image together with the low-quality image. For example, a parameter specifying the degree of image quality enhancement or a parameter specifying the image filter size used in the image processing method may be input to the image quality enhancement engine together with the input image.

Here, the input data of the learning data of the image quality improvement engine according to the present embodiment is a low image quality image acquired with the same model as the image capturing apparatus 100 and the same settings as the image capturing apparatus 100. Further, the output data of the learning data of the high image quality engine is a high quality image obtained through settings and image processing related to shooting conditions that require more man-hours than those provided by the same model. Specifically, the output data is, for example, a high-quality image (superimposed image) obtained by performing superposition processing such as averaging on a group of images (original images) obtained by shooting multiple times. can do.

Here, high-quality images and low-quality images will be explained using OCTA motion contrast data as an example. Motion contrast data is data used in OCTA and the like, which is obtained by repeatedly photographing the same part of a photographic subject and detecting temporal changes in the photographic subject between the photographing sessions. In addition, as mentioned above, by generating a frontal image using data in a desired range in the depth direction of the imaging target out of the calculated motion contrast data (an example of three-dimensional medical image data), OCTA En-Face images (OCTA frontal images) can be generated. Note that, hereinafter, the number of times OCT data at substantially the same position (substantially the same location) is repeatedly imaged will be referred to as NOR (Number of Repeats).

Regarding the learning data according to the present embodiment, two different methods will be described as examples of generating high-quality images and low-quality images by superposition processing with reference to FIGS. 5(a) and 5(b). Note that the method of generating high-quality images and low-quality images is not limited to these, and any known generation method may be used.

A first method related to an example of generating high-quality images and low-quality images will be described with reference to FIG. 5(a). In the first method, as an example of a high-quality image, a motion contrast image generated from OCT data obtained by repeatedly photographing substantially the same position of a photographic target is used. In FIG. 5A, a motion contrast image Im510 shows a three-dimensional motion contrast image (three-dimensional motion contrast data). Further, the motion contrast image Im511 indicates a two-dimensional motion contrast image (two-dimensional motion contrast data) that constitutes a three-dimensional motion contrast image.

The tomographic images Im501-1 to Im501-3 indicate OCT tomographic images (B-scan images) for generating the motion contrast image Im511. Here, in FIG. 5A, NOR corresponds to the number of OCT tomographic images in tomographic images Im501-1 to Im501-3, and in the example shown in the figure, NOR is 3. The tomographic images Im501-1 to Im501-3 are taken at predetermined time intervals (Δt). Note that the substantially same position refers to one line in the front direction (XY) of the subject's eye, and corresponds to the position of the motion contrast image Im511 in FIG. 5(a). Note that the front direction is an example of a direction intersecting the depth direction.

Since motion contrast data is data that detects temporal changes, NOR must be performed at least twice in order to generate this data. For example, if NOR is 2, one motion contrast data is generated. When NOR is 3, when motion contrast data is generated using only OCT data of adjacent time intervals (first and second, second and third), two motion contrast data are generated. When motion contrast data is generated using OCT data obtained at separate time intervals (first and third times), a total of three motion contrast data are generated. That is, as the NOR is increased 3 times, 4 times, etc., the number of motion contrast data at substantially the same position also increases. A high-quality motion contrast image can be generated by aligning a plurality of motion contrast images obtained by repeatedly photographing substantially the same position and performing superimposition processing such as averaging. Therefore, in order to generate a high quality motion contrast image, NOR can be performed at least three times or more, and to obtain a higher quality motion contrast image, for example, NOR can be performed five times or more.

On the other hand, as an example of a low-quality image corresponding to this, a motion contrast image before being subjected to overlapping processing such as averaging can be used. In this case, the low-quality image can be used, for example, as a reference image when performing superposition processing such as averaging to generate a high-quality image. When performing superimposition processing, if the position and shape of the target image are transformed and aligned with respect to the reference image, there will be almost no spatial misalignment between the reference image and the image after superposition processing. . Therefore, it is possible to easily create a pair of a low-quality image and a high-quality image. Note that the target image subjected to image deformation processing for alignment may be used as a low-quality image instead of the reference image.

By using each of the original image groups (reference image and target image) as input data and the corresponding superimposed image as output data, a plurality of pair groups can be generated. For example, when obtaining one superimposed image from a group of 15 original images, a pair of the first original image of the original image group and the superimposed image, a pair of the second original image of the original image group, Pairs with superimposed images can be generated. In this way, when one superimposed image is obtained from a group of 15 original images, it is possible to generate 15 pairs of one image from the original image group and the superimposed image. Note that three-dimensional high-quality image data can be generated by repeatedly photographing substantially the same position in the main scanning (X) direction and scanning while shifting it in the sub-scanning (Y) direction.

Next, a second method related to an example of generating high-quality images and low-quality images will be described with reference to FIG. 5(b). In the second method, a high-quality image is generated by superimposing motion contrast images obtained by photographing substantially the same area of the photographing target multiple times. Note that approximately the same area refers to an area such as 3 x 3 mm or 10 x 10 mm in the front direction (XY) of the eye to be examined, and by photographing the approximately same area of the subject multiple times, A three-dimensional motion contrast image (three-dimensional motion contrast data) including the depth direction of the image can be obtained. When the same area is photographed multiple times and superimposed processing is performed, NOR can be set to two or three times in order to shorten the number of times each photograph is taken.

Furthermore, in order to generate high-quality three-dimensional motion contrast data, at least two pieces of three-dimensional data of the same area are acquired. FIG. 5(b) shows an example of a plurality of three-dimensional motion contrast images. The motion contrast images Im520 to Im540 are three-dimensional motion contrast images, similar to the motion contrast image Im510 described with reference to FIG. 5(a). Using these two or more three-dimensional motion contrast images, alignment processing is performed in the frontal direction (X-Y) and depth direction (Z), and after removing artifact data from each data, averaging processing is performed. I do. Thereby, it is possible to generate one high-quality three-dimensional motion contrast image from which artifacts are removed.

On the other hand, the corresponding low-quality image can be used as reference data when performing overlay processing such as averaging. As explained in the first method, since there is almost no spatial positional shift between the reference image and the image after averaging, they can easily be paired as a low-quality image and a high-quality image. Note that an arbitrary three-dimensional motion contrast image generated from target data subjected to image deformation processing for alignment instead of reference data may be used as a low-quality image.

In the first method, the imaging itself is completed in one time, so there is less burden on the subject. However, as the number of NOR operations increases, the time required for one imaging session becomes longer. Furthermore, if artifacts such as cloudy eyes or eyelashes are introduced during photography, a good image may not necessarily be obtained. In the second method, the burden on the subject increases slightly because imaging is performed multiple times. However, it takes less time to take one shot, and even if an artifact appears in one shot, as long as the artifact does not appear in another shot, you can ultimately obtain a clear image with fewer artifacts. In view of these characteristics, when collecting data, any method can be selected depending on the situation of the subject.

In this embodiment, a motion contrast image is used as an example of a low-quality image and a high-quality image used as learning data, but the images used as learning data are not limited to this. Since OCT data is acquired to generate motion contrast data, it is possible to similarly generate low-quality images and high-quality images using OCT data. Furthermore, although the description of the tracking process has been omitted in this embodiment, since substantially the same position or area of the eye to be examined is photographed, it is also possible to perform imaging while tracking the eye to be examined. Tracking processing may be performed by any known method.

If a pair of three-dimensional high-quality image data and low-quality image data can be obtained, an arbitrary pair of two-dimensional images can be generated from them. For example, a high-quality OCTA plane image can be generated by projecting or integrating the generated high-quality three-dimensional motion contrast image in a desired depth range to generate an arbitrary OCTA front image. Further, the corresponding low-quality image can be any OCTA front image generated from reference data when performing overlay processing such as averaging. In this case as well, since there is almost no spatial positional shift between the reference image and the image after averaging, it is possible to easily create a pair of a low-quality image and a high-quality image. Note that an arbitrary motion contrast front image generated from target data subjected to image deformation processing for alignment instead of reference data may be used as a low-quality image.

An example of a pair of two-dimensional images used as such learning data will be described in more detail with reference to FIGS. 6(a) and 6(b). For example, if the image used as the learning data is an OCTA frontal image, the OCTA frontal image can be generated by projecting or integrating three-dimensional volume data related to motion contrast in a desired depth range, as described above. can. Here, the depth range is a range in the Z direction shown in FIGS. 5(a) and 5(b).

FIG. 6(a) shows an example of an OCTA frontal image. As the OCTA frontal images used for the learning data, use OCTA frontal images generated in different depth ranges, such as the superficial layer (image Im610), deep layer (image Im620), outer layer (image Im630), and choroidal vascular network (image Im640). I can do it. Note that the types of OCTA front images are not limited to these, and the number of types may be increased by generating OCTA front images in which different depth ranges are set by changing the reference layer and offset values. When performing learning, you can train each OCTA front image at different depths separately, or you can train by combining multiple images with different depth ranges (for example, separately for the superficial side and deep side). Alternatively, OCTA front images of all depth ranges may be learned together. When using luminance En-Face images generated from OCT data as learning data, a plurality of En-Face images generated from any depth range can be used similarly to the OCTA front image.

For example, consider a case where the image quality improvement engine includes a trained model obtained using training data including a plurality of OCTA frontal images corresponding to different depth ranges of the eye to be examined. At this time, the acquisition unit 101-1 can acquire, as the first image, an OCTA front image corresponding to a part of a long depth range including different depth ranges. That is, an OCTA front image corresponding to a depth range different from a plurality of depth ranges corresponding to a plurality of OCTA front images included in the learning data can be used as an input image during image quality enhancement processing. Of course, an OCTA front image having the same depth range as that during learning may be used as an input image during image quality enhancement processing. Further, some of the depth ranges may be set in response to an operator pressing an arbitrary button on the UI, or may be set automatically. Note that the above-mentioned content is not limited to OCTA front images, but can also be applied to, for example, brightness En-Face images.

Note that when the image to be processed by the trained model is a tomographic image, learning is performed using an OCT tomographic image that is a B-scan image or a tomographic image of motion contrast data as learning data. This will be explained with reference to FIG. 6(b). In FIG. 6(b), images Im651 to Im653 are OCT tomographic images (brightness tomographic images). The images in FIG. 6B are different because the tomographic images are shown at different positions in the sub-scanning (Y) direction. For tomographic images, learning may be performed together without worrying about the difference in position in the sub-scanning direction. However, if the images were taken at different locations (for example, the center of the macula or the center of the optic disc), you may choose to learn each location separately, or you can learn the images together without worrying about the location. It may also be possible to learn from Note that since the image feature amounts are significantly different between an OCT tomographic image and a tomographic image of motion contrast data, it is better to perform learning separately.

As the high-quality image used as the output data of the learning data, for example, a superimposed image as described above can be used. In the superimposed image that has been subjected to the superimposition process, pixels that are commonly depicted in the original image group are emphasized, resulting in a high-quality image suitable for image diagnosis. In this case, the generated high-quality image is a high-contrast image with clear differences between low-brightness areas and high-brightness areas, as a result of highlighting commonly drawn pixels. Furthermore, for example, in the superimposed images, random noise that occurs every time a photograph is taken can be reduced, and an area that was not well depicted in the original image at a certain point in time can be interpolated with another group of original images.

Furthermore, when the superimposed image is used as output data for learning data, it is also possible to generate a low-quality image from the superimposed image to be used as input data for learning data. In this case, for example, the superimposed images are first downsampled to lower their resolution, and then the lowered resolution images are upsampled using a known method (nearest neighbor method, bilinear method, etc.), and then the training data is It can be input data. It is also possible to configure a high image quality engine that improves the sense of resolution by performing learning using such a group of image pairs as learning data.

Furthermore, if input data for a machine learning model needs to be composed of a plurality of images, a necessary number of original image groups can be selected from the original image group and used as input data. For example, when obtaining one superimposed image from a group of 15 original images, if two images are required as input data for a machine learning model, it is possible to generate a group of 105 (15C2 = 105) pairs. be.

It should be noted that among the group of pairs constituting the learning data, pairs that do not contribute to higher image quality can be removed from the learning data. For example, if a high-quality image that is the output data that makes up a pair of training data has an image quality that is not suitable for image diagnosis, the image that is output by the high-quality engine that trained using the training data is also not suitable for image diagnosis. This may result in poor image quality. Therefore, by removing pairs whose output data has an image quality unsuitable for image diagnosis from the learning data, it is possible to reduce the possibility that the image quality improvement engine generates an image with an image quality unsuitable for image diagnosis.

In addition, if the average brightness or brightness distribution of a group of images in a pair is significantly different, the image quality improvement engine trained using the training data will detect images that are not suitable for image diagnosis and have brightness distributions that are significantly different from those of low-quality images. may be output. Therefore, pairs of input data and output data that have significantly different average brightness or brightness distribution can be removed from the learning data.

Furthermore, if the structure or position of the photographed object drawn in a pair of images is significantly different, the high image quality engine trained using the training data will There is a possibility of outputting an image that is not suitable for image diagnosis. Therefore, pairs of input data and output data that differ greatly in the structure or position of the imaged object to be drawn can be removed from the learning data. Further, from the viewpoint of quality maintenance, the high-quality image engine can be configured not to use the high-quality images that it outputs as learning data.

By using the high image quality engine that has been trained in this way, the high image quality processing unit 301 can improve contrast and reduce noise through superimposition processing when medical images acquired in one shot are input. It is possible to output a high-quality image that has undergone reduction or the like. Therefore, the high-quality image processing unit 301 can generate a high-quality image suitable for image diagnosis based on the low-quality image that is the input image.

Although an example in which a superimposed image is used as the output data of the learning data has been described here, the output data of the learning data of the image quality improvement engine is not limited to this. The output data of the learning data may be any high-quality image that corresponds to the input data, such as images with noise reduction suitable for diagnosis, images with contrast correction, images with high resolution, or images that require more man-hours. The image may be an image taken under many shooting conditions. Furthermore, an image obtained by performing image processing using statistical processing such as maximum a posteriori probability estimation (MAP estimation) processing on a low-quality image used as input data can also be used as output data of learning data. Note that any known method may be used to generate the high-quality image.

Further, as the image quality improvement engine, a plurality of image quality improvement engines may be prepared, each of which independently performs various image quality improvement processes such as noise reduction, contrast correction, and resolution enhancement. Alternatively, one image quality enhancement engine that performs at least two image quality enhancement processes may be provided. Note that in these cases, high-quality images according to the desired processing may be used as the output data of the learning data. For example, regarding a high-quality image that performs individual processing, high-quality images that have been subjected to individual processing such as noise reduction processing may be used as output data of learning data. Further, regarding the image quality improvement engine that performs a plurality of image quality improvement processes, for example, a high quality image that has been subjected to noise reduction processing, contrast correction processing, etc. may be used as the output data of the learning data.

(Region detection engine for medical images)
Next, the area detection engine for medical images used by the area detection unit 302 will be described using a contrast front image as an example. The region detection unit 302 according to the present embodiment detects regions by classifying them into perfusion regions and non-perfusion regions in the contrast frontal image, for example, in order to confirm the presence or absence of blood flow. Hereinafter, the non-perfusion area will be referred to as NPA (Non Perfusion Area). Further, the region detection unit 302 according to the present embodiment may detect, for example, a foveal avascular zone (FAZ) or an optic nerve head (ONH) as the region. As information about the detected area, a label of the area can be associated with a pixel in the image. Deep learning that associates labels with all pixels in an image in this way is called semantic segmentation.

Generally, in deep learning, the work of creating output data (correct data) of learning data is called annotation. Regarding the training data related to the trained model for region detection, output data can be created by giving a label indicating the region to be classified for each pixel position of the input data or the corresponding image using annotation. . An image having information indicating the label of each region generated in this way as a pixel value is called a region label image. All of the annotation work may be performed manually, or some of the annotation work may be performed automatically. It is also possible to increase the efficiency of annotation work by performing deep learning as needed after a certain amount of work is completed, and by correcting labels while referring to the segmentation results from the region detection engine in the interim. Note that when creating a region label image, not only the input data but also other information may be used. For example, when creating a region label image that will be the output data of training data with input data as a high-quality image, you can create the region label image by referring to the image before the high-quality image, etc. Good too.

In this way, a trained model that has been subjected to machine learning using training data that is a group of pairs of input data, which is a predetermined number of medical images, and output data, which is region label images corresponding to the medical images, is used as a region detection engine. be able to. Note that the region detection engine trained in this manner can output the likelihood (reliability, probability) of the region label for each pixel. Therefore, regarding the probability of the labels output from the region detection engine, the region detection unit 302 can output, for example, a label with a higher probability than other labels as the region detection result. Further, among the labels, a label with a probability higher than a threshold value can be output as a detection result. At this time, if there are multiple labels with a probability higher than the threshold, all of them may be output, or one of them with a probability higher than the other labels may be output as a detection result. Furthermore, the area detection unit 302 may determine the detection result using a machine learning model from the probability of each label obtained using the learned model. The machine learning algorithm used in this case may be a different type of machine learning algorithm than the machine learning algorithm used to obtain the label probabilities, such as a neural network, support vector machine, AdaBoost, Bayesian network, or It may be a random forest or the like.

Note that a common model may be used as the image quality improvement engine and the area detection engine. In this case, for example, a known U-Net model can be used, and by using the same image quality improvement engine and area detection engine, each inference process can be performed simply by replacing parameters.

Furthermore, as the input data for region detection, the high-quality image (overlaid image) described above may be used, or a high-quality image obtained by applying a high-quality engine to the input medical image may be used as input data. May be used. In this way, by performing learning according to the input data, it is possible to configure a region detection engine that is optimal for each input data.

Note that the region detection engine does not necessarily need to be an artificial intelligence engine based on machine learning, and may be configured using a rule-based algorithm based on the structure of the eye to be examined, etc., including various conventional filter processes. . Therefore, the area detection engine used by the area detection unit 302 may extract the FAZ and NPA areas by combining filter processing such as a known Gaussian filter, or may extract the FAZ and NPA areas in response to instructions from an operator via a GUI or the like. Areas may also be extracted. Further, the region detection engine may detect regions in the motion contrast image by classifying the regions into perfusion regions and non-perfusion regions, for example, by threshold processing using the structure of the subject's eye and a predetermined threshold.

(Image processing procedure for medical images)
Next, a series of image processing procedures by the image processing apparatus 101 according to the present embodiment will be described with reference to FIG. FIG. 7 is a flow diagram schematically showing the flow of image processing according to this embodiment.

When the image processing according to this embodiment is started, the acquisition unit 101-1 acquires a medical image in step S701. The acquisition unit 101-1 may acquire a medical image from the imaging device 100 or an external device, or may acquire a medical image generated using signal data acquired from these. In this embodiment, the acquisition unit 101-1 acquires, for example, a contrast front image as a medical image. Note that the medical image is not limited to this, and may be a tomographic image, a luminance En-Face image, an SLO image, an analysis image (analysis map) obtained by analyzing a medical image of a subject, or the like.

In step S702, the image quality improvement processing unit 301 applies image quality improvement processing, which is first image processing, to the acquired medical image using at least one image quality improvement engine described above. Obtain quality medical images. The image quality improvement processing unit 301 according to the present embodiment uses an image quality improvement engine to obtain a high quality OCTA front image from an OCTA front image.

Next, in step S703, the area detection unit 302 performs area detection processing on the medical image acquired in step S701 or the high-quality medical image acquired in step S702 using the area detection engine described above. Apply. The region detection unit 302 detects at least two regions in the medical image or the high-quality medical image through the region detection processing, and acquires information indicating each region. For example, the region detection unit 302 may obtain a region label image in which each pixel is labeled with a label indicating each region for a detected region, or a region label image in which each pixel information is labeled with a label indicating each region in a medical image. may be added. Note that the area detection unit 302 only needs to be able to output information indicating the detected area, and the information indicating the detected area is not limited to the format described above. For example, the area detection unit 302 may be information on a label associated with pixel information of a medical image. Furthermore, as described above, values indicating the likelihood (reliability, probability) of each attribute outputted from the area detection engine may be associated with each pixel of the medical image and added to the pixel information. In this embodiment, a region detection engine is used to detect a perfusion region and a non-perfusion region (NPA) in an OCTA frontal image or a high-quality OCTA frontal image. Note that the pixel positions of the detected areas in the medical image or the high-quality medical image may correspond to each other in the medical image and the high-quality medical image.

When the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the high-quality medical image based on the output information. Note that the ROI may be set for at least one of the regions detected by the region detection unit 302. In this embodiment, the ROI setting unit 303 sets the ROI for the NPA in the high-quality OCTA frontal image. Note that, as described above, the ROI may be set by the region detection unit 302.

Finally, in step S704, the blend processing unit 304 or the BC adjustment unit 305 increases the difference between the pixel value of the ROI and the pixel value of the area other than the ROI for the ROI in the high-quality medical image. Apply second image processing. At this time, the blend processing unit 304 or the BC adjustment unit 305 adjusts the ROI in the high-quality medical image so that the pixel value of the ROI is lower than the pixel value of the ROI before the second image processing. Apply image processing in step 2. For example, the blend processing unit 304 performs second image processing on the ROI in the high-quality medical image and the medical image acquired in step S701 and the high-quality medical image acquired in step S702. Performs blending processing. The blending process may be performed using any known method, and the blending ratio may be a predetermined ratio or may be set according to an operator's instruction. By performing the blending process, the image before over-correction and the image with over-correction are blended for the area where over-correction has occurred due to high image quality processing using the trained model, and the over-correction is removed. The difference between pixel values in the area where this occurs and pixel values in other areas increases. In other words, by performing the blending process, the pixel value of the area where over-correction has occurred becomes lower. With such processing, the pixel values can be reduced only in the area where overcorrection has occurred, without reducing the pixel values of the entire image. Therefore, for example, overcorrection in areas such as NPA and FAZ can be suppressed.

Furthermore, the BC adjustment unit 305 performs BC adjustment processing as second image processing on the ROI in the high-quality medical image. The BC adjustment process may be performed using any known method; for example, the brightness in the ROI may be set to a predetermined value, or may be increased or decreased by a predetermined value. Further, in the ROI, contrast may be adjusted using, for example, a tone curve or a gamma curve. Further, in the BC adjustment process, at least one of brightness and contrast in the ROI may be set to a value according to an instruction from an operator, or may be increased or decreased by the value. For example, the BC adjustment unit 305 may determine a brightness correction value or a setting value of a tone curve or the like used for contrast adjustment in response to an operator's instruction. By performing the BC adjustment process, the pixel values of the over-corrected area and the pixel values of other areas are calculated for the area where over-correction has occurred due to the high image quality processing using the learned model. At least one of brightness and contrast is adjusted to widen the difference. In other words, by performing the BC adjustment processing, the pixel value of the area where overcorrection occurs becomes lower. With such processing, the pixel values can be reduced only in the area where overcorrection has occurred, without reducing the pixel values of the entire image. Therefore, for example, overcorrection in areas such as NPA and FAZ can be suppressed.

In this embodiment, the blend processing unit 304 performs blending processing of the OCTA front image and the high quality OCTA front image on the NPA set as the ROI in the high quality OCTA front image. Furthermore, the BC adjustment unit 305 performs BC adjustment processing on the high-quality OCTA front image with respect to the NPA set as the ROI in the high-quality OCTA front image. Note that, as described above, either one of the processing by the blend processing unit 304 and the processing by the BC adjustment unit 305 may be performed, or both may be performed. The high-quality image subjected to the second image processing by the blend processing unit 304 and the BC adjustment unit 305 is displayed on the output unit 103 by the display control unit 101-5, or outputted by the output unit 103 to an external device, etc. , may be stored in the storage unit 101-3 or the external storage device 102.

By performing the image processing according to this embodiment in this way, an area where over-correction occurs due to image processing of a medical image using a trained model is detected, and an image in which the over-correction is reduced for the detected area is created. can be processed. For example, it is possible to detect NPA in an OCTA frontal image and perform blending processing or BC adjustment processing to reduce overcorrection of the NPA in a high-quality OCTA frontal image acquired using a trained model.

As mentioned above, NPA is particularly important in diagnosing fundus images. By identifying NPA, it is possible to determine whether there is no blood flow where there should be a blood vessel, or whether some kind of blood flow is observed where there should be no blood vessel (such as a new blood vessel). In particular, with regard to FAZ and NPA, by leaving some of the original state before high image quality, it is possible to support the operator's (examiner) judgment regarding the distinction between noise and blood vessels in image diagnosis.

On the other hand, visibility can also be improved by setting the brightness and contrast to a desired state and actively removing noise and the like to make the NPA look more like an NPA. In this case as well, it is possible to support the operator's judgment regarding the distinction between noise and blood vessels in image diagnosis.

Note that since diagnosis is generally performed in a complex manner including other tests, the second image processing method may be set in advance or may be selected as appropriate. Further, the second image processing method may be changed depending on the area to be classified. Furthermore, it is also possible to change the settings depending on the eye to be examined. Further, the settings may be changed depending on the depth of the OCTA front image, that is, the shallow layer and the deep layer. These settings may be stored for each user or subject.

As described above, the image processing apparatus 101 according to the present embodiment includes the image quality improvement processing section 301, the area detection section 302, the blend processing section 304, and the BC adjustment section 305. The image quality improvement processing unit 301 uses an image quality improvement model (image quality improvement engine) obtained through learning using the subject's medical images as learning data to enhance the first image of the subject's medical images. Image quality processing is performed to obtain a high-quality second image of the medical image of the subject. The region detection unit 302 detects a target region in a medical image (first image or second image) of the subject. At least one of the blend processing unit 304 and the BC adjustment unit 305 adjusts the target area in the second image so that the difference between the pixel value of the target area and the pixel value of the area other than the target area increases. Image processing is performed so that the pixel value of is lower than the pixel value of the target area before image processing.

As image processing, the blend processing unit 304 blends the first image and the second image so that the difference between the pixel value of the target area and the pixel value of the area other than the target area increases, and the pixel value of the target area becomes lower. Performs blending processing to blend with the image. In addition, as image processing, the BC adjustment unit 305 adjusts the brightness and contrast so that the difference between the pixel value of the target area and the pixel value of the area other than the target area increases and the pixel value of the target area becomes lower. BC adjustment processing is performed to correct at least one of them. Note that the medical image of the subject may be, for example, a motion contrast image of the subject's eye, and the target area may include, for example, at least one of a non-perfused area, a foveal blood vessel area, and an optic disc area. Further, the region detection unit 302 can detect a target region using a trained model obtained by learning using a medical image of a subject as learning data.

With such a configuration, the image processing apparatus according to the present embodiment detects a region such as NPA where over-correction occurs due to image processing of a medical image using a trained model, and performs the over-correction on the detected region. It is possible to perform image processing to reduce the With such processing, the pixel values can be reduced only in the area where overcorrection has occurred, without reducing the pixel values of the entire image. Therefore, over-correction due to image processing of medical images using the learned model can be reduced. Thereby, in image diagnosis, it is possible to support the operator's judgment regarding the distinction between noise and blood vessels.

Here, various modifications are possible regarding the series of image processing according to this embodiment. Hereinafter, a specific example of a series of image processing according to this embodiment will be described in detail with reference to FIGS. 8 to 13. Note that for each example, description of processes similar to the above-described image processing shown in FIG. 7 will be omitted.

<First example of image processing method>
A first example of the image processing method will be described with reference to FIG. 8. In this example, an image processing method for suppressing overcorrection by the image quality engine for FAZ and NPA will be described.

The processing in step S801 and step S802 is the same as that in step S701 and step S702, so a description thereof will be omitted. When a high-quality medical image is acquired in step S802, the process moves to step S803.

In step S803, the area detection unit 302 uses the area detection engine to detect the FAZ and NPA areas in the medical image (input image) acquired in step S801. FAZ and NPA may be detected separately or may be detected together as one area. Further, only one of the regions may be detected, or regions other than the FAZ and NPA may be detected as perfusion regions. After the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the medical image that has been improved in image quality in step S802 based on the output information. Note that the pixel positions of the detected areas in the medical image can be assumed to correspond in the medical image and the high-quality medical image.

Finally, in step S804, the blend processing unit 304 uses the input image and the high-quality medical image acquired in step S802 to perform a blending process on the detected region in the high-quality medical image. conduct. Here, the blending process may be performed using a known α blending process or the like. Further, the blending ratio may be a fixed value, or the blending processing unit 304 may change the blending ratio for each pixel according to the reliability (likelihood, probability) regarding the attribute information output by the area detection engine. Specifically, for a pixel having attribute information of a non-perfused area, the α blend ratio can be set such that the higher the reliability, the higher the blend ratio of the input image.

Further, when attribute information such as FAZ and NPA is detected separately, the blending ratio of each may be changed depending on the differentiated attribute information. Furthermore, the input image may be suitably combined with pixels in areas other than the FAZ and NPA, for example, surrounding pixels of the target area such as the FAZ and NPA. This makes it possible to reduce sharp changes at area boundaries.

Note that the blend ratio may be configured so that the operator can instruct it via the UI. For example, a GUI such as a slide bar may be used to set a blend ratio corresponding to the overcorrection suppression strength.

According to this example, regarding FAZ and NPA, the original state before the image quality improvement can be maintained to some extent. Therefore, in image diagnosis, it is possible to support the operator's judgment regarding the distinction between noise and blood vessels.

Note that when detecting a target region in a medical image (first image) that is an input image, the region detection unit 302 can use a region detection engine configured by a trained model as the region detection engine. . In this case, as learning data for the region detection engine, the input data is a medical image that is the input image, and the output data is a region label image that has information indicating the label of each region in the medical image that is the input image as a pixel value. be able to. Further, the region detection unit 302 can also use a region detection engine configured using a rule-based algorithm based on the structure of the subject. In this case as well, the region detection unit 302 detects target regions such as FAZ and NPA from characteristic parts in medical images that have not been subjected to high image quality using a learned model, in other words, that have not been overcorrected. can be detected.

<Second example of image processing method>
A second example of the image processing method will be described with reference to FIG. 9. In this example, an image processing method for emphasizing high image quality processing for NPA will be described.

When the performance of the area detection engine is high, the area detection unit 302 can more appropriately detect areas such as FAZ and NPA. For example, in FAZ and NPA, it is true that there are no blood vessels in nature. Therefore, when the performance of the region detection engine is high, images more suitable for diagnosis can be obtained by further darkening these regions by BC adjustment after processing with the high image quality engine. The processing of this example, which can obtain images more suitable for diagnosis using such image processing, will be described in more detail below.

The processing in step S901 and step S902 is the same as that in step S701 and step S702, so the explanation will be omitted. When a high-quality medical image is acquired in step S902, the process moves to step S903.

In step S903, the area detection unit 302 uses the area detection engine to detect the FAZ and NPA areas in the medical image (input image) acquired in step S901. FAZ and NPA may be detected separately, or may be detected together as one area. Further, only one of the regions may be detected, or a region other than the FAZ and NPA may be detected as the perfusion region. After the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the medical image that has been improved in image quality in step S902 based on the output information. Note that the pixel positions of the detected areas in the medical image can be assumed to correspond in the medical image and the high-quality medical image.

Finally, in step S904, the BC adjustment unit 305 performs BC adjustment processing on pixels in the detected areas (FAZ and NPA areas) in the high-quality medical image acquired using the high-quality image engine in step S902. I do. The BC adjustment may be performed by correcting the luminance value (brightness) related to the pixel value to make it darker. For example, the brightness value may be simply lowered or set to zero. Further, image processing such as gamma correction may be applied depending on the original pixel value. Note that the strength of correction may be changed for each of FAZ and NPA, and if the attribute and reliability are held as information indicating the detected area, BC adjustment may be performed according to the reliability.

Note that the operator may be able to instruct the BC adjustment method and intensity via the UI. For example, the method may be selected using a GUI such as radio buttons, or the strength may be specified using a GUI such as a slide bar.

According to this example, it is also possible to improve the visibility of areas such as the NPA and FAZ. In this case as well, it is possible to support the doctor's judgment regarding the distinction between noise and blood vessels in image diagnosis. Furthermore, the suppression process shown in the first example and the emphasis process shown in the second example may be selected for each region attribute. For example, FAZ may be an emphasis process, and NPA may be a suppression process. In this case, it is possible to generate an image more suitable for diagnosis according to each region, and it is possible to support the doctor's judgment.

<Third example of image processing method>
A third example of the image processing method will be described with reference to FIG. 10. In this example, image processing will be described in which an area detection engine is used to detect areas such as FAZ and NPA from a medical image whose image quality has been improved using an image quality improvement engine, and over-correction in these areas is suppressed.

The processing in step S1001 and step S1002 is the same as step S701 and step S702, so a description thereof will be omitted. When a high-quality medical image is acquired in step S1002, the process moves to step S1003.

In step S1003, the region detection unit 302 uses the region detection engine to detect the FAZ and NPA regions in the medical image whose image quality has been improved in step S1002. FAZ and NPA may be detected separately or may be detected together as one area. Further, only one of the regions may be detected, or regions other than the FAZ and NPA may be detected as perfusion regions. After the region detection unit 302 outputs the information indicating the detected region, the ROI setting unit 303 sets the ROI in the medical image that has been improved in image quality in step S1002 based on the output information.

Finally, in step S1004, the blending processing unit 304 performs a blending process on the detected region in the high-quality medical image using the high-quality medical image and the medical image acquired in step S1001. conduct. The blending process may be similar to the process described in the first example.

In the case of this example as well, as in the first example, regarding FAZ and NPA, the original state before image quality improvement can be maintained to some extent. Therefore, in image diagnosis, it is possible to support the operator's judgment regarding the distinction between noise and blood vessels.

Note that when detecting a target region in a high-quality medical image, the region detection unit 302 can use a region detection engine configured by a trained model. In this case, the learning data for the region detection engine is a high-quality medical image (second image) as input data, and information indicating the label of each region in the medical image (first image) that is the input image. A region label image having pixel values as pixel values can be used as output data. Here, the high-quality medical image serving as input data is a medical image that has been high-quality using a high-quality engine, and is a high-quality image in which over-correction has occurred in target areas such as NPA. be able to. On the other hand, the region label image serving as the output data may be an image obtained by labeling a medical image before the image quality is improved using an image quality improvement engine. In this case, the region detection engine configured by the trained model can output a region label image containing the label of the target region where over-correction has occurred from the input high-quality image according to the learning trend. . Furthermore, by using high-quality images as input, the region detection engine configured with the trained model will be able to more appropriately extract features within the image, and it is expected that it will be able to output more accurate region label images. be done.

<Fourth example of image processing method>
A fourth example of the image processing method will be described with reference to FIG. 11. In this example, we will explain image processing that uses an area detection engine to detect areas such as FAZ and NPA from a medical image whose image quality has been improved using a high image quality engine, and emphasizes the image quality improvement processing for these areas. do.

The processing in step S1101 and step S1102 is the same as step S701 and step S702, so a description thereof will be omitted. When a high-quality medical image is acquired in step S1102, the process moves to step S1103.

In step S1103, the area detection unit 302 uses the area detection engine to detect the FAZ and NPA areas in the medical image whose image quality has been improved in step S1102. FAZ and NPA may be detected separately or may be detected together as one area. Further, only one of the regions may be detected, or regions other than the FAZ and NPA may be detected as perfusion regions. After the region detection unit 302 outputs the information indicating the detected region, the ROI setting unit 303 sets the ROI in the medical image that has been improved in image quality in step S1102 based on the output information.

Finally, in step S1104, the BC adjustment unit 305 performs BC adjustment processing on pixels in the detected area (FAZ or NPA area) in the high-quality medical image. The BC adjustment process may be similar to the process described in the second example.

In this example, as in the second example, it is also possible to improve the visibility of areas such as the NPA and FAZ. In this case as well, it is possible to support a doctor's judgment regarding the distinction between noise and blood vessels in image diagnosis. Furthermore, the suppression processing shown in the first example or the third example and the emphasis processing shown in the fourth example may be selected for each region attribute. For example, FAZ may be an emphasis process, and NPA may be a suppression process. In this case, it is possible to generate an image more suitable for diagnosis according to each region, and it is possible to support a doctor's judgment.

<Fifth example of image processing method>
A fifth example of the image processing method will be described with reference to FIG. 12. In this example, we use an area detection engine to detect areas such as FAZ and NPA on a medical image before image quality enhancement, perform BC adjustment on these areas, and then use the image quality enhancement engine to detect areas such as FAZ and NPA. Image processing that applies image quality enhancement processing will be described.

The processing in step S1201 is the same as step S701, so a description thereof will be omitted. In step S1202, the area detection unit 302 uses the area detection engine to detect the FAZ and NPA areas in the medical image (input image) acquired in step S1201. After the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets the ROI in the input image based on the output information.

Next, in step S1203, the BC adjustment unit 305 performs BC adjustment processing on pixels in the detected area (FAZ or NPA area) in the input image. Note that the BC adjustment process may be similar to the process described in the second example.

Finally, in step S1204, the image quality improvement processing unit 301 uses the image quality improvement engine to perform image quality improvement processing on the medical image on which the BC adjustment process was performed in step S1203, and the image quality is improved. Acquire medical images. Note that blending processing and BC adjustment processing based on the detected area may be further applied to the medical image after the image quality enhancement processing by the image quality enhancement processing unit 301. The blending process and the BC adjustment process may be similar to the processes described in the first example and the second example, respectively.

As described above, in this example, the region detection unit 302 detects the target region in the first image of the medical image of the subject. In addition, the BC adjustment unit 305 adjusts the target area in the first image of the medical image of the subject so that the difference between the pixel value of the target area and the pixel value of the area other than the target area widens. BC adjustment processing is performed so that the pixel value is lower than the pixel value of the target area before BC adjustment processing. Further, the image quality improvement processing unit 301 performs image quality improvement processing on the first image that has undergone the BC adjustment process, using the image quality improvement model obtained through learning using the medical image of the subject as learning data. and obtain a high-quality second image of the medical image of the subject.

In the case of this example, image quality enhancement processing is performed on the medical image in which the pixel values in the area where overcorrection occurs are lowered by performing the BC adjustment processing. With such processing, the pixel values can be lowered only in the area where overcorrection occurs, without lowering the pixel values of the entire image. Therefore, for example, overcorrection in areas such as NPA and FAZ can be suppressed, and visibility of areas such as NPA and FAZ can also be improved. In this case as well, it is possible to support the doctor's judgment regarding the distinction between noise and blood vessels in image diagnosis.

<Sixth example of image processing method>
A sixth example of the image processing method will be described with reference to FIG. 13. In this example, an area detection engine is used to perform area detection on a medical image whose image quality has been enhanced using an image quality improvement engine. After that, for the medical image before the image quality is improved, BC adjustment is performed based on the area detected by the above-mentioned area detection, and then image processing is performed in which image quality improvement processing is applied using the image quality improvement engine. explain.

The processing in step S1301 and step S1302 is the same as step S701 and step S702, so a description thereof will be omitted. When the high-quality medical image is acquired in step S1302, the process moves to step S1303.

In step S1303, the area detection unit 302 uses the area detection engine to perform area detection processing on the image whose image quality has been improved in step S1302. Note that the detected area may be FAZ, NPA, etc., as in the first to fifth examples. In this way, by applying the region detection engine to medical images of high quality, it is expected that detection performance can be improved. After the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the input image based on the output information. Note that the pixel positions of the detected areas in the high-quality medical image can be assumed to correspond in the medical image and the high-quality medical image.

Next, in step S1304, the BC adjustment unit 305 performs BC adjustment processing on the medical image (input image) acquired in step S1301 based on the area detected in step S1303. Note that the BC adjustment process may be similar to the process described in the second example.

Finally, in step S1305, the image quality improvement processing unit 301 uses the image quality improvement engine to perform image quality improvement processing on the medical image on which the BC adjustment processing was performed in step S1303, and the image quality is improved. Acquire medical images. Note that blending processing or BC adjustment based on region classification may be further applied to the image after the image quality enhancement processing by the image quality enhancement processing unit 301 in step S1305. The blending process and BC adjustment may be the same processes as those described in the first example and the second example, respectively.

As described above, in this example, the image quality improvement processing unit 301 uses the image quality improvement model obtained by learning using the medical image of the subject as learning data to improve the image quality of the medical image of the subject. An image obtained by increasing the quality of the first medical image of the subject is obtained from the first image. The region detection unit 302 detects a target region in an image obtained by enhancing the quality of the first image of the medical image of the subject. The BC adjustment unit 305 adjusts the pixel value of the target area so that the difference between the pixel value of the target area and the pixel value of the area other than the target area increases with respect to the target area in the first image of the medical image of the subject. The BC adjustment process is performed so that the pixel value of the target area is lower than the pixel value of the target area before the BC adjustment process. Furthermore, the image quality improvement processing unit 301 uses the image quality improvement model to perform image quality improvement processing on the first image on which the BC adjustment processing has been performed, and improves the quality of the second image of the medical image of the subject. Get the image of. Note that the area detection unit 302 detects the target area using an image obtained by increasing the quality of the first image of the medical image of the subject on which the BC adjustment has not been performed by the BC adjustment unit 305.

In the case of this example, image quality enhancement processing is performed on the medical image in which the pixel values in the area where overcorrection occurs are lowered by performing the BC adjustment processing. With such processing, the pixel values can be lowered only in the area where overcorrection occurs, without lowering the pixel values of the entire image. Therefore, for example, overcorrection in areas such as NPA and FAZ can be suppressed, and visibility of areas such as NPA and FAZ can also be improved. In this case as well, it is possible to support the doctor's judgment regarding the distinction between noise and blood vessels in image diagnosis.

As described above, various modifications are possible as image processing using the image quality improvement engine and the area detection engine. These illustrated examples are susceptible to many variations and are not intended to be construed as limiting the disclosure. For example, when performing follow-up observation of medical images, blend processing and BC adjustment processing based on detected regions may be performed under the same conditions, or these conditions may be stored and managed for each subject.

[Second embodiment]
An image processing system including an image processing apparatus according to a second embodiment of the present invention will be described in detail with reference to FIG. 14. FIG. 14 is a flow diagram showing the flow of a series of image processing according to this embodiment. Note that the configuration of the image processing system according to this embodiment is similar to the configuration of the image processing system according to the first embodiment, so the same reference numerals will be used and the description will be omitted. However, in the image quality improvement unit according to this embodiment, the region detection unit 302, ROI setting unit 303, blend processing unit 304, and BC adjustment unit 305 may not be provided. The image processing system according to this embodiment will be described below, focusing on the differences from the image processing system according to the first embodiment.

In the first embodiment, a method was described for obtaining a more suitable high-quality image by combining a high-quality image engine using deep learning and a region detection engine and performing additional partial image correction. . In contrast, in this embodiment, a configuration will be described in which image quality improvement processing is performed using an image quality improvement engine constructed by applying these methods.

The machine learning algorithm of the image quality improvement engine according to this embodiment may be the same as the machine learning algorithm of the image quality improvement engine according to the first embodiment. However, the high-quality image finally generated by any of the methods described in the first embodiment is used as the output data of the learning data of the high-quality image engine according to the present embodiment. These processing flows are applied to a predetermined number of medical images to output high-quality images, and a learning data set is constructed by pairing each image. It can be expected that a high-quality image obtained by learning using such learning data can generate high-quality images in which over-correction is suppressed, following the learning tendency. Additionally, by performing machine learning all at once, including partially corrected results, it is expected that the image processing load during inference can be reduced.

The processing procedure of the image processing apparatus 101 of this embodiment will be described with reference to FIG. 14. First, in step S1401, the acquisition unit 101-1 acquires a medical image. The process in step S1401 may be similar to the process in step S701.

Next, in step S1402, the image quality enhancement processing unit 301 uses the image quality enhancement engine obtained through learning using the learning data as described above to enhance the image quality of the medical image acquired in step S1401. Perform processing and obtain high-quality images. The high-quality image acquired in this manner becomes a high-quality image in which over-correction is suppressed in accordance with the learning tendency of the high-quality image engine.

As described above, the image processing apparatus according to this embodiment includes the acquisition section 101-1 and the image quality improvement processing section 301. The acquisition unit 101-1 acquires a first image of the medical images of the subject. The image quality improvement processing unit 301 performs image quality improvement processing using a first trained model obtained by learning using a medical image of a subject as learning data, and blending processing or brightening processing in a target area of a medical image of a subject. Image quality enhancement processing is performed on the first image using a second trained model obtained through learning using, as training data, a medical image of the subject that has been subjected to correction processing of at least one of image quality and contrast. and acquire a second image of the medical images of the subject. In this case as well, a high-quality image with suppressed overcorrection can be generated.

Note that by performing various types of image analysis using medical images whose image quality has been improved by one of the methods according to the first embodiment and the second embodiment, more accurate analysis can be performed. Furthermore, for disease screening using an artificial intelligence engine, high-quality images can be used as input images, and it is expected that more accurate processing will be possible.

[Modification 1]
In the first embodiment, a plurality of front images corresponding to a plurality of depth ranges can be used as learning data when the region detection unit 302 detects a target region using a trained model. For example, a plurality of OCTA front images corresponding to a plurality of depth ranges can be used as input data for learning data of a region detection engine using a trained model. Note that as the output data of the learning data, a region label image corresponding to the OCTA front image used as the input data may be used. The area detection unit 302 uses a common learned model obtained by learning such learning data to detect an object corresponding to an arbitrary depth range, for example, a depth range selected according to an instruction from the examiner. A two-dimensional target area can be detected from the frontal image of the specimen. Note that the depth range may include, for example, the superficial layer, deep layer, outer layer, choroidal vascular network, and different depth ranges in which the reference layer and offset value are changed. Further, the front image is not limited to an OCTA front image, but may be an En-Face image.

Furthermore, a plurality of trained models corresponding to the depth ranges may be prepared using learning data for each depth range. In this case, the area detection unit 302 selects one of a plurality of trained models obtained through learning using a plurality of frontal images of the subject corresponding to a plurality of depth ranges as learning data, in response to instructions from the examiner. A trained model corresponding to the selected depth range may be selected, and a two-dimensional target area may be detected from the frontal image of the subject using the selected trained model. Further, the region detection unit 302 selects a trained model corresponding to the depth range of the medical image used for region detection processing from among the plurality of trained models, and uses the selected trained model to detect the target. A two-dimensional target area may be detected from the frontal image of the specimen.

Furthermore, a combination of a plurality of frontal images of different depth ranges may be used as learning data (for example, a plurality of frontal images are divided into a surface layer side and a deep layer side). In this case as well, the region detection unit 302 selects a trained model corresponding to the selected depth range or the depth range of the medical image used for region detection processing according to instructions from the examiner, and A two-dimensional target area can be detected from a frontal image of a subject using the following method.

Furthermore, a three-dimensional image may be used as input data for learning data of a region detection engine using a trained model. For example, a 3-dimensional tomographic image or a 3-dimensional motion contrast image is used as the input data for the learning data, and a 3-dimensional region label image that has been labeled for the 3-dimensional image is used as the output data for the learning data. be able to. In this case, the region detection unit 302 can detect a three-dimensional target region from the three-dimensional image of the subject using a trained model obtained by learning using the three-dimensional image of the subject as learning data.

Similarly, the image quality improvement engine may be trained using learning data in which a three-dimensional image is used as input data and a three-dimensional image with improved image quality is used as output data. In this case, the image quality improvement processing unit 301 uses an image quality improvement engine obtained through learning using a three-dimensional medical image of the subject as learning data to perform high-quality image processing from the three-dimensional medical image of the subject. Dimensional images can be obtained. Furthermore, the blend processing unit 304 and the BC adjustment unit 305 can be configured to perform various processes on a three-dimensional target area in a three-dimensional medical image. In this case, the same effects as in the first embodiment can be achieved by processing the three-dimensional medical image. Similarly, a three-dimensional image may be used as learning data for the image quality improvement engine according to the second embodiment. In this case, the same effects as in the second embodiment can be achieved by processing the three-dimensional medical image.

[Modification 2]
Learning is performed to adjust (tuning) the trained models used for high image quality processing and region detection processing (trained models for high image quality, trained models for region detection) for each subject, and A dedicated trained model may also be generated. For example, using medical images acquired in past examinations of a subject, a general-purpose trained model for generating high-quality medical images or transfer of a general-purpose trained model for detecting regions It is possible to perform learning and generate a trained model specifically for that subject. By linking the trained model dedicated to the examinee with the ID of the examinee and storing it in the storage unit 101-3 or an external device such as a server, the image processing device 101 can perform the current examination of the examinee. When performing this, it is possible to specify and use a trained model dedicated to the examinee based on the examinee's ID. By using a trained model dedicated to the subject, it is possible to improve the accuracy of image quality enhancement processing and area detection processing.

[Modification 3]
A high image quality engine may be prepared for each type of image that is input data. For example, a high-quality model for anterior eye images, a high-quality model for SLO images, a high-quality model for tomographic images, a high-quality model for OCTA frontal images, etc. may be prepared. Furthermore, for OCTA frontal images and En-Face images, high-quality models may be prepared for each depth range for generating images. For example, a high image quality model for the surface layer, a high image quality model for the deep layer, etc. may be prepared. Furthermore, the high image quality model may be one that is trained on images for each imaging region (for example, the center of the macula, the center of the optic disc), or may be one that is trained regardless of the imaging region. .

At this time, for example, the image quality of the fundus OCTA front image is improved using a high-quality model obtained by learning the fundus OCTA front image as learning data, and the image quality is further increased by learning the anterior eye OCTA front image as learning data. The image quality of the anterior eye OCTA front image may be increased using the high image quality model. Furthermore, the high-image quality model may be one that is trained regardless of the part to be imaged. Here, for example, the fundus OCTA en-face image and the anterior eye OCTA en-face image may be relatively similar in the distribution of blood vessels to be photographed. In this way, in a plurality of types of medical images in which the appearance of the photographic subject is relatively similar to each other, the feature amounts may be relatively similar to each other. Therefore, for example, by using a high-quality model obtained by learning the fundus OCTA frontal image as learning data, it is possible to not only improve the image quality of the fundus OCTA frontal image but also to improve the image quality of the anterior eye OCTA frontal image. may be done. For example, by using a high-quality model obtained by learning anterior eye OCTA frontal images as learning data, it is possible to not only improve the image quality of anterior eye OCTA frontal images, but also to improve the image quality of fundus OCTA frontal images. may be configured. That is, by using a high-quality model obtained by learning at least one type of frontal image, ie, a fundus OCTA frontal image and an anterior eye OCTA frontal image, as learning data, the fundus OCTA frontal image and the anterior eye OCTA frontal image are It may be configured to be able to improve the image quality of at least one type of frontal image.

Here, a case will be considered in which an OCT apparatus capable of photographing the fundus of the eye is also capable of photographing the anterior eye. At this time, as the OCTA En-Face image, for example, a fundus OCTA front image may be applied in the fundus photography mode, and an anterior eye OCTA front image may be applied in the anterior segment photography mode. At this time, when the high image quality button is pressed, in the fundus photography mode, for example, in the display area of the OCTA En-Face image, the difference between the low quality fundus OCTA front image and the high quality fundus OCTA front image is displayed. It may be configured such that one display is changed to the other display. In addition, when the high image quality button is pressed, for example, in the anterior segment imaging mode, a low quality anterior eye OCTA front image and a high quality anterior eye OCTA front image are displayed in the display area of the OCTA En-Face image. The display of one of these may be changed to the display of the other.

In addition, in an OCT apparatus capable of photographing the fundus, when the anterior eye can also be photographed, an anterior eye adapter may be configured to be attachable. Furthermore, the optical system of the OCT apparatus may be configured to be movable over a distance approximately equal to the axial length of the eye to be examined, without using an anterior ocular adapter. At this time, the focus position of the OCT apparatus may be configured to be able to be changed significantly toward the emmetropic side to the extent that an image is formed on the anterior eye.

Further, as the tomographic image, for example, a fundus OCT tomographic image may be applied in the fundus imaging mode, and an anterior ocular OCT tomographic image may be applied in the anterior segment imaging mode. Furthermore, the above-described high-quality processing for the fundus OCTA frontal image and anterior eye OCTA frontal image can also be applied as, for example, high-quality processing for the fundus OCT tomographic image and the anterior eye OCT tomographic image. At this time, when the high image quality button is pressed, for example, in the fundus imaging mode, one of the low quality fundus OCT tomographic image and the high quality fundus OCT tomographic image is displayed in the tomographic image display area. The display may be changed to the other display. In addition, when the high image quality button is pressed, for example, in the anterior segment imaging mode, one of the low quality anterior eye OCT tomographic image and the high quality anterior eye OCT tomographic image is displayed in the tomographic image display area. The display may be changed to the other display.

Further, as the tomographic image, for example, a fundus OCTA tomographic image may be applied in the fundus imaging mode, and an anterior eye OCTA tomographic image may be applied in the anterior segment imaging mode. Furthermore, the above-described high-quality processing for the fundus OCTA frontal image and the anterior eye OCTA frontal image can also be applied as, for example, high-quality processing for the fundus OCTA tomographic image and the anterior eye OCTA tomographic image. At this time, in the fundus imaging mode, for example, in the tomographic image display area, information indicating a blood vessel region in the fundus OCTA tomographic image (for example, motion contrast data above a threshold) is superimposed on the fundus OCT tomographic image at the corresponding position. It may be configured so that it is displayed as follows. Further, for example, in the anterior segment imaging mode, information indicating a blood vessel region in the anterior eye OCTA tomographic image may be displayed superimposed on the anterior eye OCT tomographic image at the corresponding position in the tomographic image display area. .

In this way, for example, if the feature values (states of the photographic subject) of multiple types of medical images are considered to be relatively similar, the The image quality improvement model obtained by learning medical images as learning data may be used to improve the image quality of at least one type of medical image among a plurality of types of medical images. Thereby, for example, it is possible to configure the image quality improvement of a plurality of types of medical images to be executable using a common trained model (common image quality improvement model).

Note that the display screen in the fundus photography mode and the display screen in the anterior segment photography mode may have the same display layout, or may have a display layout corresponding to each photography mode. Various conditions such as imaging conditions and analysis conditions may be the same or different between the fundus imaging mode and the anterior segment imaging mode.

Here, the target image for the image quality enhancement process may be, for example, a plurality of OCTA front images (an OCTA En-Face image, a motion contrast En-Face image) (corresponding to a plurality of depth ranges). Furthermore, the target image for the image quality enhancement process may be, for example, one OCTA front image corresponding to one depth range. Furthermore, instead of the OCTA frontal image, the target image for image quality enhancement processing may be, for example, a brightness frontal image (luminance En-Face image), an OCT tomographic image that is a B-scan image, or a tomographic image of motion contrast data ( OCTA tomographic image). In addition, target images for image quality enhancement processing are not only OCTA frontal images, but also various medical images such as OCT tomographic images, which are brightness frontal images and B-scan images, and motion contrast data tomographic images (OCTA tomographic images). It may be an image. That is, the target image for the image quality enhancement process may be at least one of various medical images displayed on the display screen of the output unit 103, for example. At this time, for example, since the feature amount of an image may differ depending on the type of image, a trained model for image quality improvement corresponding to each type of target image for image quality improvement processing may be used. For example, when a high image quality button is pressed in response to an instruction from the examiner, the OCTA frontal image is not only processed to improve its image quality using a trained model for high image quality corresponding to the OCTA frontal image. It may also be configured to perform image quality enhancement processing on OCT tomographic images using a trained model for image quality enhancement corresponding to OCT tomographic images. For example, when a high image quality button is pressed in response to an instruction from the examiner, a high quality OCTA frontal image generated using a trained model for high image quality corresponding to the OCTA frontal image is displayed. In addition to being changed to display a high-quality OCT tomographic image generated using a trained model for high-quality image corresponding to an OCT tomographic image. At this time, the configuration may be such that a line indicating the position of the OCT tomographic image is displayed superimposed on the OCTA front image. Further, the line may be configured to be movable on the OCTA frontal image according to instructions from the examiner. In addition, when the display of the high image quality button is active, a high quality OCT tomogram obtained by performing high image quality processing on the OCT tomographic image corresponding to the current line position after the above line is moved. The display may be configured to be changed to display an image. Further, by displaying an image quality improvement button for each image to be subjected to image quality improvement processing, the image quality improvement processing may be performed independently for each image.

Further, information indicating a blood vessel region in an OCTA tomographic image (for example, motion contrast data equal to or higher than a threshold value) may be displayed superimposed on an OCT tomographic image that is a B-scan image at a corresponding position. At this time, for example, when the OCT tomographic image is improved in quality, the OCTA tomographic image at the corresponding position may be improved in quality. Information indicating a blood vessel region in the OCTA tomographic image obtained by increasing the image quality may be displayed in a superimposed manner on the OCT tomographic image obtained by increasing the image quality. Note that the information indicating the blood vessel region may be any information that can be identified, such as color. Further, the superimposed display and non-display of information indicating the blood vessel region may be configured to be changeable according to instructions from the examiner. Further, when a line indicating the position of the OCT tomographic image is moved on the OCTA front image, the display of the OCT tomographic image may be updated according to the position of the line. At this time, since the OCTA tomographic image at the corresponding position is also updated, the superimposed display of information indicating the blood vessel region obtained from the OCTA tomographic image may be updated. Thereby, for example, the three-dimensional distribution and state of the blood vessel region can be effectively confirmed while easily confirming the positional relationship between the blood vessel region and the region of interest at an arbitrary position. Furthermore, instead of using a trained model for improving image quality, the image quality of the OCTA tomographic image may be improved by averaging processing of a plurality of OCTA tomographic images acquired at corresponding positions. . Further, the OCT tomographic image may be a pseudo OCT tomographic image reconstructed as a cross section at an arbitrary position in OCT volume data. Further, the OCTA tomographic image may be a pseudo OCTA tomographic image reconstructed as a cross section at an arbitrary position in the OCTA volume data. Note that the arbitrary position may be at least one arbitrary position, and may be configured to be changeable according to instructions from the examiner. At this time, the configuration may be such that a plurality of pseudo tomographic images corresponding to a plurality of positions are reconstructed.

Note that only one tomographic image (for example, an OCT tomographic image or an OCTA tomographic image) may be displayed, or a plurality of tomographic images may be displayed. When a plurality of tomographic images are displayed, tomographic images acquired at different positions in the sub-scanning direction may be displayed, or for example, multiple tomographic images obtained by cross-scanning etc. may be of high quality. When displaying, images in different scanning directions may be displayed. In addition, when displaying a plurality of tomographic images obtained by, for example, a radial scan with high image quality, a plurality of partially selected tomographic images (for example, two tomographic images at mutually symmetrical positions with respect to a reference line) ) may be displayed respectively. Furthermore, multiple tomographic images are displayed on a display screen for follow-up observation (display screen for follow-up), and instructions for improving image quality and analysis results (for example, the thickness of a specific layer, etc.) are displayed using the same method as described above. ) may be displayed. At this time, the plurality of tomographic images displayed may be a plurality of tomographic images of a predetermined part of the subject's eye obtained at different times, or may be a plurality of tomographic images obtained at different times on the same examination day. Good too. Furthermore, image quality enhancement processing may be performed on the tomographic image based on information stored in the database using a method similar to the method described above.

Similarly, when displaying an SLO image with high image quality, for example, the SLO images displayed on the same display screen may be displayed with high image quality. Furthermore, when displaying a bright front image with high image quality, for example, the bright front image displayed on the same display screen may be displayed with high image quality. Furthermore, multiple SLO images and brightness front images are displayed on the display screen for follow-up observation, and instructions for improving image quality and analysis results (for example, the thickness of a specific layer, etc.) are displayed using the same method as described above. may be performed. Furthermore, image quality enhancement processing may be performed on the SLO image and the brightness front image based on information stored in the database using a method similar to the method described above. Note that the display of the tomographic image, SLO image, and brightness front image are merely examples, and these images may be displayed in any manner depending on the desired configuration. Furthermore, at least two or more of the OCTA front image, tomographic image, SLO image, and luminance front image may be displayed with high image quality by a single instruction.

With such a configuration, the display control unit 101-5 can cause the output unit 103 to display a high-quality image obtained by high-quality image processing. Note that if at least one condition is selected among multiple conditions related to display of high-quality images, display of analysis results, depth range of the displayed frontal image, etc., even if the display screen changes, the selected condition will not be displayed. It may be configured such that the conditions set are maintained. Note that display control of various high-quality images, the above-mentioned lines, information indicating blood vessel regions, etc. may be performed by the display control unit 101-5.

Further, the high-quality model may be used for at least one frame of the live moving image on the preview screen displayed on the output unit 103 by the display control unit 101-5. At this time, if a plurality of live moving images of different parts or types are displayed on the preview screen, a learned model corresponding to each live moving image may be used. For example, for the anterior eye image used in the alignment process, an image whose image quality has been enhanced using a high image quality model for anterior eye images may be used. Similarly, for various images used for detection processing of a predetermined area in various images, images whose image quality has been enhanced using an image quality enhancement model for each image may be used.

At this time, for example, if the high image quality button is pressed in response to an instruction from the examiner, the display of multiple live video images of different types (for example, anterior ocular images, SLO images, tomographic images) will be changed to ( (at the same time), the display may be changed to a high-quality moving image obtained by each image quality enhancement process. At this time, the display of the high-quality moving image may be a continuous display of high-quality images obtained by performing high-quality processing on each frame. Further, for example, since the feature amount of an image may differ depending on the type of image, a trained model for image quality improvement corresponding to each type of target image for image quality improvement processing may be used. For example, when a high-quality image button is pressed in response to an instruction from the examiner, the high-quality model corresponding to the anterior eye image is used to not only perform high-quality processing on the anterior eye image, but also to process the SLO image. The SLO image may also be configured to be subjected to image quality enhancement processing using the image quality enhancement model. Also, for example, when a high-quality button is pressed in response to an instruction from the examiner, the display changes to a high-quality anterior eye image generated using a high-quality model corresponding to the anterior eye image. In addition, the display may be changed to display a high-quality SLO image generated using a high-quality model corresponding to the SLO image. For example, when a high image quality button is pressed in response to an instruction from the examiner, the SLO image is not only processed to improve the image quality using the high image quality model that corresponds to the SLO image, but also corresponds to the tomographic image. The configuration may also be such that tomographic images are also subjected to image quality enhancement processing using the image quality enhancement model. Also, for example, when a high-quality button is pressed in response to an instruction from the examiner, the display is simply changed to a high-quality SLO image generated using a high-quality model corresponding to the SLO image. Instead, the display may be changed to display a high-quality tomographic image generated using a high-quality model corresponding to the tomographic image. At this time, the configuration may be such that a line indicating the position of the tomographic image is displayed superimposed on the SLO image. Further, the line may be configured to be movable on the SLO image according to instructions from the examiner. In addition, when the display of the high image quality button is active, after the above line is moved, a high quality tomographic image obtained by performing high image quality processing on the tomographic image corresponding to the current line position will be displayed. The display may be changed. Further, by displaying an image quality improvement button for each image to be subjected to image quality improvement processing, the image quality improvement processing may be performed independently for each image.

With this, for example, even for live moving images, the processing time can be shortened, so the examiner can obtain highly accurate information before starting imaging. Therefore, for example, when the operator corrects the alignment position while checking the preview screen, it is possible to reduce failures in re-imaging, etc., and thus it is possible to improve the accuracy and efficiency of diagnosis. In addition, the image processing device 101 performs the above-described process so that a partial area such as an artifact area obtained by segmentation processing or the like is re-imaged (rescanned) during or at the end of image capture in response to an instruction regarding the start of image capture. The scanning means may be driven and controlled. Note that, depending on the state of the movement of the eye to be examined, etc., it may not be possible to take a good image with one rescan, so the drive may be controlled so that the rescan is repeated a predetermined number of times. At this time, the configuration may be such that the rescanning is ended in response to an instruction from the operator (for example, after pressing the shooting cancel button) even during the rescanning a predetermined number of times. At this time, the configuration may be such that photographic data up to the end of rescanning is saved in accordance with an instruction from the operator. Note that, for example, a confirmation dialog may be displayed after the shooting cancel button is pressed, and the user can select whether to save the shooting data or discard the shooting data in accordance with an instruction from the operator. Also, for example, after pressing the shooting cancel button, the next rescan is not executed (although the current rescan is executed until it is completed), and waits until there is an instruction (input) from the operator in the confirmation dialog. It may be configured as follows. Furthermore, for example, if information indicating the certainty of object detection results regarding the region of interest (for example, a numerical value indicating a percentage) exceeds a threshold, it may be configured to automatically perform various adjustments, start shooting, etc. good. Additionally, for example, if information indicating the certainty of object detection results regarding the region of interest (e.g., a numerical value indicating a percentage) exceeds a threshold, various adjustments and the start of imaging can be performed in accordance with instructions from the examiner. It may be configured to change the state to a state (release the execution prohibited state).

Here, during the auto-alignment, there is a possibility that the object to be photographed, such as the retina of the eye 200 to be examined, has not been successfully imaged yet. For this reason, since there is a large difference between the medical images input to the trained model and the medical images used as learning data, there is a possibility that high-quality images cannot be obtained with high accuracy. Therefore, when an evaluation value such as image quality evaluation of a tomographic image (B-scan image) exceeds a threshold value, display of high-quality moving images (continuous display of high-quality frames) may be automatically started. Furthermore, when an evaluation value such as image quality evaluation of a tomographic image exceeds a threshold value, the image quality improvement button may be changed to a state (active state) that can be specified by the examiner. Note that the image quality improvement button is a button for specifying execution of image quality improvement processing. Of course, the high-quality image button may be a button for instructing display of a high-quality image.

Alternatively, a different model for improving image quality may be prepared for each shooting mode with a different scan pattern, etc., and a trained model for improving image quality corresponding to the selected shooting mode may be selected. Alternatively, one image quality improvement model obtained by learning learning data including various medical images obtained in different imaging modes may be used.

Here, in an ophthalmological apparatus, for example, an OCT apparatus, the scan pattern of the light beam used for measurement and the imaged region differ depending on the imaging mode. Therefore, regarding trained models that use tomographic images as input data, a trained model is prepared for each imaging mode, and the trained model corresponding to the selected imaging mode is selected according to the operator's instructions. may be configured. In this case, the imaging modes may include, for example, a retina imaging mode, an anterior segment imaging mode, a vitreous body imaging mode, a macular imaging mode, an optic disc imaging mode, an OCTA imaging mode, and the like. Further, the scan pattern may include a 3D scan, a radial scan, a cross scan, a circle scan, a raster scan, a Lissajous scan (a scan along a Lissajous curve), and the like. Note that in the OCTA imaging mode, the imaging control unit 101-2 controls the above-described scanning unit so that the measurement light scans the same area (same position) of the eye to be examined multiple times. Even in the OCTA imaging mode, it is possible to set a scan pattern such as raster scan, radial scan, cross scan, circle scan, Lissajous scan, etc. Furthermore, regarding a trained model that uses tomographic images as input data, learning can be performed using tomographic images corresponding to cross sections in different directions as learning data. For example, learning may be performed using a tomographic image of a cross section in the xz direction, a tomographic image of a cross section in the yz direction, or the like as learning data.

In addition, the operator can determine whether or not it is necessary to execute the image quality enhancement process using the image quality enhancement model (or display a high quality image obtained by the image quality enhancement process) by clicking the image quality enhancement button provided on the display screen. This may be performed in response to an instruction, or may be performed in accordance with settings stored in advance in the storage unit 101-3. Note that the fact that the image quality enhancement process is being performed using a trained model (high image quality model) may be displayed by the active state of the image quality enhancement button, or by displaying a message to that effect on the display screen. good. Further, the execution of the image quality improvement process may be performed by maintaining the execution state at the time of the previous activation of the ophthalmological apparatus, or by maintaining the execution state at the time of the previous examination for each subject.

Furthermore, moving images to which various trained models such as high-quality models can be applied are not limited to live moving images, but may also be moving images stored (saved) in the storage unit 101-3, for example. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus oculi stored (saved) in the storage unit 101-3 may be displayed on the display screen. For example, if it is desired to observe the vitreous body in a suitable manner, a reference frame may be selected based on a condition such that as much of the vitreous body is present on the frame as possible. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which other frames are aligned in the XZ directions with respect to the selected reference frame may be displayed on the display screen. At this time, for example, it may be configured to continuously display high-quality images (high-quality frames) that are sequentially generated by the high-quality engine for each at least one frame of the moving image.

Note that as the above-mentioned method for positioning between frames, the same method may be applied for the positioning method in the X direction and the method for positioning in the Z direction (depth direction), or different methods may be applied. may be applied. Furthermore, alignment in the same direction may be performed multiple times using different techniques; for example, rough alignment may be followed by precise alignment. In addition, as a method for positioning, for example, positioning using retinal layer boundaries obtained by segmenting a tomographic image (B-scan image) (coarse in the Z direction), and positioning using multiple retinal layer boundaries obtained by dividing a tomographic image Positioning (accurate in the X and Z directions) using correlation information (similarity) between the region and the reference image, and one-dimensional projection image generated for each tomographic image (B scan image) (in the X direction) ) alignment, alignment (in the X direction) using a two-dimensional front image, etc. Alternatively, the configuration may be such that rough alignment is performed pixel by pixel, and then fine alignment is performed subpixel by subpixel.

Further, the image quality improvement model may be updated by additional learning using, as learning data, a ratio value set (changed) in accordance with an instruction from the examiner. For example, if the examiner tends to set a high ratio of the input image to the high-quality image when the input image is relatively dark, the trained model will be additionally trained to adopt this tendency. Thereby, for example, it is possible to customize the model as a learned model that can obtain a synthesis ratio that matches the examiner's preference. At this time, a button may be displayed on the display screen for determining whether or not to use the set (changed) ratio value as learning data for additional learning in accordance with an instruction from the examiner. Alternatively, the ratio determined using the trained model may be set as a default value, and the ratio value may then be changed from the default value in response to an instruction from the examiner. Further, the high-quality model may be a trained model obtained by additionally learning learning data including at least one high-quality image generated using the high-quality model. At this time, it may be possible to select whether or not to use the high-quality image as learning data for additional learning based on an instruction from the examiner.

[Modification 4]
The image feature acquisition unit 101-44, the extraction unit 101-462, and the area detection unit 302 may generate a label image using a trained model for image segmentation and perform image segmentation processing. Here, the label image refers to a label image in which a region label is attached to each pixel of the tomographic image. Specifically, it is an image in which arbitrary regions among a group of regions depicted in an acquired image are divided by a group of identifiable pixel values (hereinafter referred to as label values). Here, the specified arbitrary region includes a region of interest, a volume of interest (VOI), and the like.

By specifying a coordinate group of pixels having arbitrary label values from an image, it is possible to specify a coordinate group of pixels that depict a corresponding region such as a retinal layer in the image. Specifically, for example, if the label value indicating the ganglion cell layer constituting the retina is 1, a coordinate group with a pixel value of 1 is identified among the pixel groups of the image, and the coordinate group corresponding to this coordinate group is determined from the image. Extract a group of pixels. Thereby, the region of the ganglion cell layer in the image can be specified.

Note that the image segmentation process may include a process of reducing or enlarging the label image. At this time, the image complementation processing method used to reduce or enlarge the label image shall use the nearest neighbor method, etc., so as not to erroneously generate undefined label values or label values that should not exist at the corresponding coordinates. .

Image segmentation processing is a process of identifying a region called a ROI (Region of Interest) or VOI, such as an organ or lesion depicted in an image, for use in image diagnosis or image analysis. For example, according to image segmentation processing, it is possible to specify a group of regions of a group of layers constituting the retina from an image acquired by OCT imaging of the posterior segment of the eye. Note that if the area to be specified is not depicted in the image, the number of areas to be specified is 0. Furthermore, if a plurality of region groups to be specified are depicted in the image, the number of regions to be specified may be plural, or even one area surrounding the region group may be specified. good.

The identified region group is output as information that can be used in other processes. Specifically, for example, a group of coordinates of a group of pixels constituting each of the specified group of regions can be output as a group of numerical data. Further, for example, a coordinate group indicating a rectangular area, an elliptical area, a rectangular area, an ellipsoidal area, etc. including each of the identified area groups can be output as a numerical data group. Furthermore, for example, a coordinate group indicating a straight line, curve, plane, curved surface, etc. that is the boundary of the specified region group can be output as a numerical data group. Furthermore, for example, a label image indicating the specified region group can also be output.

Here, as a machine learning model for image segmentation, for example, a convolutional neural network (CNN) can be used. Note that the configuration of the CNN used in this modification is a U-net network that has an encoder function consisting of a plurality of layers including a plurality of downsampling layers, and a decoder function consisting of a plurality of layers including a plurality of upsampling layers. It can be a type of machine learning model. In the U-net type machine learning model, position information (spatial information) that is made ambiguous in multiple layers configured as encoders is transferred to layers of the same dimension (layers that correspond to each other) in multiple layers configured as decoders. ) (e.g., using skip connections).

Further, as an example of changing the configuration of the CNN, for example, a batch normalization layer or an activation layer using a normalized linear function (Rectifier Linear Unit) may be incorporated after the convolution layer. Through these steps of CNN, features of the captured image can be extracted.

Note that as the machine learning model according to this modification, for example, CNN (U-net type machine learning model), a model that combines CNN and LSTM, FCN (Fully Convolutional Network), SegNet, etc. can be used. . Furthermore, depending on the desired configuration, a machine learning model or the like that performs object recognition can also be used. As a machine learning model for object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, a machine learning model that performs object recognition in area units can also be used. As a machine learning model that performs object recognition in area units, YOLO (You Only Look Once) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used.

Further, the learning data of the machine learning model for image segmentation uses a tomographic image acquired by OCT as input data, and uses as output data a labeled image in which a region label is attached to each pixel of the tomographic image. Label images include, for example, the internal limiting membrane (ILM), nerve fiber layer (NFL), ganglion cell layer (GCL), photoreceptor inner segment outer segment junction (ISOS), retinal pigment epithelial layer (RPE), Bruch Label images labeled with membranes (BM), choroid, etc. can be used. In addition, other regions include, for example, the vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner plexiform layer (IPL), inner nuclear layer (INL), cornea, anterior chamber, iris, An image labeled with a crystalline lens or the like may also be used.

Furthermore, input data for a machine learning model for image segmentation is not limited to tomographic images. It may be an anterior eye image, an SLO image, an OCTA image, or the like. In this case, the learning data can be input data of various images, and can be output data of label images in which region names and the like are labeled for each pixel of the various images. For example, when the input data of the learning data is an SLO image, the output data may be an image labeled with the periphery of the optic disc, Disc, Cup, or the like.

Note that the labeled image used as output data may be an image in which each region in a tomographic image has been labeled by a doctor or the like, or an image in which each region has been labeled by a rule-based region detection process. There may be. However, if machine learning is performed using label images that are not properly labeled as output data of training data, images obtained using a trained model trained using the training data will also be labeled appropriately. There is a possibility that you will end up with a label image that has not been subjected to this process. Therefore, by removing pairs including such label images from the training data, it is possible to reduce the possibility that inappropriate label images will be generated using the trained model. Here, the rule-based area detection process refers to a detection process that utilizes known regularities, such as regularities in the shape of the retina, for example.

The image feature acquisition unit 101-44, the extraction unit 101-462, and the area detection unit 302 perform image segmentation processing using such trained models for image segmentation to identify specific areas in various images. It is expected that high-speed and accurate detection will be possible. Note that a trained model for image segmentation may also be prepared for each type of image that is input data. Furthermore, for OCTA frontal images and En-Face images, trained models may be prepared for each depth range for generating images. Furthermore, the trained model for image segmentation may be one that has been trained on images for each imaging region (for example, the center of the macula, the center of the optic disc), or one that has been trained regardless of the imaging region. It's okay.

Furthermore, additional learning may be performed on the learned model for image segmentation using data that has been manually corrected according to an operator's instructions as learning data. Furthermore, the determination of whether additional learning is necessary and the determination of whether or not to transmit data to the server may be performed in a similar manner. In these cases as well, it can be expected that the accuracy of each process can be improved and that processes can be performed in accordance with the examiner's preferences.

Further, when detecting a partial region (for example, a region of interest, an artifact region, an abnormal region, etc.) of the eye 200 using the learned model, the image processing device 101 generates a predetermined image for each detected partial region. Treatment can also be applied. As an example, a case will be described in which at least two partial regions of the vitreous region, the retinal region, and the choroid region are detected. In this case, when performing image processing such as contrast adjustment on the at least two detected partial regions, by using different image processing parameters, adjustment suitable for each region can be performed. By displaying images that have been adjusted appropriately for each region, the operator can more appropriately diagnose diseases, etc. for each partial region. Note that the configuration in which different image processing parameters are used for each detected partial region is similarly applied to the partial region of the eye 200 to be examined that is obtained by detecting the partial region of the eye 200 without using a trained model. You can.

[Modification 5]
The display control unit 101-5 in the various embodiments and modifications described above may display analysis results such as the thickness of a desired layer and various blood vessel densities on the report screen of the display screen after tomographic imaging. . In addition, the optic disc, macular area, blood vessel area, capillary area, artery area, vein area, nerve fiber bundle, vitreous area, macular area, choroid area, sclera area, lamina cribrosa area, retinal layer boundary, retina Values (distribution) of parameters related to the region of interest, including at least one of layer boundary edges, photoreceptor cells, blood cells, blood vessel walls, inner vessel wall boundaries, outer vessel boundaries, ganglion cells, corneal regions, angle regions, Schlemm's canal, etc. may be displayed as the analysis result. Here, the site of interest may be, for example, a vortex vein that is an outlet of a blood vessel in the Haller layer (an example of a blood vessel in a depth range of a part of the choroidal region) to the outside of the eye. At this time, parameters related to the region of interest include, for example, the number of vortex veins (for example, the number of vortex veins in each region), the distance from the optic disc to each vortex vein, the angle at which each vortex vein is located around the optic disc, etc. It may be. This makes it possible to accurately diagnose, for example, various diseases related to pachychoroid (thickened choroid) (eg, choroidal neoangiopathy). Further, for example, by analyzing a medical image to which various artifact reduction processes have been applied, the various analysis results described above can be displayed as highly accurate analysis results. Note that artifacts include, for example, false image areas caused by light absorption by blood vessel areas, projection artifacts, and band-shaped artifacts in the front image that occur in the main scanning direction of the measurement light due to the condition of the eye being examined (movement, blinking, etc.). There may be. Furthermore, the artifact may be of any type, as long as it is a defective area that randomly appears on a medical image of a predetermined region of a subject each time the image is taken. Furthermore, the display control unit 101-5 may cause the output unit 103 to display values (distribution) of parameters regarding an area including at least one of the various artifacts (unselected areas) as described above as an analysis result. Further, the values (distribution) of parameters regarding a region including at least one of abnormal sites such as drusen, new blood vessels, vitiligo (hard vitiligo), and pseudodrusen may be displayed as an analysis result. Further, a comparison result obtained by comparing the standard value or standard range obtained using the standard database with the analysis result may be displayed.

Further, the analysis results may be displayed as an analysis map, sectors showing statistical values corresponding to each divided area, or the like. Note that the analysis results may be generated using a trained model (analysis result generation engine, trained model for analysis result generation) obtained by learning the analysis results of medical images as learning data. . At this time, the trained model is trained using learning data that includes a medical image and the analysis result of the medical image, or learning data that includes the medical image and the analysis result of a different type of medical image than the medical image. It may be obtained by

Further, the learning data for performing image analysis may include a label image generated using a trained model for image segmentation processing and an analysis result of a medical image using the label image. In this case, the image processing device 101 may function as an example of an analysis result generation unit that generates analysis results of tomographic images from the results of image segmentation processing using, for example, a trained model for analysis result generation. can. Furthermore, the trained model uses learning data that includes input data that is a set of multiple medical images of different types of a predetermined region, such as a brightness En-Face image and a motion contrast frontal image (OCTA En-Face image). It may be obtained through learning using .

Further, it may be configured such that analysis results obtained using high-quality images generated using a high-quality model are displayed. In this case, the input data included in the training data may be a high-quality image generated using a trained model for high-quality images, or a set of a low-quality image and a high-quality image. Good too. Note that the learning data may be an image that has been manually or automatically modified at least in part with respect to an image that has been enhanced in quality using a trained model.

In addition, the learning data includes, for example, at least analysis values obtained by analyzing the analysis region (e.g., average value, median value, etc.), a table containing the analysis values, an analysis map, the position of the analysis region such as a sector in an image, etc. The input data may be labeled (annotated) with information including one as the correct data (for supervised learning). Note that the configuration may be such that analysis results obtained using a learned model for generating analysis results are displayed in response to instructions from the operator.

Furthermore, the display control unit 101-5 in the embodiments and modifications described above may display various diagnostic results such as diabetic retinopathy, glaucoma, age-related macular degeneration, etc. on the report screen of the display screen. At this time, for example, by analyzing a medical image to which the various artifact reduction processes described above have been applied, highly accurate diagnostic results can be displayed. In addition, the diagnosis result may be displayed by displaying the position of the identified abnormal region on the image, or by displaying the state of the abnormal region by text or the like. Furthermore, the classification results (for example, Curtin classification) of abnormal parts, etc. may be displayed as the diagnosis results. Further, as the classification result, for example, information indicating the probability of each abnormal site (for example, a numerical value indicating a ratio) may be displayed. Further, information necessary for a doctor to confirm a diagnosis may be displayed as a diagnosis result. The necessary information may include, for example, advice on additional photography. For example, if an abnormal site is detected in a blood vessel region in an OCTA image, a message may be displayed to the effect that fluorescence imaging using a contrast agent that allows the blood vessels to be observed in more detail than in OCTA will be performed. Further, the diagnosis result may be information regarding the subject's future medical treatment policy, etc. In addition, the diagnosis results include, for example, the diagnosis name, the type and condition (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion relative to the region of interest, findings (image interpretation findings, etc.), and the basis for the diagnosis name (positive The information may include at least one of the following: negative medical support information, etc.), and grounds for denying the diagnosis (negative medical support information). At this time, for example, a diagnosis result that is more likely than a diagnosis result such as a diagnosis name input in response to an instruction from the examiner may be displayed as medical support information. Furthermore, when a plurality of types of medical images are used, for example, the type of medical image that can serve as the basis for the diagnosis result may be displayed in an identifiable manner. In addition, the basis for the diagnosis result is a map (attention map, activation map) that visualizes the features extracted by the trained model, such as a color map (heat map) that shows the features in color. good. At this time, for example, a heat map may be displayed superimposed on the medical image used as input data. Note that heat maps are, for example, Gradient-weighted Class Activation Mapping (Gradient-weighted Class Activation Mapping) and Guided Grad, which are methods for visualizing areas that have a large contribution to the output value of the predicted (estimated) class (areas with large gradients). - Can be obtained using CAM etc.

Note that the diagnostic results may be generated using a trained model (a diagnostic result generation engine, a trained model for generating diagnostic results) obtained by learning the diagnostic results of medical images as learning data. . In addition, the trained model is trained using training data that includes a medical image and the diagnosis result of the medical image, or training data that includes the medical image and the diagnosis result of a different type of medical image than the medical image. It may be something you have obtained.

Further, the learning data may include a label image generated using a region recognition engine or a trained model for segmentation processing, and a diagnosis result of a medical image using the label image. In this case, the image processing device 101 may function as an example of a diagnostic result generation unit that generates diagnostic results of tomographic images from the results of image segmentation processing using, for example, a trained model for generating diagnostic results. can.

Furthermore, the system may be configured to display diagnostic results obtained using high-quality images generated using a high-quality image engine. In this case, the input data included in the learning data may be a high-quality image generated using a high-quality engine, or may be a set of a low-quality image and a high-quality image. Note that the learning data may be an image that has been manually or automatically modified at least in part with respect to an image that has been enhanced in quality using a trained model.

In addition, the learning data includes, for example, the diagnosis name, the type and condition (degree) of the lesion (abnormal part), the position of the lesion in the image, the position of the lesion relative to the region of interest, findings (image interpretation findings, etc.), and the basis for the diagnosis name (positive The input data is labeled (annotated) as the correct data (for supervised learning), including at least one of the following: (medical support information, etc.), grounds for denying the diagnosis (negative medical support information), etc. Data may also be used. Note that the configuration may be such that the diagnostic results obtained using the trained model for generating diagnostic results are displayed in response to instructions from the examiner.

Note that a trained model may be prepared for each type of information or information used as input data, and the diagnostic results may be obtained using the trained model. In this case, the final diagnosis result may be determined by performing statistical processing on the information output from each learned model. For example, the ratio of information output from each trained model may be added for each type of information, and information with a higher total ratio than other information may be determined as the final diagnosis result. Note that the statistical processing is not limited to calculation of the total, but may also be calculation of the average value, median value, etc. Furthermore, for example, the diagnosis result may be determined using information that has a higher ratio than other information (information that has the highest ratio) among the information output from each trained model. Similarly, the diagnosis result may be determined using information whose proportion is equal to or greater than a threshold among the information output from each trained model.

Further, it may be configured such that it is possible to determine (approve) the quality of the determined diagnostic result in accordance with the operator's instruction (selection). Furthermore, the diagnosis result may be determined from the information output from each trained model in accordance with an instruction (selection) by the operator. At this time, for example, the display control unit 101-5 may cause the output unit 103 to display the information output from each learned model and its ratio side by side. Then, the operator may select, for example, information with a higher percentage than other information, and thereby determine the selected information as the diagnosis result. Furthermore, a machine learning model may be used to determine the diagnosis result from the information output from each learned model. In this case, the machine learning algorithm may be a different type of machine learning algorithm from the machine learning algorithm used to generate the diagnostic results, such as a neural network, support vector machine, AdaBoost, Bayesian network, or random Forest et al. may be used.

Note that the learning of the various trained models described above may be performed not only by supervised learning (learning using labeled learning data) but also by semi-supervised learning. In semi-supervised learning, for example, multiple classifiers (classifiers) each perform supervised learning, then identify (classify) the unlabeled learning data, and then classify the data according to the reliability of the classification result (classification result). This is a method that automatically labels (annotates) identification results (for example, identification results whose certainty is higher than a threshold) and performs learning using the labeled learning data. Semi-supervised learning may be, for example, co-training or multiview. At this time, the trained model for generating diagnostic results uses, for example, a first classifier that identifies medical images of normal subjects and a second classifier that identifies medical images that include a specific lesion. It may also be a trained model obtained by semi-supervised learning (for example, co-training). Note that the purpose is not limited to diagnosis, but may also be used, for example, to support imaging. In this case, the second classifier may, for example, identify a medical image that includes a partial region such as a region of interest or an artifact region.

In addition, the display control unit 101-5 according to the various embodiments and modifications described above also displays object recognition results ( Object detection results) or segmentation results may also be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Furthermore, for example, a color or the like may be superimposed and displayed on an object in the image. Note that object recognition results and segmentation results are generated using a trained model (object recognition engine, object recognition It may be generated using a trained model, a segmentation engine, or a trained model for segmentation. Note that the analysis result generation and diagnosis result generation described above may be obtained by using the object recognition results and segmentation results described above. For example, analysis result generation or diagnostic result generation processing may be performed on a region of interest obtained through object recognition or segmentation processing.

Furthermore, when detecting an abnormal region, the image processing apparatus 101 may use a generative adversarial network (GAN) or a variational auto-encoder (VAE). For example, a DCGAN (Deep Convolutional GAN) consists of a generator obtained by learning to generate medical images and a discriminator obtained by learning to distinguish between new medical images generated by the generator and real medical images. can be used as a machine learning model.

When using DCGAN, for example, a discriminator encodes an input medical image into a latent variable, and a generator generates a new medical image based on the latent variable. Thereafter, the difference between the input medical image and the generated new medical image can be extracted (detected) as an abnormal site. Further, when using VAE, for example, an input medical image is encoded by an encoder to become a latent variable, and a new medical image is generated by decoding the latent variable by a decoder. Thereafter, the difference between the input medical image and the generated new medical image can be extracted as an abnormal site.

Furthermore, the image processing apparatus 101 may detect an abnormal site using a convolutional auto-encoder (CAE). When using CAE, the same medical images are used as input data and output data during learning. As a result, when a medical image with an abnormal region is input to CAE at the time of estimation, a medical image without an abnormal region is output according to the learning tendency. Thereafter, the difference between the medical image input to CAE and the medical image output from CAE can be extracted as an abnormal site.

In these cases, the image processing device 101 converts information about the difference between the medical image obtained using the adversarial generative network or autoencoder and the medical image input to the adversarial generative network or autoencoder into information about the abnormal site. It can be generated as Thereby, the image processing apparatus 101 can be expected to detect an abnormal region at high speed and with high accuracy. For example, even if it is difficult to collect a large number of medical images containing abnormal parts as training data to improve the detection accuracy of abnormal parts, medical images of normal subjects, which are relatively easy to collect, can be used as training data. Can be used. Therefore, for example, learning for accurately detecting an abnormal region can be efficiently performed. Here, the autoencoder includes VAE, CAE, and the like. Further, at least a part of the generating unit of the generative adversarial network may be configured with a VAE. Thereby, for example, a relatively clear image can be generated while reducing the phenomenon of generating similar data. For example, the image processing device 101 may provide information regarding the difference between a medical image obtained from various medical images using an adversarial generative network or an autoencoder, and a medical image input to the adversarial generative network or the autoencoder. , can be generated as information regarding the abnormal site. Furthermore, for example, the display control unit 101-5 can display medical images obtained from various medical images using an adversarial generative network or an autoencoder, and medical images input to the adversarial generative network or the autoencoder. Information regarding the difference can be displayed on the output unit 103 as information regarding the abnormal site.

In particular, the trained model for generating diagnostic results may be a trained model obtained by training with training data that includes input data that is a set of multiple medical images of different types of predetermined parts of the subject. good. At this time, as the input data included in the learning data, for example, input data including a set of a motion contrast frontal image and a brightness frontal image (or a brightness tomographic image) of the fundus can be considered. Further, as the input data included in the learning data, for example, input data including a set of a tomographic image of the fundus (B scan image) and a color fundus image (or a fluorescent fundus image) can be considered. Furthermore, the plurality of medical images of different types may be of any type as long as they are obtained using different modalities, different optical systems, or different principles.

In particular, the trained model for generating diagnostic results may be a trained model obtained by learning using learning data that includes input data that is a set of a plurality of medical images of different parts of the subject. At this time, as input data included in the learning data, for example, input data including a set of a tomographic image of the fundus (B-scan image) and a tomographic image of the anterior segment (B-scan image) can be considered. Furthermore, as input data included in the learning data, for example, input data that is a set of a three-dimensional OCT image (three-dimensional tomographic image) of the macula in the fundus and a circle scan (or raster scan) tomographic image of the optic disc in the fundus, etc. can also be considered.

Note that the input data included in the learning data may be a plurality of medical images of different parts and different types of the subject. At this time, the input data included in the learning data may be, for example, input data that includes a tomographic image of the anterior segment of the eye and a color fundus image. Further, the above-mentioned trained model may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of a predetermined region of a subject at different photographing angles of view. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-divisionally dividing a predetermined region into a plurality of regions, such as a panoramic image. At this time, by using wide-angle images such as panoramic images as learning data, it is possible to obtain image features with higher accuracy because they contain more information than narrow-angle images, so processing results can be improved. Furthermore, the input data included in the learning data may be input data that is a set of a plurality of medical images of a predetermined region of the subject at different times.

Furthermore, the display screen on which at least one of the above-mentioned analysis results, diagnosis results, object recognition results, and segmentation results is displayed is not limited to the report screen. Such display screens include, for example, at least one display screen such as a shooting confirmation screen, a display screen for progress observation, and a preview screen for various adjustments before shooting (a display screen on which various live video images are displayed). may be displayed. For example, by displaying the above-mentioned at least one result obtained using the above-described learned model on the photographing confirmation screen, the operator can confirm highly accurate results even immediately after photographing.

Further, for example, when a specific object is recognized, a frame surrounding the recognized object may be displayed superimposed on the live video image. At this time, if information indicating the certainty of the object recognition result (for example, a numerical value indicating a percentage) exceeds a threshold, it may be highlighted, for example by changing the color of the frame surrounding the object. good. This allows the examiner to easily identify objects on the live video.

Note that a learned model for generating correct data for generating correct data such as labeling (annotation) may be used to generate correct data used for learning the various learned models described above. At this time, the trained model for generating correct answer data may be obtained by (sequentially) additionally learning correct answer data obtained by labeling (annotation) by the examiner. That is, the learned model for generating correct data may be obtained by additionally learning learning data in which unlabeled data is input data and labeled data is output data. In addition, in multiple consecutive frames such as a moving image, the system is configured to take into account the results of object recognition, segmentation, etc. of the previous and subsequent frames and correct the results of frames determined to have low accuracy. Good too. At this time, the corrected results may be additionally learned as correct data in response to instructions from the examiner. For example, for medical images with low accuracy of results, the examiner can create a feature map (attention map, activation map), which is an example of a map (attention map, activation map) that visualizes the feature values extracted by the trained model, on the medical image. It may be configured such that additional learning is performed using an image labeled (annotated) as input data while checking a color map (heat map) showing the image in color. For example, in a heat map on a layer immediately before outputting the results of a trained model, if the points of interest are different from the examiner's intention, label (annotate) the points that the examiner thinks should be noted. It is also possible to additionally learn medical images. As a result, for example, the trained model prioritizes feature amounts of partial regions on a medical image that have a relatively large influence on the output results of the trained model over other regions ( additional learning can be performed (by adding weights).

Here, the various learned models described above can be obtained by machine learning using learning data. Machine learning includes, for example, deep learning that consists of multilayer neural networks. Further, for example, a convolutional neural network can be used for at least a portion of the multi-layer neural network. Further, a technique related to an autoencoder (self-encoder) may be used for at least a portion of the multi-layer neural network. Furthermore, a technique related to backpropagation (error backpropagation method) may be used for learning. Furthermore, a method (dropout) of randomly inactivating each unit (each neuron or each node) may be used for learning. Further, for learning, a method (batch normalization) may be used in which data transmitted to each layer of a multilayer neural network is normalized before an activation function (for example, a ReLu function) is applied. However, machine learning is not limited to deep learning, and may be any type of learning that uses a model that can self-extract (express) features of learning data such as images through learning. Here, the machine learning model refers to a learning model using a machine learning algorithm such as deep learning. Further, a trained model is a model that has been trained (performed learning) using appropriate learning data in advance for a machine learning model based on an arbitrary machine learning algorithm. However, the trained model does not undergo any further learning, and additional learning can be performed on the trained model. Further, the learning data is composed of a pair of input data and output data (correct data). Here, the learning data may be referred to as teacher data, or the correct answer data may be referred to as teacher data.

Note that the GPU can perform efficient calculations by processing more data in parallel. For this reason, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in this modification, a GPU is used in addition to a CPU for processing by the image processing device 101, which is an example of a learning unit (not shown). Specifically, when a learning program including a learning model is executed, learning is performed by the CPU and GPU working together to perform calculations. Note that the processing of the learning section may be performed by only the CPU or GPU. Furthermore, the processing unit (estimation unit) that executes processing using the various learned models described above may also use a GPU, similar to the learning unit. Further, the learning section may include an error detection section and an updating section (not shown). The error detection unit obtains an error between output data output from the output layer of the neural network and correct data according to input data input to the input layer. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the correct data. Furthermore, the updating section updates the connection weighting coefficients between the nodes of the neural network, etc., based on the error obtained by the error detection section, so that the error becomes smaller. This updating unit updates the connection weighting coefficients and the like using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting connection weighting coefficients between nodes of each neural network so that the above-mentioned error is reduced.

In addition, the machine learning model used for object recognition, segmentation, high image quality, etc. mentioned above has the function of an encoder consisting of multiple layers including multiple downsampling layers, and the function of an encoder consisting of multiple layers including multiple upsampling layers. A U-net type machine learning model having a decoder function consisting of In the U-net type machine learning model, position information (spatial information) that is made ambiguous in multiple layers configured as encoders is transferred to layers of the same dimension (layers that correspond to each other) in multiple layers configured as decoders. ) (e.g., using skip connections).

Moreover, as a machine learning model used for the above-mentioned object recognition, segmentation, high image quality, etc., for example, FCN (Fully Convolutional Network), SegNet, etc. can also be used. Alternatively, a machine learning model that performs object recognition in area units may be used depending on a desired configuration. As a machine learning model for object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, YOLO (You Only Look Once) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used as a machine learning model that performs object recognition in area units.

Further, the machine learning model may be, for example, a capsule network (CapsNet). Here, in a general neural network, each unit (each neuron or each node) is configured to output a scalar value, so that, for example, the spatial positional relationship (relative position) between features in an image is configured such that spatial information regarding the location is reduced. Thereby, for example, learning can be performed such that the influence of local distortion or parallel movement of the image is reduced. On the other hand, in a capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is held. Thereby, for example, learning can be performed that takes into account the spatial positional relationship between features in an image.

[Modification 6]
In the preview screen in the various embodiments and modifications described above, the various learned models described above may be used for each at least one frame of a live video image. At this time, if a plurality of live moving images of different parts or types are displayed on the preview screen, a learned model corresponding to each live moving image may be used. With this, for example, even for live moving images, the processing time can be shortened, so the examiner can obtain highly accurate information before starting imaging. Therefore, for example, it is possible to reduce failures in re-imaging, etc., and thus it is possible to improve the accuracy and efficiency of diagnosis.

Note that the plurality of live moving images may be, for example, a moving image of the anterior segment of the eye for alignment in the XYZ directions, and a frontal moving image of the fundus for focus adjustment of the fundus observation optical system or OCT focus adjustment. Further, the plurality of live moving images may be, for example, tomographic moving images of the fundus for OCT coherence gate adjustment (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length). When such a preview image is displayed, the various adjustments described above are performed so that the regions detected using the trained model for object recognition and the trained model for segmentation meet predetermined conditions. The image processing apparatus 101 may be configured as follows. For example, a value (for example, a contrast value or an intensity value) related to a predetermined retinal layer such as the vitreous region or RPE detected using a trained model for object recognition or a trained model for segmentation exceeds a threshold value ( Alternatively, the configuration may be such that various adjustments such as OCT focus adjustment are performed so that the peak value is reached. In addition, for example, OCT may be applied so that a predetermined retinal layer such as the vitreous region or RPE detected using a trained model for object recognition or a trained model for segmentation is at a predetermined position in the depth direction. It may be configured such that coherence gate adjustment is performed.

In these cases, the image processing apparatus 101 can generate a high-quality moving image by performing image quality enhancement processing on the moving image using the learned model. The imaging control unit 101-2 also controls the reference mirror 221 so that a partial area such as a region of interest obtained through segmentation processing or the like is located at a predetermined position in the display area while a high-quality moving image is displayed. It is possible to drive and control the optical member for changing the photographing range. In such a case, the imaging control unit 101-2 can automatically perform alignment processing based on highly accurate information so that the desired area is located at a predetermined position in the display area. Note that the optical member that changes the imaging range may be, for example, an optical member that adjusts the coherence gate position, and specifically, may be the reference mirror 221 that reflects the reference light. Further, the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measurement optical path length and the reference optical path length. It may also be a mirror, etc. Note that the optical member that changes the photographing range may be, for example, the stage section 100-2. In addition, the imaging control unit 101-2 performs scanning so that partial areas such as artifact areas obtained by segmentation processing etc. are re-imaged (rescanned) during or at the end of imaging, in response to an instruction regarding the start of imaging. The means may be driven and controlled. Furthermore, for example, if information indicating the certainty of object recognition results regarding the region of interest (for example, a numerical value indicating a percentage) exceeds a threshold, it may be configured to automatically perform various adjustments, start shooting, etc. good. Additionally, for example, if information indicating the certainty of object recognition results regarding the region of interest (e.g., a numerical value indicating a percentage) exceeds a threshold, various adjustments and start of imaging can be performed in accordance with instructions from the examiner. It may be configured to change the state to a state (release the execution prohibited state).

Furthermore, the moving images to which the various learned models described above can be applied are not limited to live moving images, and may be, for example, moving images stored (saved) in the storage unit 101-3. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus oculi stored (saved) in the storage unit 101-3 may be displayed on the display screen. For example, if you want to observe the vitreous body in a suitable manner, you may first select a reference frame based on conditions such as as much vitreous body as possible on the frame. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which other frames are aligned in the XZ directions with respect to the selected reference frame may be displayed on the display screen. At this time, for example, high-quality images (high-quality frames) sequentially generated by a trained model for improving image quality may be continuously displayed for each at least one frame of the moving image.

Note that as the above-mentioned method for positioning between frames, the same method may be applied for the positioning method in the X direction and the method for positioning in the Z direction (depth direction), or different methods may be applied. may be applied. Furthermore, alignment in the same direction may be performed multiple times using different techniques; for example, rough alignment may be followed by precise alignment. In addition, as a method for positioning, for example, positioning using retinal layer boundaries obtained by segmenting a tomographic image (B-scan image) (coarse in the Z direction), and positioning using multiple retinal layer boundaries obtained by dividing a tomographic image Positioning (accurate in the X and Z directions) using correlation information (similarity) between the region and the reference image, and one-dimensional projection images generated for each tomographic image (B scan image) (in the X direction) ) alignment, alignment (in the X direction) using a two-dimensional front image, etc. Alternatively, the configuration may be such that rough alignment is performed pixel by pixel, and then fine alignment is performed subpixel by subpixel.

Here, during various adjustments, there is a possibility that the object to be photographed, such as the retina of the eye to be examined, has not yet been successfully imaged. For this reason, since there is a large difference between the medical images input to the trained model and the medical images used as learning data, there is a possibility that high-quality images cannot be obtained with high accuracy. Therefore, when an evaluation value such as image quality evaluation of a tomographic image (B scan) exceeds a threshold value, display of high-quality moving images (continuous display of high-quality frames) may be automatically started. Further, when an evaluation value such as image quality evaluation of a tomographic image (B scan) exceeds a threshold value, the image quality improvement button may be changed to a state (active state) that can be specified by the examiner.

In addition, for example, a different trained model for high image quality is prepared for each shooting mode with a different scan pattern, etc., and the trained model for high image quality corresponding to the selected shooting mode is selected. may be done. Alternatively, one trained model for improving image quality obtained by learning training data including various medical images obtained in different imaging modes may be used.

[Modification 7]
In the embodiments and modified examples described above, when various trained models are performing additional learning, it may be difficult to output (estimate/predict) using the trained model itself that is currently undergoing additional learning. be. For this reason, it is preferable that the input of medical images other than learning data to the learned model during the execution of additional learning is prohibited. Further, another trained model that is the same as the trained model before execution of additional learning may be prepared as a preliminary trained model. At this time, it is preferable to configure the system so that medical images other than learning data can be input to the preliminary trained model while the additional learning is being performed. Then, after the additional learning is completed, the trained model after performing the additional learning is evaluated, and if there is no problem, the preliminary trained model may be replaced with the trained model after performing the additional learning. Furthermore, if there is a problem, a preliminary trained model may be used.

In addition, as an evaluation of the trained model after performing additional learning, for example, the trained model for classification for classifying high-quality images obtained with the trained model for high image quality from other types of images is evaluated. may be used. For example, a trained model for classification uses as input data multiple images including high-quality images and low-quality images obtained by a trained model for high image quality, and the types of these images are labeled (annotated). The model may be a trained model obtained by learning training data that includes the correct data as correct data. At this time, the type of image of the input data at the time of estimation (during prediction) is displayed together with information indicating the probability of each type of image included in the correct data at the time of learning (for example, a numerical value indicating a percentage). Good too. In addition to the above-mentioned images, the input data for the trained model for classification may also include superposition processing of multiple low-quality images (for example, averaging processing of multiple low-quality images obtained by positioning), etc. may include high-quality images that have undergone high contrast, noise reduction, and the like. In addition, to evaluate the trained model after performing additional learning, for example, the trained model after performing additional learning and the trained model before performing additional learning (preliminary trained model) are used to evaluate the same model. A plurality of high-quality images obtained from the images may be compared, or analysis results of the plurality of high-quality images may be compared. At this time, for example, the comparison result of the plurality of high-quality images (an example of a change due to additional learning) or the comparison result of the analysis result of the plurality of high-quality images (an example of a change due to additional learning) is within a predetermined range. It may be determined whether or not, and the determination result may be displayed.

Further, it may be possible to selectively use a learned model obtained by learning for each part to be imaged. Specifically, a first trained model obtained using training data including a first imaging region (e.g., anterior eye segment, posterior eye segment, etc.) and a second trained model that is different from the first imaging region. A plurality of trained models can be prepared, including a second trained model obtained using training data including the imaged body part. The image processing device 101 may include a selection unit for selecting one of the plurality of learned models. At this time, the image processing device 101 may include a control unit that performs additional learning on the selected learned model. The control means searches for data in which a photographed region corresponding to the selected trained model and a photographed image of the photographed region are paired, and uses the data obtained through the search as learning data. This learning can be performed as additional learning on the selected trained model. Note that the imaged region corresponding to the selected learned model may be obtained from information in the header of the data, or may be manually input by the examiner. Further, the data search may be performed via a network, for example, from a server in an external facility such as a hospital or research institute. Thereby, additional learning can be efficiently performed for each imaged body part using the photographed images of the body part corresponding to the learned model.

Note that the selection means and the control means may be constituted by a software module executed by a processor such as a CPU or an MPU of the image processing apparatus 101. Furthermore, the selection means and the control means may be constituted by a circuit that performs a specific function, such as an ASIC, or an independent device.

In addition, when acquiring learning data for additional learning via a network from a server in an external facility such as a hospital or research institute, it is necessary to reduce reliability degradation due to falsification or system trouble during additional learning. is useful. Therefore, the validity of the learning data for additional learning may be detected by checking the consistency using a digital signature or hashing. Thereby, learning data for additional learning can be protected. At this time, if the validity of the learning data for additional learning cannot be detected as a result of checking the consistency using digital signatures and hashing, a warning to that effect will be issued and additional learning will be performed using that learning data. Make it not exist. Note that the server may take any form, such as a cloud server, fog server, or edge server, regardless of its installation location. Note that when configuring a network within a facility, within a site that includes a facility, within an area that includes multiple facilities, etc. to enable wireless communication, for example, if a network is configured to enable wireless communication, The reliability of the network may be improved by configuring it to use radio waves in a dedicated wavelength band. Furthermore, the network may be constructed using wireless communication that is capable of high speed, large capacity, low delay, and multiple simultaneous connections.

Further, data protection by checking the consistency as described above is applicable not only to learning data for additional learning but also to data including medical images. Furthermore, the image management system may be configured such that transactions of data including medical images between servers in multiple facilities are managed by a distributed network. Further, the image management system may be configured to connect a plurality of blocks in chronological order in which the transaction history and the hash value of the previous block are recorded together. In addition, as a technology for checking consistency, etc., even if cryptography such as a quantum gate method that is difficult to calculate even using a quantum computer (for example, lattice cryptography, quantum cryptography using quantum key distribution, etc.) is used. good. Here, the image management system may be a device and a system that receives and stores images taken by a photographing device or images subjected to image processing. The image management system can also send images in response to requests from connected devices, perform image processing on stored images, and request image processing from other devices. can. The image management system may include, for example, a picture archiving and communication system (PACS). The image management system also includes a database that can store various information such as information about the subject and time of imaging associated with the received images. In addition, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with stored images in response to requests from other devices. .

Note that when additional learning is performed on various trained models, processing can be performed at high speed using the GPU. GPUs can perform efficient calculations by processing more data in parallel, so when learning multiple times using a learning model such as deep learning, it is possible to perform processing on the GPU. It is valid. Note that the additional learning process may be performed in cooperation with the GPU, CPU, and the like.

[Modification 8]
In the various embodiments and modified examples described above, the instructions from the examiner may be not only manual instructions (for example, instructions using a user interface or the like) but also instructions by voice or the like. At this time, for example, a machine learning model including a speech recognition model (speech recognition engine, trained model for speech recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by inputting characters using a keyboard, a touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (character recognition engine, trained model for character recognition) obtained by machine learning may be used. Further, the instruction from the examiner may be an instruction using a gesture or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, trained model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be the result of detecting the examiner's line of sight on the display screen of the output unit 103, or the like. The line of sight detection result may be, for example, a pupil detection result using a moving image of the examiner taken from the periphery of the display screen in the output unit 103. At this time, the pupil detection from the moving image may be performed using the object recognition engine as described above. Further, the instructions from the examiner may be instructions based on brain waves, weak electrical signals flowing through the body, or the like.

In such a case, for example, the learning data may be text data or audio data (waveform data) indicating instructions for displaying the results of processing the various trained models as described above, and The learning data may be an execution command for actually displaying the results of model processing on the output unit 103 as correct data. In addition, the learning data may be, for example, learning data whose correct answer data is an execution command for determining whether or not to automatically set shooting parameters, and an execution command for changing the button for the command to an active state. good. Furthermore, as the learning data, for example, an execution command for determining whether to perform high-quality image processing to obtain a high-quality image, and an execution command for changing the button for the command to an active state, etc., are used as correct answer data. It may be learning data. Note that any learning data may be used as long as the instruction content and the execution command content indicated by character data or audio data correspond to each other, for example. Furthermore, audio data may be converted into character data using an acoustic model, a language model, or the like. Furthermore, processing may be performed to reduce noise data superimposed on audio data using waveform data obtained by a plurality of microphones. Further, instructions using text or voice, etc., and instructions using a mouse, touch panel, etc. may be selectable in accordance with instructions from the examiner. Further, the device may be configured to be able to select whether to turn on or off instructions using text, voice, etc. in accordance with instructions from the examiner.

Here, machine learning includes deep learning as described above, and for example, RNN can be used for at least a part of a multilayer neural network. Here, as an example of a machine learning model according to this modification, an RNN, which is a neural network that handles time-series information, will be described with reference to FIGS. 15(a) and 15(b). Further, a long short-term memory (hereinafter referred to as LSTM), which is a type of RNN, will be explained with reference to FIGS. 16(a) and 16(b).

FIG. 15(a) shows the structure of RNN, which is a machine learning model. The RNN 1520 has a loop structure in its network, receives data x ^t 1510 at time t, and outputs data h ^t 1530. Since the RNN 1520 has a loop function in the network, it is possible to take over the state at the current time to the next state, so it can handle time-series information. FIG. 15(b) shows an example of input/output of the parameter vector at time t. The data x ^t 1510 includes N pieces of data (Params1 to ParamsN). Furthermore, the data h ^t 1530 output from the RNN 1520 includes N pieces of data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information during error backpropagation, LSTM is sometimes used. LSTM can learn long-term information by providing a forgetting gate, an input gate, and an output gate. Here, the structure of LSTM is shown in FIG. 16(a). In the LSTM 1640, the information that the network takes over at the next time t is the internal state c ^t-1 of the network called a cell and the output data h ^t-1 . Note that lowercase letters (c, h, x) in the figure represent vectors.

Next, details of the LSTM 1640 are shown in FIG. 16(b). In FIG. 16(b), a forgetting gate network FG, an input gate network IG, and an output gate network OG are shown, each of which is a sigmoid layer. Therefore, a vector in which each element has a value between 0 and 1 is output. The forgetting gate network FG determines how much past information to retain, and the input gate network IG determines which value to update. Further, in FIG. 16(b), a cell update candidate network CU is shown, and the cell update candidate network CU is an activation function tanh layer. This creates a vector of new candidate values that is added to the cell. The output gate network OG selects the elements of the cell candidates and selects how much information to convey at the next time.

Note that since the LSTM model described above is a basic form, it is not limited to the network shown here. Coupling between networks may be changed. Instead of LSTM, QRNN (Quasi Recurrent Neural Network) may be used. Furthermore, the machine learning model is not limited to neural networks, and boosting, support vector machines, etc. may also be used. Furthermore, if the examiner's instructions are input by text or voice, techniques related to natural language processing (for example, sequence to sequence) may be applied. At this time, as a technique related to natural language processing, for example, a model that is output for each input sentence may be applied. Furthermore, the various learned models described above may be applied not only to instructions from the examiner but also to outputs to the examiner. Further, a dialogue engine (a dialogue model, a trained model for dialogue) that responds to the examiner by outputting text or voice may be applied.

Further, as a technique related to natural language processing, a trained model obtained by pre-learning document data by unsupervised learning may be used. Further, as a technique related to natural language processing, a trained model obtained by further performing transfer learning (or fine tuning) on a trained model obtained by pre-learning according to the purpose may be used. Further, as a technique related to natural language processing, for example, BERT (Bidirectional Encoder Representations from Transformers) may be applied. Further, as a technology related to natural language processing, a model that can extract (express) the context (feature amount) by itself by predicting a specific word in a sentence from both left and right contexts may be applied. Furthermore, as a technology related to natural language processing, a model that can determine the relationship (continuity) between two sequences (sentences) in input time series data may be applied. Further, as a technology related to natural language processing, a model may be applied in which a Transformer encoder is used in the hidden layer and a sequence of vectors is input and output.

Here, instructions from the examiner to which this modification can be applied include changing the display of various images and analysis results as described in the various embodiments and modifications described above, and generating En-Face images. selection of depth range, selection of whether to use it as learning data for additional learning, selection of trained models, output (display, transmission, etc.) and storage of results obtained using various trained models, etc. Any instruction may be used as long as it includes at least one instruction. Further, instructions from the examiner to which this modification is applicable may be not only instructions after imaging but also instructions before imaging, such as instructions regarding various adjustments and instructions regarding setting various imaging conditions. , or an instruction regarding the start of shooting. Further, the instruction from the examiner to which this modification is applicable may be an instruction regarding changing the display screen (screen transition).

Note that the machine learning model may be a machine learning model that combines a machine learning model related to images such as CNN and a machine learning model related to time series data such as RNN. In such a machine learning model, for example, it is possible to learn the relationship between a feature amount related to an image and a feature amount related to time-series data. When the input layer side of a machine learning model is a CNN and the output layer side is an RNN, for example, a medical image is used as input data, and sentences related to the medical image (e.g. presence or absence of a lesion, type of lesion, recommendation for the next examination) are input. Learning may be performed using learning data whose output data is (e.g.). As a result, for example, medical information related to medical images is automatically explained in text, so even an examiner with little medical experience can easily understand medical information related to medical images. In addition, when the input layer side of the machine learning model is an RNN and the output layer side is a CNN, for example, text related to medical care such as lesions, findings, diagnoses, etc. is input data, and a medical image corresponding to the text related to medical care is output. Learning may be performed using learning data as data. Thereby, for example, the examiner can easily search for medical images related to the case he or she wants to confirm.

In addition, a machine translation engine (machine translation model, trained model for machine translation) that machine translates texts such as text and audio into any language may be used for instructions from the examiner and output to the examiner. good. Note that the arbitrary language may be configured to be selectable according to instructions from the examiner. Furthermore, any language may be configured to be automatically selectable by using a trained model that automatically recognizes the type of language. Furthermore, the automatically selected language type may be modified in accordance with instructions from the examiner. For example, the above-mentioned technology related to natural language processing (eg, sequence to sequence) may be applied to the machine translation engine. For example, after a sentence input to a machine translation engine is machine translated, the machine translated sentence may be input to a character recognition engine or the like. Further, for example, sentences output from the various trained models described above may be input to a machine translation engine, and the sentences output from the machine translation engine may be output.

Further, the various learned models described above may be used in combination. For example, characters corresponding to instructions from the examiner are input into a character recognition engine, and the audio obtained from the input characters is input into another type of machine learning engine (for example, a machine translation engine). may be done. Furthermore, for example, characters output from another type of machine learning engine may be input to the character recognition engine, and the voice obtained from the input characters may be output. Also, for example, the voice corresponding to instructions from the examiner is input into a speech recognition engine, and the characters obtained from the input voice are input into another type of machine learning engine (for example, a machine translation engine, etc.). may be configured. Further, for example, the configuration may be such that speech output from another type of machine learning engine is input to the speech recognition engine, and characters obtained from the input speech are displayed on the output unit 103. At this time, the apparatus may be configured to be able to select whether the output to the examiner is in text or voice according to an instruction from the examiner. Further, it may be configured such that it is possible to select whether to input text or voice as an instruction from the examiner. Furthermore, the various configurations described above may be adopted by selection based on instructions from the examiner.

[Modification 9]
Label images, high-quality images, and the like related to images acquired through actual photography may be stored in the storage unit 101-3 according to instructions from the operator. At this time, for example, after receiving an instruction from the operator to save a high-quality image, when registering a file name, any part of the file name (for example, the first part, or In the last part), the file name containing information (for example, characters) indicating that the image is generated by processing using a trained model for high image quality (high image quality processing) is specified by the operator. It may be displayed in an editable state according to instructions. Note that, similarly, for label images and the like, a file name including information that the image is an image generated by processing using a learned model may be displayed.

In addition, when displaying a high-quality image on the output unit 103 on various display screens such as a report screen, it is possible to confirm that the displayed image is a high-quality image generated by processing using a high-quality model. The display shown may be displayed together with a high-quality image. In this case, the display allows the operator to easily identify that the displayed high-quality image is not the image obtained by photography, thereby reducing misdiagnosis and improving diagnostic efficiency. be able to. Note that the display indicating that the image is a high-quality image generated by processing using a high-quality model may be any form as long as it is possible to distinguish between the input image and the high-quality image generated by the processing. It may also be from. In addition, it is shown that not only processing using a high-quality model but also processing using various trained models such as those described above are results generated by processing using that type of trained model. A display may be displayed with the results. For example, when displaying the analysis results of segmentation results using a trained model for image segmentation processing, the display indicating that the analysis results are based on the results using a trained model for image segmentation It may be displayed along with the results.

At this time, the display screen such as the report screen may be saved as image data in the storage unit 101-3 according to an instruction from the operator. For example, a report screen may be stored in the storage unit 101-3 as one image in which high-quality images, etc., and a display indicating that these images are images generated by processing using a trained model. Good too.

In addition, regarding the display indicating that the image is a high-quality image generated by processing using a high-quality model, there is also a display on the output section that indicates what kind of training data the high-quality model was trained with. 103 may be displayed. The display may include an explanation of the types of the input data and correct data of the learning data, and any display related to the correct data such as the imaged body part included in the input data and the correct data. Furthermore, for processing using the various trained models described above, such as image segmentation processing, for example, the output unit 103 displays a display indicating what kind of learning data was used to train the trained model of that type. May be displayed.

Further, information (for example, text) indicating that the image is generated by processing using a learned model may be displayed or saved in a state superimposed on the image. At this time, the location to be superimposed on the image may be any region (for example, the edge of the image) that does not overlap with the region in which the region of interest to be photographed is displayed. Alternatively, a non-overlapping area may be determined and the area may be overlapped with the determined area. Note that, in addition to the processing using the high image quality model, images obtained by processing using the various learned models described above, such as image segmentation processing, may also be processed in the same way.

In addition, if the initial display screen of the report screen is set so that the high image quality processing button etc. is in the active state (high image quality processing is turned on), the It may be configured such that a report image corresponding to a report screen including a quality image etc. is transmitted to the server. In addition, if the button is set to be active by default, it will be activated at the end of the examination (for example, when the shooting confirmation screen or preview screen is changed to the report screen in response to instructions from the examiner). Furthermore, the report image corresponding to the report screen including high-quality images may be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, whether or not the analysis map is superimposed, whether or not it is a high-quality image, the display screen for follow-up observation) A report image generated based on at least one setting such as whether or not the report image is sent to the server may be configured. Note that the same processing may be performed when the button represents switching of image segmentation processing.

[Modification 10]
In the embodiments and modifications described above, images obtained with the first type of trained model among the various trained models described above (e.g., high-quality images, images showing analysis results such as analysis maps, (an image showing a region recognition result, an image showing a segmentation result) may be input to a second type of trained model different from the first type. At this time, the configuration may be such that results (for example, analysis results, diagnosis results, area recognition results, and segmentation results) are generated by processing the second type of learned model.

Further, among the various trained models described above, the results of processing the first type of trained model (for example, analysis results, diagnosis results, area recognition results, segmentation results) are used to generate the first type of trained model. An image to be input to a second type of trained model different from the first type may be generated from an image input to the trained model. At this time, the generated image is highly likely to be an image suitable for processing using the second type of learned model. For this reason, images obtained by inputting the generated image to the second type of trained model (for example, high-quality images, images showing analysis results such as analysis maps, images showing area recognition results, and segmentation results) The accuracy of images shown) can be improved.

Note that by inputting a common image to the first type of trained model and the second type of trained model, it is possible to generate (or display) each processing result using these trained models. may be configured to execute. At this time, for example, the generation (or display) of each processing result using these learned models may be executed at once (in conjunction with each other) in response to an instruction from the examiner.

In addition, the types of images to be input (e.g., high-quality images, region recognition results, object recognition results, segmentation results, similar case images), and the types of processing results to be generated (or displayed) (e.g., high-quality images, region recognition results) , diagnosis results, analysis results, object recognition results, segmentation results, similar case images), input type, output type (e.g. text, voice, language), etc. can be selected according to instructions from the examiner. may be done. Furthermore, the type of input may be configured to be automatically selectable by using a trained model that automatically recognizes the type of input. Further, the type of output may be configured to be automatically selectable so as to correspond to (for example, be the same type) as the type of input. Furthermore, the automatically selected type may be modified in accordance with instructions from the examiner. At this time, at least one trained model may be selected depending on the selected type. At this time, if a plurality of trained models are selected, how to combine the plurality of trained models (for example, the order in which data is input, etc.) may be determined according to the selected type. Note that, for example, the type of image to be input and the type of processing result to be generated (or displayed) may be configured so that they can be selected differently, or if they are the same, they may be selected differently. It may be configured to output prompting information to the examiner.

Furthermore, each trained model may be executed at any location. For example, some of the plurality of trained models may be used in a cloud server, and the others may be configured to be used in another server such as a fog server or an edge server. Note that when configuring a network within a facility, within a site that includes a facility, within an area that includes multiple facilities, etc. to enable wireless communication, for example, if a network is configured to enable wireless communication, The reliability of the network may be improved by configuring it to use radio waves in a dedicated wavelength band. Furthermore, the network may be constructed using wireless communication that is capable of high speed, large capacity, low delay, and multiple simultaneous connections. As a result, for example, operations such as vitreous body, cataract, glaucoma, corneal refractive correction, external eye surgery, and treatments such as laser photocoagulation can be supported in real time even remotely. At this time, for example, a fog server or an edge server that wirelessly receives at least one of various medical images obtained by these devices related to surgery or treatment uses information obtained using at least one of various learned models. may be configured to wirelessly transmit the information to a surgical or therapeutic device. Further, for example, the information received wirelessly by a device related to surgery or treatment may be the amount of movement (vector) of an optical system or optical member as described above, and in this case, the device related to surgery or treatment is automatically controlled. It may also be configured to do so. Furthermore, for example, automatic control (semi-automatic control) with permission from the examiner may be configured for the purpose of supporting operations by the examiner.

Further, a similar case image search may be performed using an external database stored in a server or the like using the analysis results, diagnosis results, etc. obtained by processing the learned model as described above as a search key. Further, a similar case image search may be performed using an external database stored in a server or the like using object recognition results, segmentation results, etc. obtained by processing various trained models as described above as search keys. Note that in cases where multiple medical images stored in a database are already managed using machine learning, etc., with the feature amounts of each of the multiple medical images attached as additional information, the medical images themselves may be A similar case image search engine (similar case image search model, trained model for similar case image search) may be used as a search key. For example, the image processing device 101 uses a trained model for similar case image retrieval (different from a trained model for image quality improvement) to search for similar case images related to the medical image from various medical images. It can be performed.

Further, for example, the display control unit 101-5 can cause the output unit 103 to display similar case images obtained from various medical images using a learned model for searching similar case images. At this time, the similar case image is, for example, an image with a feature amount similar to the feature amount of the medical image input to the learned model. In addition, for example, when a medical image input to a trained model includes a partial region such as an abnormal region, the similar case image is an image with a feature amount similar to the feature amount of the partial region such as the abnormal region. . Therefore, for example, not only can learning be performed efficiently to accurately search for similar case images, but also when medical images include abnormal areas, the examiner can efficiently diagnose the abnormal areas. be able to. Further, a plurality of similar case images may be searched, and a plurality of similar case images may be displayed such that the order in which the feature amounts are similar can be identified. In addition, the trained model for similar case image search is additionally trained using training data that includes an image selected according to instructions from the examiner from among multiple similar case images and the feature values of the image. It may be configured so that

In addition, the learning data for various trained models is not limited to data obtained using the ophthalmological apparatus itself that performs actual imaging, but may also be data obtained using the same type of ophthalmological apparatus or data obtained using the same type of ophthalmological apparatus depending on the desired configuration. It may also be data obtained using an ophthalmological device.

Note that various learned models according to the embodiments and modifications described above can be provided in the image processing apparatus 101. The trained model may be configured with a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured with a circuit that performs a specific function such as an ASIC. Further, these trained models may be provided in another server device connected to the image processing device 101. In this case, the image processing apparatus 101 can use the trained model by connecting to a server or the like that includes the trained model via an arbitrary network such as the Internet. Here, the server provided with the trained model may be, for example, a cloud server, a fog server, an edge server, or the like. Note that when configuring a network within a facility, within a site that includes a facility, within an area that includes multiple facilities, etc. to enable wireless communication, for example, if a network is configured to enable wireless communication, The reliability of the network may be improved by configuring it to use radio waves in a dedicated wavelength band. Furthermore, the network may be constructed using wireless communication that is capable of high speed, large capacity, low delay, and multiple simultaneous connections.

[Modification 11]
The medical images processed by the image processing apparatus 101 according to the various embodiments and modifications described above include images acquired using any modality (imaging device, imaging method). The medical images to be processed can include medical images acquired with any imaging device or the like, and images created by a medical image processing device or a medical image processing method.

Furthermore, the medical image to be processed is an image of a predetermined region of a subject (subject), and the image of the predetermined region includes at least a part of the predetermined region of the subject. Further, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black and white image or a color image. Furthermore, the medical image may be an image representing the structure (morphology) of a predetermined region, or an image representing its function. Images representing functions include, for example, images representing blood flow dynamics (blood flow volume, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. The predetermined parts of the subject may be determined depending on the object to be imaged, and may include human eyes (eyes to be examined), organs such as the brain, lungs, intestines, heart, pancreas, kidneys, and liver, head, chest, Includes arbitrary parts such as legs and arms.

Furthermore, the medical image may be a tomographic image of the subject or a frontal image. The frontal image includes, for example, an SLO image of the fundus or anterior segment of the eye, a fluorescently photographed fundus image, and data acquired by OCT (three-dimensional OCT data) in at least a partial range in the depth direction of the object to be imaged. Contains the En-Face image generated using the En-Face image. The En-Face image is an OCTA En-Face image (motion contrast frontal image) generated using data of at least a partial range in the depth direction of the imaging target regarding the three-dimensional OCTA data (three-dimensional motion contrast data). ). Furthermore, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Here, the motion contrast data is data indicating changes among a plurality of volume data obtained by controlling the measurement light to scan the same area (same position) of the eye to be examined multiple times. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, by obtaining data indicating changes between a plurality of tomographic images obtained at substantially the same position at each different position, motion contrast data can be obtained as volume data. Note that the motion contrast front image is also called an OCTA front image (OCTA En-Face image) related to OCT angiography (OCTA) that measures the movement of blood flow, and the motion contrast data is also called OCTA data. Motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value) between two tomographic images or their corresponding interference signals. , may be determined by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measuring light to scan the same area (same position) of the eye to be examined multiple times. Note that when controlling the scanning means so that the measurement light scans approximately the same position multiple times, the time interval between one scan (one B scan) and the next scan (next B scan) ) may be changed (determined). Thereby, for example, even if the blood flow velocity varies depending on the condition of the blood vessel, the blood vessel region can be visualized with high accuracy. At this time, the time interval may be configured to be changeable, for example, according to instructions from the examiner. Further, for example, it may be configured such that one of the motion contrast images can be selected from a plurality of motion contrast images corresponding to a plurality of preset time intervals in response to an instruction from the examiner. Furthermore, for example, the storage unit 101-3 may be configured to be able to store the motion contrast data in association with the time interval at which the motion contrast data is acquired. Further, for example, the display control unit 101-5 may cause the output unit 103 to display the time interval at which the motion contrast data was acquired and the motion contrast image corresponding to the motion contrast data. Further, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. At this time, the time interval may be determined (output) from the motion contrast image using, for example, a machine learning model. Such a machine learning model, for example, uses multiple motion contrast images corresponding to multiple time intervals as input data, and calculates the difference between the multiple time intervals and the time interval when the desired motion contrast image was acquired as the correct answer. It can be obtained by learning learning data used as data.

Further, the En-Face image is, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions. At this time, the front image is data that corresponds to at least a partial depth range of volume data (three-dimensional tomographic image) obtained using optical interference, and that corresponds to a depth range determined based on two reference planes. It is generated by projecting or integrating onto a two-dimensional plane. The En-Face image is a frontal image generated by projecting data corresponding to a depth range determined based on the detected retinal layer out of the volume data onto a two-dimensional plane. Note that as a method of projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on the two-dimensional plane. method can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Further, the depth range related to the En-Face image is, for example, a range that includes a predetermined number of pixels in a deeper direction or a shallower direction with one of the two layer boundaries regarding the detected retinal layer as a reference. Good too. Furthermore, the depth range related to the En-Face image may be, for example, a range changed (offset) from the range between the two layer boundaries regarding the detected retinal layers in accordance with the operator's instructions. good.

Further, the photographing device is a device for photographing images used for diagnosis. The imaging device is, for example, a device that obtains an image of a predetermined region of a subject by irradiating the subject with radiation such as light, X-rays, electromagnetic waves, or ultrasound, or a device that detects radiation emitted from the subject. includes a device for obtaining an image of a predetermined region. More specifically, the imaging devices according to the various embodiments and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, and a fundus. Includes cameras, endoscopes, etc. Note that the configurations according to the above-described embodiments and modifications can be applied to these imaging devices. In this case, the movement of the subject corresponding to the movement of the eye to be predicted described above may be, for example, the movement of the face or body, the movement of the heart (heartbeat), or the like.

Note that the OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. Further, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a wavelength swept OCT (SS-OCT) device. Further, the OCT device may include a Line-OCT device (or SS-Line-OCT device) using line light. Further, the OCT device may include a Full Field-OCT device (or SS-Full Field-OCT device) using area light. Additionally, the OCT device may include a Doppler-OCT device. Furthermore, the SLO device and OCT device may include a wavefront compensated SLO (AO-SLO) device using a wavefront adaptive optical system, a wavefront compensated OCT (AO-OCT) device, and the like. Furthermore, the SLO device and OCT device may include a polarization SLO (PS-SLO) device, a polarization OCT (PS-OCT) device, etc. for visualizing information regarding polarization phase difference and depolarization. Furthermore, the SLO device and OCT device may include a pathological microscope SLO device, a pathological microscope OCT device, and the like. Furthermore, the SLO device and OCT device may include a handheld SLO device, a handheld OCT device, and the like. Further, the SLO device and OCT device may include a catheter SLO device, a catheter OCT device, and the like. Further, the SLO device and OCT device may include a head-mounted SLO device, a head-mounted OCT device, and the like. Furthermore, the SLO device and OCT device may include a binocular-type SLO device, a binocular-type OCT device, and the like. Further, the SLO device and the OCT device may be configured to have variable optical magnification, so that the imaging angle of view can be changed. In addition, the SLO device is capable of acquiring color images and fluorescence images by using RGB light sources and having a configuration in which one light receiving element receives light in a time-division manner or a configuration in which multiple light receiving elements simultaneously receive light. Good too.

Further, in the above-described embodiments and modified examples, the image processing device 101 is configured as a part of the OCT device, but the image processing device 101 may be configured separately from the OCT device. In this case, the image processing device 101 may be connected to the imaging device 100 of the OCT device or the like via the Internet or the like. Further, the configuration of the OCT device is not limited to the above configuration, and a part of the configuration included in the OCT device, such as an SLO imaging unit, may be configured separately from the OCT device.

In addition, in trained models for voice recognition, character recognition, gesture recognition, etc., the slope between input continuous time-series data values is extracted as part of the feature quantity and used for estimation processing. it is conceivable that. Such a trained model is expected to be able to perform highly accurate estimation by using the influence of temporal changes in specific numerical values in estimation processing. In addition, even in the trained models for high image quality, area recognition, segmentation processing, image analysis, and diagnostic result generation according to the above-mentioned embodiments and modifications, the brightness values of tomographic images, bright areas, It is thought that the order, slope, position, distribution, continuity, etc. of dark areas are extracted as part of the feature quantities and used in the estimation process.

(Other embodiments)
The present invention provides software (programs) that implement one or more functions of the various embodiments and modifications described above to a system or device via a network or a storage medium, and the computer of the system or device executes the programs. This can also be realized by reading and executing processing. A computer has one or more processors or circuits and may include separate computers or a network of separate processors or circuits for reading and executing computer-executable instructions.

The processor or circuit may include a central processing unit (CPU), microprocessing unit (MPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), or field programmable gateway (FPGA). The processor or circuit may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Although the present invention has been described above with reference to the embodiments and modified examples, the present invention is not limited to the above embodiments and modified examples. The present invention includes inventions modified within the scope of the spirit of the present invention and inventions equivalent to the present invention. Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of the invention.

101: Image processing device, 101-47: Image quality improvement unit, 200: Eye to be examined (subject), 301: Image quality improvement processing unit (high image quality improvement unit) 302: Area recognition unit, 304: Blend processing unit (image processing unit), 305: BC adjustment unit (image processing unit)

Claims (27)

By performing image quality enhancement processing on the first image, which is the medical image of the subject, using a trained model obtained through learning using the medical image of the subject as learning data, the medical image of the subject is a high image quality improvement unit that obtains a second image that is
a detection unit that detects a target area in a medical image of a subject;
An image in which image processing is performed on the target area in the second image so that the difference between the pixel value of the target area and the pixel value of an area other than the target area increases and the pixel value of the target area becomes lower. a processing section;
A display control unit that displays a high-quality image obtained by inputting a medical image of the subject as input data of a trained model obtained by learning learning data including images related to the subject as a live moving image on the display unit. and,
Equipped with
The display control unit is configured to display, as the live video image, a frontal image obtained as the high-quality image, on which a line indicating the position of the tomographic image obtained as the high-quality image is superimposed and displayed. The image processing device displays the tomographic image corresponding to the position of the line on the front image on the display unit as the live moving image. The image processing includes processing to blend the first image and the second image so that a difference between a pixel value of a target region and a pixel value of a region other than the target region is widened. image processing device. The detection unit detects a target area using the first image,
The image processing device according to claim 1 or 2, wherein the image processing unit performs the image processing on a target area in the second image that corresponds to a detected target area in the first image. The image processing device according to claim 1 or 2, wherein the detection unit detects a target area using the second image. a detection unit that detects a target area in a medical image of a subject;
With respect to the target region in the first image, which is a medical image of the subject, the difference between the pixel value of the target region and the pixel value of a region other than the target region is widened, and the pixel value of the target region is lowered. an image processing unit that performs image processing;
By performing image quality enhancement processing on the first image that has been subjected to the image processing using a trained model obtained by learning using the medical image of the subject as learning data, the medical image of the subject is a high image quality improvement unit that obtains a second image that is
An image processing device comprising: The image processing device according to claim 5, wherein the detection unit detects a target area using the first image. The image processing device according to claim 5, wherein the detection unit detects the target area using an image obtained by performing high-quality processing on the first image on which the image processing has not been performed. The image processing device according to any one of claims 1 to 7, wherein the detection unit detects the target area using a learned model obtained by learning using a medical image of a subject as learning data. Medical images are frontal images;
9. The detection unit detects a two-dimensional target area using a trained model obtained by learning using a plurality of frontal images of a subject corresponding to a plurality of depth ranges as learning data. The image processing device according to any one of the items. Medical images are frontal images;
The detection unit selects a depth selected according to an instruction from the examiner from among a plurality of trained models obtained through learning using a plurality of frontal images of the subject corresponding to a plurality of depth ranges as learning data respectively. The image processing device according to any one of claims 1 to 8, wherein a trained model corresponding to a range is selected, and a two-dimensional target area is detected using the selected trained model. A medical image is a three-dimensional medical image,
The detection unit detects a three-dimensional target area using a trained model obtained by learning using a three-dimensional medical image of a subject as learning data. image processing device. The image according to any one of claims 1 to 3, 5, and 6, wherein the detection unit uses the first image to detect the target area by rule-based processing based on the structure of the subject. Processing equipment. 13. The image processing includes processing for correcting at least one of brightness and contrast so as to widen the difference between pixel values in the target area and pixel values in areas other than the target area. The image processing device described in . The medical image of the subject is a motion contrast image of the subject's eye,
The image processing device according to any one of claims 1 to 13, wherein the target area includes at least one of a non-perfused area, a foveal blood vessel area, and an optic disc area. The medical image of the subject was used as the learning data, and the medical image of the subject was subjected to image quality enhancement processing using the first trained model obtained through learning using the medical image of the subject as learning data, and the image processing in the target area was performed. By performing image quality enhancement processing on the first image, which is a medical image of the subject, using the second trained model obtained through learning, the second image, which is a medical image of the subject, is A high image quality unit to obtain,
A display control unit that displays a high-quality image obtained by inputting a medical image of the subject as input data of a trained model obtained by learning learning data including images related to the subject as a live moving image on the display unit. and,
Equipped with
The display control unit is configured to display, as the live video image, a frontal image obtained as the high-quality image, on which a line indicating the position of the tomographic image obtained as the high-quality image is superimposed and displayed. The image processing device displays the tomographic image corresponding to the position of the line on the front image on the display unit as the live moving image. A display control unit that displays a high-quality image obtained by inputting a medical image of the subject as input data of a trained model obtained by learning learning data including images related to the subject as a live moving image on the display unit . The image processing device according to any one of claims 5 to 7 , further comprising: The image processing device according to claim 16, wherein the display control unit displays the anterior eye image obtained as the high-quality image on the display unit as the live image . The display control unit is configured to select, as the live video image , a frontal image obtained as the high-quality image, on which a line indicating the position of the tomographic image obtained as the high-quality image is superimposed and displayed. The image processing device according to claim 16 or 17, wherein the display unit displays the tomographic image corresponding to the position of the line on the front image as the live moving image. The display control unit displays information indicating a blood vessel region in the tomographic image generated as the high-quality image corresponding to the position of the line, superimposed on the tomographic image corresponding to the position of the line. The image processing apparatus according to any one of claims 1 to 4, 15, and 18. By performing high-quality processing on the first image, which is the medical image of the subject, using a trained model obtained through training using the medical image of the subject as training data, the medical image of the subject is a high image quality improvement unit that obtains a certain second image;
A frontal image on which a line indicating the position of the tomographic image, which is the second image, is superimposed is displayed on the display unit as a live moving image, and the tomographic image corresponding to the line on the frontal image is displayed as a live moving image. a display control unit that causes the display unit to display information;
An image processing device comprising: The display control section includes:
An analysis result generated using a trained model for generating analysis results obtained by learning training data including images related to a subject, the analysis results obtained by inputting images related to the subject;
Diagnosis results generated using a trained model for generating diagnostic results obtained by learning learning data including images related to the subject, and obtained by inputting images related to the subject;
An image generated using a generative adversarial network or an autoencoder, which is obtained by inputting an image related to a subject, and an image related to the subject inputted to the generative adversarial network or autoencoder. Information regarding the abnormal area, which is information regarding the difference;
A similar case image searched using a trained model for searching similar case images obtained by learning learning data including images related to the subject, and similar case images obtained by inputting images related to the subject. ,
Object recognition results or segmentation results generated using a trained model for object recognition or a trained model for segmentation obtained by learning learning data that includes images related to the subject, and input images related to the subject. The image processing device according to any one of claims 1 to 4 and 15 to 20 , wherein at least one of the obtained object recognition result or segmentation result is displayed on the display unit. The operator's instruction to obtain the second image uses at least one trained model among a trained model for character recognition, a trained model for speech recognition, and a trained model for gesture recognition. The image processing device according to any one of claims 1 to 21 , wherein the image processing device is information obtained by. By performing image quality enhancement processing on the first image, which is the medical image of the subject, using a trained model obtained through learning using the medical image of the subject as learning data, the medical image of the subject is obtaining a second image with
Detecting a target area in a medical image of a subject;
Image processing is performed on the target area in the second image so that the difference between the pixel value of the target area and the pixel value of an area other than the target area is widened and the pixel value of the target area is lowered. and,
Displaying a high-quality image obtained by inputting a medical image of the subject as input data of a trained model obtained by learning learning data including images related to the subject as a live video image on a display unit;
including;
The displaying includes displaying, as the live video image, a frontal image obtained as the high-quality image, on which a line indicating the position of the tomographic image obtained as the high-quality image is superimposed and displayed. and displaying the tomographic image corresponding to the position of the line on the front image on the display unit as the live moving image. Detecting a target area in a medical image of a subject;
With respect to the target region in the first image, which is a medical image of the subject, the difference between the pixel value of the target region and the pixel value of a region other than the target region is widened, and the pixel value of the target region is lowered. Performing image processing on
By performing image quality enhancement processing on the first image that has been subjected to the image processing using a trained model obtained by learning using the medical image of the subject as learning data, the medical image of the subject is obtaining a second image with
image processing methods, including; Learning using a medical image of a subject as learning data, which has been subjected to image quality enhancement processing using the first trained model obtained through learning using a medical image of the subject as learning data, and image processing in the target area. A second image, which is a medical image of the subject , is obtained by performing image quality enhancement processing on the first image, which is the medical image of the subject, using the second trained model obtained by to do and
Displaying a high-quality image obtained by inputting a medical image of the subject as input data of a trained model obtained by learning learning data including images related to the subject as a live video image on a display unit;
including ;
The displaying includes displaying, as the live video image, a frontal image obtained as the high-quality image, on which a line indicating the position of the tomographic image obtained as the high-quality image is superimposed and displayed. and displaying the tomographic image corresponding to the position of the line on the front image on the display unit as the live moving image. By performing high-quality processing on the first image, which is the medical image of the subject, using a trained model obtained through training using the medical image of the subject as training data, the medical image of the subject is obtaining a certain second image;
A frontal image on which a line indicating the position of the tomographic image, which is the second image, is superimposed is displayed on the display unit as a live moving image, and the tomographic image corresponding to the line on the frontal image is displayed as a live moving image. displaying it on the display section;
image processing methods, including A program that, when executed by a computer, causes the computer to execute each step of the image processing method according to any one of claims 23 to 26 .

Images