JP2022155690A - Image processing device, image processing method, and program - Google Patents

Abstract

To use a medical image of an image size suitable for learning and inference (estimation) as input data for a learned model.SOLUTION: An image processing device is a learned model obtained by learning by learning data including input data which is a medical image having a first image size, and includes an image processing part for outputting a second image having a second image size as output data by inputting, to the learned model, a first image which is a medical image having a second image size larger than the first image size, as input data.SELECTED DRAWING: Figure 4

Description

The technology disclosed herein relates to an image processing device, an image processing method, and a program.

An apparatus (OCT apparatus) using optical coherence tomography (OCT) is known as an apparatus for obtaining a tomographic image of a subject. By using a medical tomographic imaging apparatus such as an OCT apparatus, it is possible to observe the state inside the retinal layers three-dimensionally, and such a medical tomographic imaging apparatus is useful for ophthalmic retinal diseases such as AMD. is useful for the diagnosis of In recent years, OCT used in clinical practice is broadly classified into two methods, for example, SD-OCT (Spectral Domain OCT) and SS-OCT (Swept Source OCT) as a method of acquiring images at high speed. SD-OCT uses a broadband light source and acquires an interferogram with a spectrometer. In contrast, SS-OCT uses a fast wavelength swept light source as a light source to measure spectral interference with a single-channel photodetector.

Recently, OCT angiography (OCTA), in which blood vessels are imaged without using a contrast agent, has attracted attention in both types of OCT. OCTA generates motion contrast data from OCT images acquired by OCT. Here, the motion contrast data is data obtained by repeatedly photographing the same section of the object to be measured by OCT and detecting a temporal change in the object to be measured between the photographing. is calculated from the difference, ratio, correlation, or the like.

In addition, when displaying an OCTA image, it is becoming common to display a two-dimensional OCTA front image by projecting three-dimensional motion contrast data calculated from an acquired three-dimensional OCT image onto a two-dimensional plane. be. In this regard, Patent Document 1 discloses a technique for generating a two-dimensional front image by designating a range in the depth direction of motion contrast data to be projected. Furthermore, Patent Document 2 discloses a technique of generating a high-resolution image from a low-resolution image by an artificial intelligence engine obtained by machine learning using a low-resolution image and a high-resolution image.

JP 2017-6179 A JP 2018-5841 A

Here, generally, the larger the number of learning data, the better. Inference processing (estimation processing) should be performed at high speed.

Therefore, one of the purposes of the technology disclosed herein is to use medical images having an image size suitable for learning and inference (estimation) as input data for a trained model.

In addition to the above object, it is also another object of the present invention to achieve functions and effects that are derived from each configuration shown in the mode for carrying out the invention described later and that cannot be obtained by the conventional technology. can be positioned as one.

An image processing apparatus according to at least one embodiment of the technology disclosed herein is a trained model obtained by learning using learning data including input data that is a medical image having a first image size, By inputting a first image, which is a medical image having a second image size larger than the image size of , to the trained model as input data, a second image having the second image size is output as output data and an image processing unit that outputs as

According to at least one embodiment of the disclosed technology, medical images having an image size suitable for learning and inference (estimation) can be used as input data for a trained model.

1 is a block diagram showing a schematic configuration of an OCT apparatus according to a first embodiment; FIG. It is a figure explaining the schematic structure of the imaging device which concerns on 1st Embodiment. 3 is a block diagram showing a schematic configuration of an image quality enhancing unit according to the first embodiment; FIG. An example of the configuration of a neural network for an image quality enhancement engine is shown. An example of learning data related to image quality improvement processing is shown. An example of an input image for image quality improvement processing is shown. FIG. 4 is a flow diagram showing a schematic flow of image processing according to the first embodiment; 4 is a flow chart showing an example of the flow of image processing according to the first embodiment; FIG. 4 is a flow chart showing an example of the flow of image processing according to the first embodiment; FIG. 4 is a flow chart showing an example of the flow of image processing according to the first embodiment; FIG. 4 is a flow chart showing an example of the flow of image processing according to the first embodiment; FIG. 4 is a flow chart showing an example of the flow of image processing according to the first embodiment; FIG. 4 is a flow chart showing an example of the flow of image processing according to the first embodiment; FIG. FIG. 11 is a flowchart showing a schematic flow of image processing according to the second embodiment; 11 shows an example of a machine learning model according to Modification 8. FIG. 11 shows an example of a machine learning model according to Modification 8. FIG. An example of a plurality of subset regions obtained by dividing an input image is shown. An example of Self-Attention used in Transformer is shown.

Exemplary embodiments for carrying out the present invention will now be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of components, etc. described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, the same reference numbers are used in the drawings to indicate identical or functionally similar elements. Also, hereinafter, the axial direction of the eye is described as Z, the horizontal direction of the fundus plane is described as X, and the vertical direction of the fundus plane is described as Y.

In the following description, a machine learning model refers to a learning model based on a machine learning algorithm. Specific algorithms of machine learning include nearest neighbor method, naive Bayes method, decision tree, support vector machine, and the like. Another example is deep learning in which a neural network is used to generate feature values and connection weighting coefficients for learning. As appropriate, any of the above algorithms can be used and applied to the following embodiments and modifications. Also, teacher data refers to learning data, and is composed of a pair of input data and output data. Further, correct data means output data of learning data (teaching data).

A trained model is a model that has undergone training (learning) in advance using appropriate teacher data (learning data) for a machine learning model that follows any machine learning algorithm such as deep learning. . However, although the trained model is obtained in advance using appropriate training data, it is not the case that further learning is not performed, and additional learning can be performed. Additional learning can also take place after the device has been installed at the point of use.

(First embodiment)
An image processing system according to a first embodiment of the present invention will be described in detail below with reference to FIGS. 1 to 13. FIG. In this embodiment, as an example of an image processing system, an OCT apparatus including an image processing apparatus that processes a tomographic image of a subject obtained by OCT will be described.

(Configuration of image processing device)
The configuration of the image processing apparatus 101 according to the present embodiment and connection with other devices will be described with reference to FIG. The image processing apparatus 101 is connected to the photographing apparatus 100 , the external storage device 102 , the output section 103 and the input section 104 . These connections may be wired connections or wireless connections. Also, these connections may be connections via a network. For example, the image processing device 101 may be connected to the imaging device 100 via a network such as the Internet. Further, for example, the external storage device 102 may be placed on a network such as the Internet so that data can be shared by a plurality of image processing apparatuses.

The image processing apparatus 101 is provided with an acquisition unit 101-1, an imaging control unit 101-2, an image processing unit 101-4, a display control unit 101-5, and a storage unit 101-3, which are functional blocks. there is The image processing apparatus 101 can be configured using a general computer including a processor, memory, etc., but may be configured as a dedicated computer for the OCT apparatus. Here, the image processing apparatus 101 may be a built-in (internal) computer of the OCT apparatus, or may be a separate (external) computer communicably connected to the OCT apparatus. Also, the image processing apparatus 101 may be, for example, a personal computer, and may be a desktop PC, a notebook PC, or a tablet PC (portable information terminal). Note that the processor may be a CPU (Central Processing Unit). Also, the processor may be, for example, an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or an FPGA (Field-Programmable Gate Array).

Each function of the image processing apparatus 101 may be implemented by a processor such as a CPU or MPU executing software modules stored in the storage unit 101-3. Note that the processor may be, for example, a GPU, FPGA, or the like. Also, each function may be configured by a circuit or the like that performs a specific function, such as an ASIC. For example, the image processing unit 101-4 may be realized by dedicated hardware such as ASIC, and the display control unit 101-5 may be realized by using a dedicated processor such as a GPU different from the CPU. The storage unit 101-3 may be configured by an arbitrary storage medium such as an optical disk such as a hard disk or a memory, for example.

The acquisition unit 101-1 acquires signal data such as an SLO image, a tomographic image, and an anterior segment image obtained by imaging the subject with the imaging device 100, an image such as an SLO image, a tomographic image, and an anterior observation image, and patient information. It is a functional block that acquires Further, the acquisition unit 101-1 can generate an image such as an SLO image, a tomographic image, or an anterior observation image using the acquired signal data. The acquisition unit 101-1 is provided with a tomographic image generation unit 101-11 and a motion contrast data generation unit 101-12. The acquisition unit 101-1 may acquire various data including images from an external device (not shown) such as a server connected to the image processing apparatus 101. FIG. Here, the image processing apparatus 101 and an external device (not shown) may be connected via any network such as the Internet. Acquisition unit 101-1 may acquire various data stored in storage unit 101-3.

The tomographic image generating unit 101-11 performs signal processing on the acquired signal data (interference signal) of the tomographic image to generate a tomographic image, and stores the generated tomographic image in the storage unit 101-3. Any known method may be used for generating a tomographic image from an interference signal.

The motion contrast data generation unit 101-12 generates motion contrast data based on a plurality of tomographic images at approximately the same position (mutually corresponding regions of the subject) generated by the tomographic image generation unit 101-11. A method of generating motion contrast data will be described below.

First, the tomographic image generation unit 101-11 generates a plurality of tomographic images from a plurality of interference signals acquired by photographing substantially the same position of the subject a plurality of times. Here, a scanning group of measurement light when photographing substantially the same position of the object a plurality of times to generate motion contrast data is referred to as one cluster. Also, a plurality of tomographic images corresponding to a plurality of interference signals acquired by imaging the same position of the subject a plurality of times is called a tomographic image for one cluster (one group). More specifically, the tomographic image generating unit 101-11 performs wavenumber transform, fast Fourier transform (FFT), and absolute value transform ( Acquisition of amplitude) is performed to generate a tomographic image for one cluster. Note that the method of generating a tomographic image is not limited to this, and any other known image processing method may be used.

Next, the alignment unit 101-41 of the image processing unit 101-4, which will be described later, aligns the tomographic images belonging to the same cluster and performs superimposition processing. After that, the image feature acquisition unit 101-44 of the image processing unit 101-4, which will be described later, acquires layer boundary data from the superimposed tomographic image. In this embodiment, a deformable model is used as a layer boundary acquisition method, but any known layer boundary acquisition method may be used. Note that the layer boundary acquisition process is not essential. For example, if the motion contrast image is generated only in three dimensions and a two-dimensional motion contrast image projected in the depth direction is not generated, the layer boundary acquisition process can be omitted. .

The motion contrast data generator 101-12 calculates motion contrast data between adjacent tomographic images within the same cluster. In this embodiment, the motion contrast data generator 101-12 obtains a decorrelation value Mxz as motion contrast data based on the following equation (1).

Here, Axz indicates the amplitude (of complex number data after FFT processing) at position (x, z) of tomographic image data A, and Bxz indicates the amplitude of tomographic data B at the same position (x, z). 0≦Mxz≦1, and takes a value closer to 1 as the difference between the two amplitude values increases. The motion contrast data generation unit 101-12 performs decorrelation calculation processing such as Equation (1) between arbitrary temporally adjacent tomographic images (belonging to the same cluster). It should be noted that the tomographic images to be subjected to decorrelation arithmetic processing do not necessarily have to be temporally adjacent as long as they are images within a predetermined period. The motion contrast data generator 101-12 generates, as a final motion contrast image, an image having, as a pixel value, the average value of the obtained motion contrast data (number of tomographic images per cluster−1).

Although motion contrast data is calculated here based on the amplitude of complex number data after FFT processing, the motion contrast data calculation method is not limited to the above. For example, motion contrast data may be calculated based on the phase information of the complex data, or may be calculated based on both amplitude and phase information. Alternatively, motion contrast data may be calculated based on the real and imaginary parts of complex data.

Also, in the present embodiment, the decorrelation value is calculated as the motion contrast data, but the calculation method of the motion contrast data is not limited to this, and any known calculation method may be used. For example, motion contrast data may be calculated based on the difference between two values, or motion contrast data may be calculated based on the ratio of two values. Motion contrast data can also be obtained, for example, as a variance value between two tomographic images or their corresponding interference signals, or as a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value). Any known method may be used for these calculation methods.

Also, when controlling the scanning means so that the measurement light scans approximately the same position a plurality of times, the time interval between one scan (one B scan) and the next scan (next B scan) ) may be changed (determined). As a result, for example, even if the blood flow velocity varies depending on the state of the blood vessel, the blood vessel region can be visualized with high accuracy.

At this time, for example, the time interval may be configured to be changeable according to an instruction from an operator (examiner). Further, for example, any motion contrast image may be selectable from a plurality of motion contrast images corresponding to a plurality of preset time intervals according to an instruction from the operator. Further, for example, the time interval when the motion contrast data is acquired and the motion contrast data may be associated with each other and stored in the storage unit 101-3. Further, for example, the display control unit 101-5 may cause the output unit 103 to display the time interval when the motion contrast data is acquired and the motion contrast image corresponding to the motion contrast data. Further, for example, the time interval may be determined automatically, or at least one candidate for the time interval may be determined. At this time, for example, a machine learning model may be used to determine (output) the time interval from the motion contrast image. Such a machine learning model, for example, uses a plurality of motion contrast images corresponding to a plurality of time intervals as input data, and corrects the difference from the plurality of time intervals to the time interval when the desired motion contrast image is acquired. It can be obtained by learning learning data to be data.

Furthermore, here, the final pixel value of the motion contrast image is obtained by calculating the average value of the obtained multiple decorrelation values, but the final pixel value is not limited to this. For example, an image having, as a pixel value, the median value or the maximum value of a plurality of acquired decorrelation values may be generated as the final motion contrast image.

The imaging control unit 101 - 2 is a functional block that controls imaging with respect to the imaging apparatus 100 . For example, the imaging control unit 101-2 can drive and control a light source, a scanning unit, a focusing lens driving device, and the like included in the imaging apparatus 100. FIG. The imaging control unit 101-2 can control the alignment operation of the stage unit 100-2 of the imaging apparatus 100, which will be described later. Furthermore, the imaging control unit 101-2 can also instruct the imaging apparatus 100 to set imaging parameters and to start or end imaging.

The storage unit 101-3 can store an operating system (OS), device drivers for peripheral devices, and programs for realizing various types of application software including programs for performing processes described later. The storage unit 101-3 can also store information acquired by the acquisition unit 101-1, various images processed by the image processing unit 101-4, and the like. For example, the storage unit 101-3 stores a medical image such as a tomographic image acquired by the acquisition unit 101-1, or stores an image enhanced in image quality by an image enhancement unit 101-47, which will be described later. be able to.

The image processing unit 101-4 is a functional block that performs image processing on various images such as tomographic images, motion contrast images, SLO images, and anterior eye observation images. The image processing unit 101-4 includes a positioning unit 101-41, a synthesizing unit 101-42, a correcting unit 101-43, an image feature acquiring unit 101-44, a projecting unit 101-45, an analyzing unit 101-46, and an image processing unit 101-46. An image quality improving unit 101-47 is provided.

The alignment unit 101-41 is a functional block that performs alignment processing between images. For example, the aligning unit 101-41 can align tomographic images belonging to the same cluster and perform overlay processing. Note that the alignment processing and superimposition processing may be performed using any known technique.

The synthesizer 101-42 is a functional block that synthesizes one image from each of a plurality of two-dimensional images. The synthesizing unit 101-42 is provided with, for example, a synthesizing method specifying unit 101-421, a same modality image synthesizing unit 101-422, and a different modality image synthesizing unit 101-423.

The combining method designating unit 101-421 designates the type of image to be combined (eg, tomographic image/motion contrast image/tomographic image and motion contrast image) and the combining processing method (eg, superimposition/stitching/side-by-side display). do. The same modality image synthesizing unit 101-422 performs synthesizing processing between tomographic images or between motion contrast images, for example. The heteromodality image synthesizing unit 101-423 performs synthesizing processing between images obtained by different modalities, such as between a tomographic image and a motion contrast image.

The correction unit 101-43 is a functional block that performs processing for suppressing projection artifacts that occur in motion contrast images. Here, the projection artifact is a phenomenon in which the motion contrast in the superficial retinal blood vessels is reflected on the deep layer side (deep retina, outer retina, and choroid), and a high decorrelation value occurs in the deep layer region where no blood vessels actually exist. Point. For example, the corrector 101-43 performs processing to reduce projection artifacts in the generated motion contrast data. Therefore, the correction unit 101-43 corresponds to an example of a processing unit that performs processing for reducing projection artifacts on the generated motion contrast data.

The image feature acquisition unit 101-44 acquires layer boundary data from a tomographic image, a motion contrast tomographic image, or the like. Specifically, the image feature acquisition unit 101-44 performs image segmentation processing on a tomographic image or the like, extracts the layer structure in the tomogram of the subject's eye, and acquires layer boundary data such as boundary positions. Note that any known method may be used as the image segmentation processing method.

The projection unit 101-45 is a functional block that projects or integrates a tomographic image or motion contrast image in a set depth range to generate a luminance front image (luminance En-Face image) or an OCTA front image. Note that the depth range can be set using two reference planes based on an operator's instruction or layer boundary data acquired by the image feature acquisition unit 101-44. The set depth range may be any depth range. For example, it is possible to set the depth ranges of the surface layer of the retina and the outer layer of the retina, and generate two types of synthetic OCTA enface images by the synthesizing unit 101-42.

Here, as a method of projecting data corresponding to a depth range set based on two reference planes onto a two-dimensional plane, for example, a representative value of data within the depth range is used as a pixel value on the two-dimensional plane. method can be used. The representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of the area surrounded by the two reference planes. Specifically, as a projection method, for example, maximum intensity projection (MIP) or average intensity projection (AIP) can be selected.

Also, the depth range of the front image may be, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with reference to one of the two layer boundaries of the detected retinal layer. . Further, the depth range related to the En-Face image is, for example, a range changed (offset) according to the operator's instruction from the range between two layer boundaries related to the detected retinal layer. good. Here, the depth range for generating the front image can be changed by the operator selecting from a default depth range set displayed in a selection list or the like (not shown). In addition, the operator can change the type and offset position of the layer boundary used to specify the projection range from the user interface (UI), or operate the input unit 104 to move the layer boundary data superimposed on the tomographic image. You can also change the projection range by

The generated luminance front image and OCTA front image can be output by the output unit 103 . The motion contrast image output to the output unit 103 is not limited to the OCTA front image, and may be a three-dimensional rendered three-dimensional motion contrast image or a motion contrast tomographic image corresponding to a tomographic image. good. When the generated motion contrast image or the like is displayed by the output unit 103, the above-described projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a UI such as a context menu. For example, a motion contrast image after projection artifact suppression processing may be displayed as a three-dimensional image.

The analysis unit 101-46 is a functional block that analyzes various images such as tomographic images and motion contrast images. The analysis unit 101-46 is provided with an enhancement unit 101-461, an extraction unit 101-462, a measurement unit 101-463, and a comparison unit 101-464.

The enhancement unit 101-461 can, for example, perform enhancement processing for enhancing data in an arbitrary region in an image in accordance with instructions from the operator. The extraction unit 101-462 can extract features, regions, etc. in the image used by the analysis unit 101-46. For example, the extraction unit 101-462 can acquire the layer boundaries of the retina and choroid, the boundaries of the anterior and posterior surfaces of the cribriform plate, the positions of the fovea fovea and the center of the optic disc, and the like from the tomographic image. Also, the extraction unit 101-462 can extract a blood vessel region from the OCTA front image. The measurement unit 101-463, for example, calculates measurement values to be analyzed from various images. For example, the measurement unit 101-463 calculates the thickness of a specific layer, or uses the blood vessel center line data obtained by thinning the extracted blood vessel region or the blood vessel region to obtain measured values such as blood vessel density. can be calculated. Also, the comparison unit 101-464 can compare multiple images such as multiple tomographic images and multiple motion contrast images. Further, the comparison unit 101-464 can also compare analysis results of multiple images, such as measurement values calculated by the measurement unit 101-463.

The image quality enhancing unit 101-47 is a functional block that enhances the image quality of various images. Therefore, the image quality enhancing unit 101-47 is an example of an image quality enhancing unit that enhances the image quality of medical images such as tomographic images and OCTA front images. Details of the configuration and functions of the image quality enhancing unit 101-47 will be described later.

The display control unit 101-5 can control the display of the output unit 103 connected to the image processing apparatus 101. FIG. The display control unit 101-5 can cause the output unit 103 to display, for example, various data such as a tomographic image of the subject's eye and patient information.

The external storage device 102 can store programs for tomography, patient information, image information, and the like. For example, the external storage device 102 stores patient information (patient name, age, sex, etc.), eye information (left eye, right eye, eye axial length, etc.), captured images (tomographic images, SLO images, and OCTA images, etc.), composite images, imaging parameters, image data and measurement data of past examinations, parameters set by the operator, and the like can be stored in association with each other.

The input unit 104 includes, for example, a mouse, a keyboard, a touch panel, and the like, and the operator can give instructions to the image processing apparatus 101 and the image capturing apparatus 100 via the input unit 104 . The output unit 103 can output data such as various images processed by the image processing apparatus 101 . The output unit 103 can be composed of an arbitrary display or the like, and can display, for example, patient information and various images related to the subject under the control of the display control unit 101-5. In this case, the output unit 103 can function as an example of a display unit that displays various images and information. Note that the output unit 103 may have a touch UI or the like.

A part of the configuration of the OCT apparatus may be configured as a separate device, or may be configured as an integrated device. For example, the output unit 103 may be integrated with the input unit 104 as a touch panel display.

(Structure of imaging device)
The imaging apparatus 100 is an apparatus for imaging an eye to be inspected in order to obtain a tomographic image, an SLO image, or the like of the eye to be inspected. The configuration of the measurement optical system and the spectroscope in the imaging apparatus 100 according to this embodiment will be described below with reference to FIGS. 1 and 2. FIG. In this embodiment, an imaging device including an SD-OCT (Spectral Domain OCT) optical system is used as the imaging device 100 . Not limited to this, for example, an imaging device including an optical system such as SS-OCT or TD-OCT (Time Domain OCT) may be used.

As shown in FIG. 1, the imaging device 100 is provided with a measurement optical system 100-1, a stage section 100-2, and a base section 100-3. The measurement optical system 100-1 is an optical system for obtaining an anterior segment image, an SLO fundus image of the subject's eye, and a tomographic image. The stage unit 100-2 holds the measurement optical system 100-1 so as to be movable in the front, rear, left, and right directions, and can move the measurement optical system 100-1 under the control of the imaging control unit 101-2. The base section 100-3 incorporates a spectroscope 230, which will be described later.

The configuration of the measurement optical system 100-1 will be described with reference to FIG. In the measurement optical system 100-1, an objective lens 201 is installed facing an eye 200 to be inspected, and a first dichroic mirror 202 and a second dichroic mirror 203 are arranged on the optical axis thereof. The optical path from the objective lens 201 is split by these dichroic mirrors into an optical path 250 for the OCT optical system, an optical path 251 for the SLO optical system and fixation lamp, and an optical path 252 for anterior eye observation for each wavelength band. .

In this embodiment, an optical path 252 for anterior eye observation is arranged in the reflection direction of the first dichroic mirror 202, and an optical path 250 for the OCT optical system and an optical path 250 for the SLO optical system and the fixation lamp are arranged in the transmission direction of the first dichroic mirror 202. of optical paths 251 are arranged. An optical path 250 for the OCT optical system is arranged in the reflection direction of the second dichroic mirror 203 , and an optical path 251 for the SLO optical system and fixation lamp is arranged in the transmission direction of the second dichroic mirror 203 . However, the arrangement of each optical path with respect to the dichroic mirror may be reversed.

An SLO scanning unit 204 , lenses 205 and 206 , a mirror 207 , a third dichroic mirror 208 , an APD (Avalanche Photodiode) 209 , an SLO light source 210 , and a fixation light 211 are included in an optical path 251 for the SLO optical system and the fixation light. is provided. The mirror 207 is configured using a prism in which a perforated mirror or a hollow mirror is vapor-deposited, and separates the illumination light from the SLO light source 210 and the illumination light returned from the eye 200 to be inspected. The third dichroic mirror 208 splits the optical path from the third dichroic mirror 208 into the optical path of the SLO light source 210 and the optical path of the fixation lamp 211 for each wavelength band. In this embodiment, the optical path of the SLO light source 210 is arranged in the reflection direction of the third dichroic mirror 208 and the fixation lamp 211 is arranged in the transmission direction of the third dichroic mirror 208 . However, the arrangement of each optical path with respect to the dichroic mirror may be reversed.

The SLO scanning unit 204 scans the illumination light emitted from the SLO light source 210 on the subject's eye 200, and is composed of an X scanner that scans in the X direction and a Y scanner that scans in the Y direction. The SLO scanning unit 204 is controlled by the imaging control unit 101-2. In this embodiment, the X scanner is a polygon mirror that performs high-speed scanning, and the Y scanner is a galvanomirror. However, the configuration of the SLO scanning unit 204 is not limited to this, and the X scanner and Y scanner may be configured using arbitrary deflection mirrors according to a desired configuration.

The lens 205 is driven along the optical axis by a motor (not shown) for focusing the SLO optical system and the fixation lamp 211 . A motor for driving the lens 205 is controlled by the imaging control unit 101-2.

The SLO light source 210 generates light with a wavelength around 780 nm. The APD 209 detects return light from the eye 200 to be examined. The fixation lamp 211 emits visible light to prompt the subject to fixate.

Illumination light emitted from the SLO light source 210 is reflected by the third dichroic mirror 208, passes through the mirror 207, passes through the lenses 206 and 205, reaches the SLO scanning unit 204, and is illuminated by the SLO scanning unit 204 on the subject's eye 200. Scanned. Return light from the subject's eye 200 returns along the same path as the illumination light, is reflected by the mirror 207 , and is guided to the APD 209 .

The APD 209 generates SLO fundus image signal data based on the returned light, and outputs the signal data to the image processing apparatus 101 . The acquisition unit 101-1 of the image processing apparatus 101 generates an SLO image based on the signal data output from the APD 209. FIG. An SLO image of the fundus of the eye 200 to be examined can be acquired by scanning the fundus of the eye 200 to be examined with illumination light emitted from the SLO light source 210 . On the other hand, it is also possible to obtain an SLO image of the anterior segment of the eye 200 to be inspected by scanning the anterior segment of the eye to be inspected with illumination light emitted from the SLO light source 210 .

Light emitted from the fixation lamp 211 passes through the third dichroic mirror 208 and the mirror 207, passes through the lenses 206 and 205, and is formed into a predetermined shape at an arbitrary position on the subject's eye 200 by the SLO scanning unit 204. FIG. By making the subject gaze at the light emitted from the fixation lamp 211, the subject's fixation can be encouraged.

Lenses 212 and 213, a split prism 214, and a CCD 215 for anterior eye observation that detects infrared light are arranged in an optical path 252 for anterior eye observation. The CCD 215 has sensitivity to the wavelength of illumination light for observing the anterior eye segment (not shown), specifically around 970 nm. A data signal output from the CCD 215 is output to the image processing device 101, and the image processing device 101 can generate an anterior segment observed image based on the input signal data.

The split prism 214 is arranged at a position conjugate with the pupil of the eye 200 to be examined. The image processing apparatus 101 can detect the distance in the Z-axis direction (optical axis direction) of the measurement optical system 100-1 from the eye 200 to be examined using the split image of the anterior segment based on the light that has passed through the split prism 214. FIG.

An OCT optical system is provided in the optical path 250 for the OCT optical system, and the OCT optical system is used to capture a tomographic image of the eye 200 to be examined. More specifically, OCT optics are used to obtain interference signals for generating tomographic images. An XY scanner 216, lenses 217 and 218, and an optical fiber 224 are provided in an optical path 250 for the OCT optical system.

An XY scanner (OCT scanning unit) 216 is for scanning the subject's eye 200 with measurement light. The XY scanner 216 is controlled by the imaging control section 101-2. Although the XY scanner 216 is illustrated as one mirror in FIG. 2, it is actually a galvanomirror that performs scanning in the XY two-axis directions. However, the configuration of the XY scanner 216 is not limited to this, and may be configured using any deflection mirrors according to a desired configuration. For example, the XY scanner 216 may be configured using arbitrary deflection means such as a MEMS mirror capable of scanning the measurement light in two-dimensional directions with a single sheet.

The lens 217 is driven in the optical axis direction by a motor (not shown) in order to focus the measurement light emitted from the optical fiber 224 connected to the optical coupler 219 onto the eye 200 to be examined. By this focusing, the return light of the measurement light from the eye 200 to be inspected is simultaneously focused on the tip of the optical fiber 224 in the form of a spot and is incident thereon. A motor for driving the lens 217 is controlled by the imaging control unit 101-2.

Next, the configuration of the optical path from the OCT light source 220, the reference optical system, and the spectroscope will be described. The OCT optical system further includes an OCT light source 220, a reference mirror 221, a dispersion compensating glass 222, a lens 223, an optical coupler 219, single-mode optical fibers 224 to 227 connected to and integrated with the optical coupler, and a spectroscope 230. It is The OCT optical system constitutes a Michelson interferometer with these configurations.

The OCT light source 220 is an SLD (Super Luminescent Diode), which is a typical low coherent light source. The center wavelength is 855 nm and the wavelength bandwidth is about 100 nm. Here, the bandwidth is an important parameter because it affects the resolution of the obtained tomographic image in the optical axis direction. As for the type of light source, SLD is selected here, but ASE (Amplified Spontaneous Emission) or the like may be used as long as it can emit low coherent light. Near-infrared light is suitable for the center wavelength in view of measuring the eye. Also, since the central wavelength affects the lateral resolution of the obtained tomographic image, the wavelength can be as short as possible. In this embodiment, the center wavelength of the OCT light source 220 is set to 855 nm for both reasons.

Light emitted from the OCT light source 220 passes through the optical fiber 225 and is split into measurement light that enters the optical fiber 224 side via the optical coupler 219 and reference light that enters the optical fiber 226 side. The measurement light irradiates the eye 200 to be observed, which is an observation target, through the optical path 250 for the OCT optical system, and reaches the optical coupler 219 as return light through the same optical path due to reflection and scattering by the eye 200 . On the other hand, the reference light reaches the reference mirror 221 and is reflected through the optical fiber 226, the lens 223, and the dispersion compensating glass 222 inserted to match the wavelength dispersion of the measurement light and the reference light. The reference light reflected by the reference mirror 221 returns along the same optical path and reaches the optical coupler 219 .

The measurement light and the reference light are combined by the optical coupler 219 to become interference light. Here, the measurement light and the reference light interfere with each other when the optical path length of the measurement light and the optical path length of the reference light are substantially the same. The reference mirror 221 is held so as to be adjustable in the optical axis direction by a motor and drive mechanism (not shown) controlled by the imaging control unit 101-2, and the optical path length of the reference light can be matched with the optical path length of the measurement light. be. The interference light is guided to spectroscope 230 via optical fiber 227 .

Polarization adjusters 228 and 229 are provided in the optical fibers 224 and 226, respectively. Polarization adjusters 228 and 229 adjust the polarization of the measurement light and the reference light, respectively. The polarization adjusters 228 and 229 have several looped portions of optical fibers. In the polarization adjustment units 228 and 229, the loop-shaped portions are rotated around the longitudinal direction of the optical fiber to twist the optical fiber, thereby adjusting and matching the polarization states of the measurement light and the reference light. .

The spectroscope 230 is provided with lenses 232 and 234 , a diffraction grating 233 and a line sensor 231 . The interference light emitted from the optical fiber 227 becomes parallel light through the lens 234 , is dispersed by the diffraction grating 233 , and is imaged on the line sensor 231 by the lens 232 .

The interference light is measured by the line sensor 231 as intensity information for each wavelength. The intensity information for each wavelength measured by the line sensor 231 is output to the image processing apparatus 101 as tomographic image signal data (interference signal). The image processing apparatus 101 can generate a tomographic image of the subject's eye 200 using the received signal data.

In this embodiment, a Michelson interferometer is used as an interferometer, but a Mach-Zehnder interferometer may be used. For example, depending on the light amount difference between the measurement light and the reference light, a Mach-Zehnder interferometer can be used when the light amount difference is large, and a Michelson interferometer can be used when the light amount difference is relatively small.

(High image quality processing)
The image quality enhancement processing for motion contrast data acquired by OCT will be specifically described below, but the terms used in the description of the embodiments will be briefly defined. First, three-dimensional volume data information relating to interference signals and brightness tomographic images based thereon is referred to as OCT data, and three-dimensional volume data information relating to motion contrast is referred to as OCTA data. An OCT image is two-dimensional information that can be extracted from three-dimensional volume data on a tomographic image of brightness based on an interference signal, and an OCTA image is two-dimensional information that can be extracted from three-dimensional volume data on motion contrast. In particular, an image generated by projecting or accumulating three-dimensional volume data relating to an interference signal or a luminance tomographic image based thereon within a specified range in the depth direction is referred to as an OCT front image (luminance En-Face). An image generated by projecting or integrating three-dimensional volume data relating to motion contrast is referred to as an OCTA front image. Furthermore, a two-dimensional image including data in the depth direction is referred to as a tomographic image.

(High image quality part)
FIG. 3 is a block diagram showing a more detailed configuration of the image quality enhancement unit 101-47. The image quality improvement unit 101-47 includes an image quality improvement processing unit 301, an area detection unit 302, an ROI setting unit 303, a blend processing unit 304, and a BC adjustment unit 305. FIG.

The image quality enhancement processing unit 301 performs image quality enhancement processing on the medical image acquired by the acquisition unit 101-1 using a trained model for image quality enhancement (image quality enhancement engine, image quality enhancement model) described later. to produce high-quality images. The image quality enhancement processing unit 301 performs image quality enhancement processing on the medical image of the subject (first image) using a trained model obtained by learning using the medical image of the subject as learning data, It can function as an example of an image quality enhancing unit that acquires a high quality image (second image) of the subject. The medical images to be processed for high image quality include OCT images, OCTA images, tomographic images, OCT front images, OCTA front images, SLO images, anterior observation images, and images such as analysis maps obtained by analyzing these images. can be

The region detection unit 302 performs region detection processing on a medical image or a high-quality medical image using a trained model for region detection (region detection engine, region detection model) described later, and detects at least one region. (eg, region of interest). For example, the area detection unit 302 can detect the target area and areas other than the target area. The region detection unit 302 can function as an example of a detection unit that detects a target region in a medical image or a high-quality image of a subject. For example, the region detection unit 302 detects perfused regions and non-perfused regions from the OCTA frontal image using a region detection engine. In diagnosing the fundus, by identifying perfused and non-perfused areas, it is possible to determine whether there is no blood flow where there should be blood vessels, or whether there is some blood flow where there should be no blood vessels (new blood vessels, etc.). can judge. Note that the target region is not limited to the perfusion region and non-perfusion region, and may include any region in which overcorrection occurs by the image quality enhancement engine. For example, the regions of interest may include perfused and non-perfused regions, as well as foveal avascular regions, optic discs, and the like.

Note that the classification of areas may be managed by attribute information (label) indicating each area. For example, the region detection unit 302 may generate a label image having attribute information as pixel values corresponding to each pixel position of the medical image. In addition to the attribute information, a value indicating the likelihood (reliability, probability) of each piece of attribute information output from the region detection engine may be stored in association with each pixel of the medical image.

The ROI setting unit 303 sets a ROI (Region Of Interest) based on the area detected by the area detection unit 302 . Note that the ROI can be at least one of the regions (target regions) detected by the region detection unit 302 . Note that the ROI may be automatically set as a region having predetermined attribute information by the region detection unit 302, in which case the ROI setting unit 303 may be omitted.

The blend processing unit 304 performs blend processing, which is a type of image processing, using a medical image and a high-quality image of the medical image, for example, on a set ROI, which is an area having predetermined attribute information. As the blending process, for example, any known process such as α-blending process may be used. The blend processing unit 304 is applied to a target region in a medical image or a high-quality image so that the difference between the pixel values of the target region and the pixel values of regions other than the target region widens and the pixel values of the target region become lower. It can function as an example of an image processing unit that performs image processing as described above.

The BC adjustment unit 305 performs, for example, BC (Brightness, Contrast) adjustment processing, which is correction processing for at least one of brightness and contrast, on a set ROI, which is an area having predetermined attribute information. Here, the contrast correction processing may include any known contrast correction processing, and may include, for example, correction using a tone curve or gamma curve and level correction. The BC adjustment unit 305 also adjusts the pixel values of the target region in the medical image or the high-quality image such that the difference between the pixel values of the target region and the pixel values of the regions other than the target region widens and the pixel values of the target region become lower. It can function as an example of an image processing unit that performs image processing as described above.

Only one of the blend processing unit 304 and the BC adjustment unit 305 may be provided. Further, when both the blend processing unit 304 and the BC adjustment unit 305 are provided, the image processing apparatus 101 performs either blend processing or BC adjustment processing as image processing to be executed in accordance with an instruction from the operator. It may be configured so that both can be selected. Note that when both the blending process and the BC adjustment process are executed, the image processing apparatus 101 causes the output unit 103 to display the images subjected to the respective image processing by switching or arranging them in accordance with an operation such as a UI. may be configured to

(Medical image quality enhancement engine)
The image quality enhancement engine used by the image quality enhancement processing unit 301 will be described below with reference to FIGS. 4 to 7. FIG. Although the image quality improvement engine is stored in the storage unit 101 - 3 in this embodiment, the image quality improvement engine may be provided in an external device connected to the image processing apparatus 101 . In this case, the image quality enhancement processing unit 301 can perform image quality enhancement processing using an image quality enhancement engine provided in the external device.

The image quality enhancement engine according to this embodiment is a learned model obtained by performing training (learning) related to a machine learning algorithm. In this embodiment, for training a machine learning model related to a machine learning algorithm, input data that is a low-quality image having specific shooting conditions assumed to be processed, and output data that is a high-quality image corresponding to the input data. We use learning data composed of pairs of The specific imaging conditions specifically include a predetermined imaging region, imaging method, imaging angle of view, image size, and the like.

Here, a general trained model will be briefly described. A trained model is a machine learning model that has been trained (learned) in advance using appropriate learning data for any machine learning algorithm. The learning data is composed of one or more pairs of input data and output data (correct data). The format and combination of the input data and the output data of the paired groups that make up the learning data may be one that is an image and the other that is a numerical value, one that is composed of a plurality of image groups and the other that is a character string, or both. may be suitable for a desired configuration, such as an image.

Specifically, for example, learning data (hereinafter referred to as first learning data) configured by a pair group of an image obtained by OCT and an imaging site label corresponding to the image can be used. Note that the imaging region label is a unique numerical value or character string representing the region. Another example of the learning data is a pair group of a low-quality image with a lot of noise obtained by normal OCT imaging and a high-quality image obtained by imaging multiple times by OCT and processing to improve the image quality. learning data (hereinafter referred to as second learning data) and the like.

At this time, when input data is input to the trained model, output data according to the design of the trained model is output. A trained model outputs output data that is likely to correspond to input data, for example, according to a tendency trained using learning data. In addition, the learned model, for example, according to the tendency trained using the learning data, outputs the certainty (reliability, probability) corresponding to the input data as a numerical value for each type of output data. can be done.

Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first learning data, the machine learning model outputs the imaging part label of the imaging part photographed in the image. or output the probability for each imaging site label. Further, for example, when a low-quality image with a lot of noise obtained by normal OCT imaging is input to the machine learning model trained with the second learning data, the machine learning model is imaged multiple times by OCT to improve the image quality. A high-quality image equivalent to the processed image is output. From the viewpoint of quality maintenance, the machine learning model may be configured not to use its own output data as learning data.

Machine learning algorithms also include techniques related to deep learning such as convolutional neural networks (CNNs). In methods related to deep learning, if the parameter settings for the layers and nodes that make up the neural network are different, the degree of reproducibility of the tendencies trained using the learning data in the output data may differ. For example, in a deep learning machine learning model that uses the first learning data, setting more appropriate parameters may increase the probability of outputting a correct imaging region label. Further, for example, in a deep learning machine learning model using the second learning data, if more appropriate parameters are set, it may be possible to output a higher quality image.

Specifically, the parameters in the CNN are, for example, the kernel size of the filter, the number of filters, the value of the stride, the value of the dilation, and the number of nodes output by the fully connected layer, which are set for the convolution layer etc. Note that the parameter group and the number of training epochs can be set to values that are preferable for the mode of use of the trained model based on learning data. For example, based on the learning data, it is possible to set a parameter group and the number of epochs that can output a correct imaging site label with a higher probability or output a higher quality image.

One method for determining such a parameter group and the number of epochs will be exemplified. First, 70% of the pairs constituting learning data are randomly set for training and the remaining 30% for evaluation. Next, the training pair group is used to train the machine learning model, and at the end of each epoch of training, the evaluation pair group is used to calculate a training evaluation value. The training evaluation value is, for example, the average value of a group of values obtained by evaluating the output when inputting each pair of input data into the machine learning model under training and the output data corresponding to the input data using a loss function. be. Finally, the parameter group and epoch number when the training evaluation value is the smallest are determined as the parameter group and epoch number of the machine learning model. It should be noted that the machine learning model may overfit the pair group for training by dividing the pair group that constitutes the learning data into the pair group for training and the pair group for evaluation in this way and determining the number of epochs. can be prevented.

Here, the image quality enhancement engine according to the present embodiment is configured as a module for outputting a high quality image obtained by enhancing the image quality of an input low quality image. Here, the term “improvement in image quality” as used herein refers to conversion of an input image into an image having a quality more suitable for image diagnosis. image. In addition, a low-quality image is, for example, a two-dimensional image or a three-dimensional image obtained by X-ray imaging, CT, MRI, OCT, PET, or SPECT, or a three-dimensional moving image of continuous CT. It was taken without any settings for high image quality. Specifically, low-quality images include, for example, low-dose imaging by X-ray imaging equipment or CT, imaging by MRI that does not use a contrast agent, images acquired by short-time imaging of OCT, etc., and images with a small amount of imaging. Including OCTA images etc. acquired at the number of times.

In addition, if high-quality images with little noise or high contrast are used for image analysis such as blood vessel analysis processing of images such as OCTA, and region segmentation processing of images such as CT and OCT, low-quality images can be used. In many cases, analysis can be performed with higher accuracy than using it. Therefore, the high-quality image output by the image quality enhancement engine may be useful not only for image diagnosis but also for image analysis.

Further, the content of image quality suitable for image diagnosis depends on what is desired to be diagnosed in various image diagnoses. For this reason, it cannot be said unconditionally, but for example, the image quality suitable for image diagnosis has little noise, high contrast, shows the object to be photographed in colors and gradations that are easy to observe, has a large image size, etc. Includes image quality, which may be high resolution. In addition, it is possible to include an image quality in which non-existent objects and gradations that have been drawn in the process of image generation are removed from the image.

In the image processing method constituting the image quality improvement method by the image quality improvement processing unit 301 in this embodiment, processing using various machine learning algorithms such as deep learning is performed. In addition to processing using machine learning algorithms, this image processing method includes various image filter processing, matching processing using a database of high-quality images corresponding to similar images, and existing arbitrary image processing such as knowledge-based image processing. may be processed.

A configuration example of the CNN related to the image quality enhancement engine according to the present embodiment will be described below with reference to FIG. FIG. 4 shows an example of the configuration of the image quality enhancement engine. The configuration shown in FIG. 4 is composed of a plurality of layer groups that are responsible for processing and outputting an input value group. As shown in FIG. 4, types of layers included in the configuration include a convolution layer, a downsampling layer, an upsampling layer, and a merger layer.

The convolution layer is a layer that performs convolution processing on an input value group according to set parameters such as the kernel size of filters, the number of filters, the stride value, and the dilation value. Note that the number of dimensions of the kernel size of the filter may also be changed according to the number of dimensions of the input image. A down-sampling layer is a process that reduces the number of output value groups to the number of input value groups by thinning out or synthesizing input value groups. Specifically, for example, there is a Max Pooling process. The upsampling layer is a process that makes the number of output value groups larger than the number of input value groups by duplicating the input value groups or adding values interpolated from the input value groups. Specifically, for example, there is linear interpolation processing. The synthesizing layer is a layer that performs a process of synthesizing a group of values such as a group of output values of a certain layer and a group of pixel values forming an image from a plurality of sources and connecting or adding them.

In such a configuration, the value group output from the pixel value group forming the input image Im410 through the convolution processing block and the pixel value group forming the input image Im410 are synthesized in the synthesis layer. The combined pixel values are then shaped into a high quality image Im420 in a final convolutional layer. Although not shown in the figure, as an example of changing the configuration of the CNN, for example, a batch normalization layer after the convolution layer or an activation layer using a normalized linear function (Rectifier Linear Unit) is incorporated. You may

Here, the GPU can perform efficient operations by processing more data in parallel. Therefore, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing using a GPU. Therefore, in this embodiment, the GPU is used in addition to the CPU for processing by the image processing unit 101-4, which is an example of the learning unit. Specifically, when a learning program including a learning model is executed, the CPU and the GPU cooperate to perform calculations for learning. In addition, in the processing of the learning unit, the calculation may be performed only by the CPU or the GPU. Also, the image quality enhancement processing unit 301 may be implemented using a GPU, like the learning unit.

Also, the learning unit may include an error detection unit and an updating unit (not shown). The error detection unit obtains an error between correct data and output data output from the output layer of the neural network according to input data input to the input layer. The error detector may use a loss function to calculate the error between the output data from the neural network and the correct data. Also, the updating unit updates the weighting coefficients for coupling between nodes of the neural network based on the error obtained by the error detecting unit so as to reduce the error. This updating unit updates the connection weighting coefficients and the like using, for example, the error backpropagation method. The error backpropagation method is a method of adjusting the connection weighting coefficients and the like between nodes of each neural network so as to reduce the above error.

Note that when using some image processing methods such as image processing using CNN, it is necessary to pay attention to the image size. Specifically, it should be noted that different image sizes may be required for the input low-quality image and the output high-quality image in order to deal with the problem that the peripheral part of the high-quality image is not sufficiently high-quality. should.

Although not explicitly stated in this embodiment for the sake of clarity, if an image quality enhancement engine that requires different image sizes for an image input to the image quality enhancement engine and an image output from the image quality enhancement engine is adopted, the image size may be changed as appropriate. is adjusted. Specifically, for input images such as images used as learning data for training machine learning models and images input to the image quality improvement engine, padding is performed, and the shooting area around the input image is adjusted. Combine and adjust the image size. Note that the area to be padded is filled with a constant pixel value, filled with neighboring pixel values, or mirror-padding in accordance with the characteristics of the image quality improvement method so as to effectively improve the image quality.

Further, the image quality enhancement method by the image quality enhancement processing unit 301 may be implemented using only one image processing method, or may be implemented by combining two or more image processing methods. Alternatively, a plurality of image quality improvement technique groups may be performed in parallel to generate a plurality of high quality image groups, and then the highest quality image may be finally selected as the high quality image. The selection of the high-quality image with the highest image quality may be performed automatically using the image quality evaluation index, or a plurality of high-quality image groups may be displayed on a UI provided in the output unit 103 or the like, It may be performed according to an instruction from an examiner (operator).

In some cases, an input image whose image quality has not been enhanced is more suitable for image diagnosis, so the input image may be added to the final selection of images. Also, a parameter may be input to the high image quality engine together with the low image quality image. For example, a parameter designating the degree of image quality improvement or a parameter designating the image filter size used in the image processing technique may be input together with the input image to the image quality improvement engine.

Here, the input data for the learning data of the image quality enhancement engine according to the present embodiment is a low image quality image acquired with the same model as the image capturing apparatus 100 and the same settings as the image capturing apparatus 100 . In addition, the output data of the learning data of the image quality improvement engine is a high quality image obtained by image processing and settings related to shooting conditions that require more man-hours than those of the same model. Specifically, the output data is, for example, a high-quality image (overlay image) obtained by performing superimposition processing such as averaging on a group of images (original images) obtained by photographing multiple times. can do.

Here, motion contrast data of OCTA will be used as an example to describe high-quality images and low-quality images. Motion contrast data is data used in OCTA or the like, which is obtained by repeatedly photographing the same part of an object to be photographed and detecting a temporal change in the object to be photographed between shots. Further, as described above, out of the calculated motion contrast data (an example of three-dimensional medical image data), by generating a front image using data in a desired range in the depth direction of the imaging target, the OCTA can be obtained. An En-Face image (OCTA en face image) can be generated. In the following description, the number of times the OCT data is repeatedly captured at substantially the same position (substantially the same location) is referred to as NOR (Number Of Repeat).

Regarding learning data according to the present embodiment, two different methods will be described with reference to FIGS. Note that the method of generating the high-quality image and the low-quality image is not limited to these, and any known generating method may be used.

A first method relating to an example of generating a high-quality image and a low-quality image will be described with reference to FIG. 5(a). The first method uses, as an example of a high-quality image, a motion contrast image generated from OCT data obtained by repeatedly capturing approximately the same position of an object to be captured. In FIG. 5A, a motion contrast image Im510 indicates a three-dimensional motion contrast image (three-dimensional motion contrast data). A motion contrast image Im511 indicates a two-dimensional motion contrast image (two-dimensional motion contrast data) that forms a three-dimensional motion contrast image.

Tomographic images Im501-1 to Im501-3 represent OCT tomographic images (B-scan images) for generating the motion contrast image Im511. Here, NOR corresponds to the number of OCT tomographic images in tomographic images Im501-1 to Im501-3 in FIG. 5(a), and NOR is 3 in the example shown. The tomographic images Im501-1 to Im501-3 are captured at predetermined time intervals (Δt). Note that the substantially same position indicates one line in the front direction (XY) of the subject's eye, and corresponds to the position of the motion contrast image Im511 in FIG. 5(a). Note that the front direction is an example of a direction intersecting the depth direction.

Since motion contrast data is data obtained by detecting temporal changes, NOR must be performed at least twice to generate this data. For example, when NOR is 2, one motion contrast data is generated. When NOR is 3, two pieces of motion contrast data are generated when motion contrast data is generated using only OCT data of adjacent time intervals (first and second, second and third). If the motion contrast data is also generated using separate time intervals (first and third) of OCT data, a total of three motion contrast data are generated. That is, when NOR is increased to 3 times, 4 times, . . . , the number of motion contrast data at substantially the same position also increases. A high-quality motion contrast image can be generated by aligning a plurality of motion contrast images obtained by repeatedly photographing substantially the same position and performing superimposition processing such as averaging. Therefore, in order to generate a motion contrast image of high quality, NOR can be performed at least 3 times or more, and in order to obtain a motion contrast image of higher quality, NOR can be performed 5 times or more, for example.

On the other hand, as an example of a corresponding low image quality image, a motion contrast image before performing superimposition processing such as averaging can be used. In this case, the low-quality image can be used as a reference image for superimposition processing such as averaging for generating a high-quality image. If the position and shape of the target image are deformed with respect to the reference image when performing the superimposition processing, there is almost no spatial positional deviation between the reference image and the image after the superimposition processing. . Therefore, a low quality image and a high quality image can be easily paired. It should be noted that the low-quality image may be a target image that has been subjected to image deformation processing for alignment instead of the reference image.

A plurality of pair groups can be generated by using each of the original image groups (reference image and target image) as input data and the corresponding superimposed image as output data. For example, when obtaining one superimposed image from a group of 15 original images, a pair of the first original image and the superimposed image in the group of original images, the second original image in the group of original images, and A pair of superimposed images can be generated. Thus, when one superimposed image is obtained from a group of 15 original images, 15 pair groups of one image in the group of original images and the superimposed image can be generated. Three-dimensional high-quality data can be generated by repeatedly photographing substantially the same position in the main scanning (X) direction and scanning while shifting it in the sub-scanning (Y) direction.

Next, a second method related to an example of generating a high-quality image and a low-quality image will be described with reference to FIG. 5(b). In the second method, a high-quality image is generated by superimposing motion contrast images obtained by photographing substantially the same region of an object to be photographed a plurality of times. Note that the substantially identical region indicates a region such as 3 × 3 mm or 10 × 10 mm in the front direction (XY) of the eye to be examined. A three-dimensional motion contrast image (three-dimensional motion contrast data) can be acquired including the depth direction of the image. When the same area is photographed a plurality of times and superimposition processing is performed, the NOR can be performed twice or three times in order to shorten the photographing per time.

Also, in order to generate high-quality 3D motion contrast data, at least two pieces of 3D data of the same region are acquired. FIG. 5(b) shows an example of a plurality of 3D motion contrast images. The motion contrast images Im520 to Im540 are three-dimensional motion contrast images, like the motion contrast image Im510 described with reference to FIG. 5(a). Using these two or more three-dimensional motion contrast images, alignment processing in the front direction (XY) and depth direction (Z) is performed, and after removing artifact data in each data, averaging processing I do. Thereby, one high-quality three-dimensional motion contrast image from which artifacts are removed can be generated.

On the other hand, the corresponding low-quality image can be used as reference data for superimposition processing such as averaging. As described in the first method, since there is almost no spatial positional deviation between the reference image and the image after averaging, it is possible to easily pair the low-quality image and the high-quality image. An arbitrary three-dimensional motion contrast image generated from target data subjected to image deformation processing for alignment may be used as the low image quality image instead of the reference data.

In the first method, since the imaging itself is completed in one time, the burden on the subject is small. However, as the number of times of NOR is increased, the time taken for one shot becomes longer. Also, if artifacts such as cloudy eyes or eyelashes appear during imaging, it is not always possible to obtain a good image. In the second method, imaging is performed a plurality of times, which slightly increases the burden on the subject. However, one shooting time is short, and even if artifacts appear in one shooting, if no artifacts appear in another shooting, a clear image with few artifacts can finally be obtained. In view of these characteristics, any method can be selected according to the condition of the subject when collecting data.

In the present embodiment, a motion contrast image is used as an example of a low image quality image used as learning data and a high image quality image, but the image used as learning data is not limited to this. Since the OCT data is acquired to generate the motion contrast data, it is possible to similarly generate the low quality image and the high quality image using the OCT data. Furthermore, although description of the tracking process is omitted in this embodiment, since substantially the same position or substantially the same area of the eye to be examined is photographed, the photographing can be performed while tracking the eye to be examined. Tracking processing may be performed by any known method.

If a pair of three-dimensional high image quality data and low image quality data can be acquired, an arbitrary pair of two-dimensional images can be generated from these. For example, a high-quality OCTA planar image can be generated by projecting or integrating the generated high-quality three-dimensional motion contrast image in a desired depth range to generate an arbitrary OCTA frontal image. Also, the corresponding low-quality image can be an arbitrary OCTA front image generated from reference data when performing superimposition processing such as averaging. In this case also, since there is almost no spatial positional deviation between the reference image and the image after the averaging, it is possible to easily pair the low-quality image and the high-quality image. An arbitrary motion contrast front image generated from target data subjected to image transformation processing for alignment may be used as the low image quality image instead of the reference data.

An example of a pair of two-dimensional images used as such learning data will be described in more detail with reference to FIGS. 6(a) and 6(b). For example, when the image used for learning data is an OCTA frontal image, as described above, the OCTA frontal image can be generated by projecting or integrating the three-dimensional volume data related to motion contrast in a desired depth range. can. Here, the depth range is the range in the Z direction shown in FIGS. 5(a) and 5(b).

FIG. 6A shows an example of an OCTA front image. As the OCTA front images used for learning data, OCTA front images generated in different depth ranges such as superficial layer (image Im610), deep layer (image Im620), outer layer (image Im630), and choroidal vascular network (image Im640) should be used. can be done. Note that the types of OCTA front images are not limited to this, and the number of types may be increased by generating OCTA front images in which different depth ranges are set by changing reference layers and offset values. When learning, each OCTA frontal image of different depths may be learned separately, or a plurality of images of different depth ranges may be combined (for example, divided into the surface layer side and the deep layer side) for learning. Alternatively, OCTA enface images of all depth ranges may be learned together. When using luminance En-Face images generated from OCT data as learning data, a plurality of En-Face images generated from an arbitrary depth range can be used as in the case of OCTA frontal images.

For example, consider a case where the image quality enhancement engine includes a trained model obtained using training data including a plurality of OCTA frontal images corresponding to different depth ranges of the subject's eye. At this time, the acquisition unit 101-1 can acquire, as the first image, an OCTA front image corresponding to a partial depth range of the long depth range including different depth ranges. That is, an OCTA frontal image corresponding to a depth range different from the plurality of depth ranges corresponding to the plurality of OCTA frontal images included in the learning data can be used as an input image for image quality enhancement processing. Of course, the OCTA front image in the same depth range as that used during learning may be used as the input image during image quality enhancement processing. Also, a part of the depth range may be set according to the operator pressing an arbitrary button on the UI, or may be set automatically. Note that the above-described content is not limited to the OCTA front image, and can also be applied to, for example, a luminance En-Face image.

Note that when the image to be processed by the trained model is a tomographic image, learning is performed using an OCT tomographic image, which is a B-scan image, or a tomographic image of motion contrast data as learning data. This will be described with reference to FIG. 6(b). In FIG. 6B, images Im651 to Im653 are OCT tomographic images (luminance tomographic images). The reason why the images are different in FIG. 6B is that the tomographic images are shown at different positions in the sub-scanning (Y) direction. In the tomographic image, the learning may be performed together without worrying about the difference in position in the sub-scanning direction. However, in the case of images taken at different locations (for example, the center of the macula or the center of the optic papilla), each region may be learned separately, or they may be learned together regardless of the location. You may make it learn to. Since the OCT tomographic image and the tomographic image of the motion contrast data differ greatly in image feature amount, it is better to perform learning separately.

As a high-quality image used as output data for learning data, for example, a superimposed image can be used as described above. A superimposed image that has undergone the superimposing process has a high-quality image that is suitable for image diagnosis because the pixels commonly drawn in the original image group are emphasized. In this case, the generated high-quality image is a high-contrast image in which the difference between the low-brightness region and the high-brightness region is clear as a result of the commonly drawn pixels being emphasized. Also, for example, in the superimposed image, random noise that occurs each time an image is captured can be reduced, and an area that was not well rendered in the original image at a certain point in time can be interpolated with another original image group.

Furthermore, when a superimposed image is used as output data for learning data, a low image quality image to be used as input data for learning data can be generated from the superimposed image. In this case, for example, the superimposed image is once down-sampled to a low resolution, and the low-resolution image is up-sampled by a known method (nearest neighbor method, bilinear method, etc.) and used as training data. Can be input data. It is also possible to configure an image quality enhancement engine that improves the sense of resolution by performing learning using such a pair group of images as learning data.

Further, when it is necessary to configure the input data of the machine learning model with a plurality of images, a necessary number of original image groups can be selected from the original image group and used as input data. For example, when obtaining one superimposed image from a group of 15 original images, if two images are required as input data for a machine learning model, 105 (15C2=105) pair groups can be generated. be.

Note that pairs that do not contribute to high image quality can be removed from the learning data in the pair group that constitutes the learning data. For example, if a high-quality image, which is output data that constitutes a pair of learning data, has an image quality that is not suitable for image diagnosis, the image output by the image quality enhancement engine trained using the learning data is also not suitable for image diagnosis. image quality may be poor. Therefore, by removing pairs whose output data has an image quality unsuitable for image diagnosis from the learning data, it is possible to reduce the possibility that the image quality enhancement engine generates an image with an image quality unsuitable for image diagnosis.

In addition, when the average brightness and brightness distribution of the pair of images are significantly different, the image quality improvement engine trained using the training data will not be able to detect images that are not suitable for image diagnosis with brightness distributions that are significantly different from those of low-quality images. may be output. For this reason, pairs of input data and output data having significantly different average luminances and luminance distributions can be removed from the learning data.

Furthermore, if the structures and positions of the shooting targets drawn in the pair of images are significantly different, the high-quality image engine trained using the learning data will draw the shooting targets in structures and positions that are significantly different from those of the low-quality images. There is a possibility of outputting an image that is not suitable for diagnostic imaging. For this reason, pairs of input data and output data in which the structures and positions of the objects to be drawn are significantly different can be removed from the learning data. In addition, from the viewpoint of maintaining quality, the image quality improvement engine can be configured not to use the high quality image it outputs as learning data.

By using the image quality improvement engine that has learned in this way, the image quality improvement processing unit 301 performs superimposition processing to increase contrast and reduce noise when a medical image acquired in a single imaging process is input. It is possible to output a high-quality image that seems to have undergone reduction or the like. Therefore, the image quality enhancement processing unit 301 can generate a high quality image suitable for image diagnosis based on the low quality image that is the input image.

Although an example in which a superimposed image is used as output data of learning data has been described here, output data of learning data of the image quality enhancement engine is not limited to this. The output data of the learning data may be a high-quality image corresponding to the input data. It may be an image or the like shot under many shooting conditions. Also, an image obtained by subjecting a low-quality image used as input data to image processing using statistical processing such as maximum a posteriori probability estimation (MAP estimation) processing can be used as output data for learning data. Note that any known method may be used as a method for generating a high-quality image.

Also, as the image quality improvement engine, a plurality of image quality improvement engines may be prepared that individually perform various image quality improvement processes such as noise reduction, contrast correction, and resolution enhancement. Also, one image quality enhancement engine that performs at least two image quality enhancement processes may be prepared. In these cases, a high-quality image corresponding to desired processing may be used as the output data of the learning data. For example, with respect to an image quality enhancement engine that performs individual processing, a high image quality image that has been subjected to individual processing such as noise reduction processing may be used as output data for learning data. As for the image quality enhancement engine that performs a plurality of image quality enhancement processes, for example, a high quality image subjected to noise reduction processing, contrast correction processing, etc. may be used as output data of learning data.

(Area detection engine for medical images)
Next, a medical image area detection engine used by the area detection unit 302 will be described with a contrast front image as an example. The region detection unit 302 according to the present embodiment classifies the contrast front image into perfusion regions and non-perfusion regions and detects regions in order to confirm the presence or absence of blood flow, for example. Hereinafter, the non-perfusion area is referred to as NPA (Non Perfusion Area). Further, the region detection unit 302 according to the present embodiment may detect, for example, a foveal avascular zone (FAZ) or an optic nerve head (ONH) as a region. For the detected region information, the label of the region can be associated with the pixels in the image. Deep learning that associates labels with all pixels in an image in this way is called semantic segmentation.

Generally, in deep learning, the work of creating output data (correct data) of learning data is called annotation. With regard to learning data related to a trained model for area detection, it is possible to create output data by giving a label indicating an area to be classified to each pixel position of the input data or the corresponding image by annotation. . An image having, as a pixel value, information indicating the label of each area generated in this way is called an area label image. The annotation work may be performed entirely manually or partially automatically. The efficiency of annotation work can be improved by performing deep learning as needed after a certain amount of work is completed, and correcting the labels while referring to the segmentation results of the region detection engine in the middle stage. It should be noted that other information may be used in addition to the input data when creating an area label image. For example, when creating an area label image to be output data of learning data whose input data is a high-quality image, an image before high-quality enhancement is used, for example, to create an area label image by referring to the image. good too.

In this way, a trained model that has been machine-learned using learning data that is a pair group of a predetermined number of input data, which is a medical image, and output data, which is a region label image corresponding to the medical image, is used as a region detection engine. be able to. Note that the area detection engine that has learned in this way can output the certainty (reliability, probability) of the area label for each pixel. Therefore, the region detection unit 302 can output a label with a higher probability than other labels as the region detection result, for example, among the labels output from the region detection engine. In addition, it is possible to output a label with a probability higher than the threshold among the labels as a detection result. At this time, if there are multiple labels with probabilities higher than the threshold, all of them may be output, or a label with a higher probability than the other labels may be output as the detection result. Furthermore, the region detection unit 302 may determine the detection result using a machine learning model from the probability of each label obtained using the trained model. The machine learning algorithm used in this case may be a different type of machine learning algorithm than the one used to obtain the label probabilities, such as neural networks, support vector machines, Adaboost, Bayesian networks, or It may be a random forest or the like.

Note that a common model may be used as the image quality enhancement engine and the area detection engine. In this case, for example, a known U-Net model can be used, and by using the same image quality enhancement engine and area detection engine, each inference process (estimation process) can be performed simply by replacing parameters. can.

In addition, known U-Net models that can be used for image quality enhancement engines and region detection engines perform training with a predetermined image data size, so the image data input to the trained model is the same size as during training. It is necessary to make an inference (estimation). If the input image is larger than the image size input to the trained model, the image quality enhancement processing unit 301 divides the input image into a plurality of subset regions (FIG. 17) and performs inference processing on each region. to synthesize. Also, when a part of the inference area is set as the valid area, the non-valid area may be set as the margin area, and the subset area may be set so as to overlap the margin area. When setting the margin area, the area of the input image may be expanded in advance by the margin (not shown). General processing such as mirroring can be used as the enlargement method.

On the other hand, depending on the type of network, the subset size can be changed from the training size. In the case of a network that includes a downsampling layer or an upsampling layer, although there are certain constraints, if the size of the subset to be inferred under those constraints is increased, the number of inference processes can be reduced, thereby increasing the speed. becomes possible. At this time, the image quality improvement processing unit 301 divides the input image into subset regions with a size different from that at the time of learning, and executes inference processing. Conversely, by setting the size of the image data at the time of learning to be small, there is an effect that the number of learning data can be increased. Also, consider a case where the trained model is obtained by learning with learning data including input data that is a medical image having a first image size. At this time, during inference, the first image, which is a medical image having a second image size larger than the first image size, is input to the trained model as input data. Thereby, the image processing section can output the second image having the second image size as the output data of the trained model. A medical image having an image size suitable for learning and inference (estimation) can be used as input data for the trained model. Specifically, since the image size of the input data of the learning data is smaller than the image size of the input data at the time of inference, the number of learning data can be increased. On the other hand, since the image size of the input data at the time of inference is larger than the image size of the input data of the learning data, the inference processing can be speeded up. Of course, a large discrepancy in the size of the subset area between training and inference may lead to undesirable results, but it is necessary to check the difference in the inference result due to the size change of the subset area in advance to see if it is within an acceptable range. You should check. For example, when inferring an input image of 232×232 pixels for a model with an input size of 224×224 pixels and setting margin areas of 12 pixels each on the top, bottom, left, and right, setting four subset areas will make the effective area can be inferred by On the other hand, if the input size of the learning model is set to 256×256 and expanded only at the time of inference, only one subset area is required and the inference process is completed in one time. In general, an architecture for streamlining inference processing called batch processing can be used, so inference processing may be streamlined by setting the number of subset regions according to the number of batch processing. At this time, the input data of the learned model at the time of inference is a plurality of images each having a second image size obtained by dividing a medical image having a third image size larger than the second image size into a plurality of images. It may be a medical image. In this case, a plurality of images are output as the output data of the trained model. Then, an image of the third image size may be generated by synthesizing the plurality of output images.

Furthermore, as input data for region detection, the high-quality image (superimposed image) described above may be used, or a high-quality image obtained by applying an image quality enhancement engine to an input medical image may be used as input data. may be used. In this way, by performing learning according to input data, it is possible to configure an optimal area detection engine for each input data.

Although the known UNet model is shown as an example of the region detection engine, region detection may be performed by DETR (Detection Transformer) using Encoder-Decoder type Transformer. FIG. 18 shows an example of Self-Attention used in Transformer. As shown in FIG. 18A, Query, Key, and Value are created from the output of Convolution of the input image using One by One Convolution or the like. As shown in FIG. 18B, focusing on the upper left pixel, the inner product of Query and Key is obtained and Softmax is taken to determine which pixel should characterize the upper left pixel. Then, the values are added while weighting all the pixels according to the result of this Softmax. This series of processing is performed on all pixels, and the result is added to the input and output. In other words, Self-Attention is the mechanism of giving attention to oneself and reflecting the result on oneself. It is possible to realize a mechanism that characterizes By performing this pixel-by-pixel self-attention, for example, individual FAZ/NPA/vessel pixels can be well characterized even if they are located far apart in the image, or if their positions vary too much. be possible. In conventional CNN, in order to capture the features of pixels at distant positions, it is necessary to make the hierarchy deeper, increase the filter size, or use dilated convolution to widen the reference range. There was a problem that the cost increased. By using Self-Attention in this way, even in object recognition and area detection in image processing, calculation costs can be suppressed by constructing a model using a Transformer. Since the Transformer learns while giving unique values to the positions of pixels or pixel regions, it also learns positional dependencies as a result. Dynamically changing the connection of the neural network according to the input data by dynamically changing where to focus attention is a very versatile method. It may be applied to estimation of the non-perfusion region from the map, and the image quality improvement processing described above may be applied. Alternatively, region detection may be performed by combining DETR and the Unet model. That is, semantic segmentation by the Unet model may be performed after narrowing down the non-perfused region by DETR. Conversely, after the semantic segmentation processing of the Unet model, detailed labeling may be further performed by DETR.

Note that the region detection engine does not necessarily have to be an artificial intelligence engine based on machine learning, and may be configured using a rule-based algorithm based on the structure of the subject's eye, including conventional various filtering processes. . Therefore, the area detection engine used by the area detection unit 302 may, for example, extract the FAZ or NPA area by combining filter processing such as a known Gaussian filter, or according to an operator's instruction via a GUI or the like Regions may be extracted. Further, the region detection engine may classify the motion contrast image into a perfused region and a non-perfused region and detect the region, for example, by threshold processing using the structure of the eye to be inspected and a predetermined threshold value.

(Image processing procedure for medical images)
Next, a series of image processing procedures by the image processing apparatus 101 according to this embodiment will be described with reference to FIG. FIG. 7 is a flowchart showing a schematic flow of image processing according to this embodiment.

When the image processing according to this embodiment is started, the acquisition unit 101-1 acquires a medical image in step S701. The acquisition unit 101-1 may acquire a medical image from the imaging device 100 or an external device, or may acquire a medical image generated using signal data acquired from these devices. In this embodiment, the acquisition unit 101-1 acquires, for example, a contrast front image as a medical image. The medical image is not limited to this, and may be a tomographic image, a brightness En-Face image, an SLO image, an analysis image (analysis map) obtained by analyzing a medical image of a subject, or the like.

In step S702, the image quality enhancement processing unit 301 applies image quality enhancement processing, which is the first image processing, to the acquired medical image using at least one image quality enhancement engine described above. To obtain a medical image that has been improved in quality. The image quality enhancement processing unit 301 according to the present embodiment uses an image quality enhancement engine to obtain an OCTA frontal image whose image quality has been enhanced from the OCTA frontal image.

Next, in step S703, the region detection unit 302 performs region detection processing on the medical image acquired in step S701 or the high-quality medical image acquired in step S702 using the aforementioned region detection engine. Apply. The region detection unit 302 detects at least two regions in the medical image or the high-quality medical image by the region detection processing, and acquires information indicating each region. For example, the region detection unit 302 may acquire a region label image in which the label indicating each region of the detected region is information of each pixel, or a label indicating each region in the information of each pixel of the medical image. may be added. Note that the area detection unit 302 only needs to output information indicating the detected area, and the information indicating the detected area is not limited to the format described above. For example, the region detection unit 302 may be label information associated with pixel information of a medical image. Further, as described above, the value indicating the likelihood (reliability, probability) of each attribute output from the region detection engine may be associated with each pixel of the medical image and added to the pixel information. In this embodiment, a region detection engine is used to detect perfused and non-perfused regions (NPA) in an OCTA enface image or an enhanced OCTA enface image. Note that the pixel positions of the detected regions in the medical image or the high-quality medical image can correspond to each other in the medical image and the high-quality medical image.

When the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets the ROI in the high-quality medical image based on the output information. Note that the ROI may be set for at least one of the areas detected by the area detection unit 302 . In this embodiment, the ROI setting unit 303 sets the ROI for the NPA in the OCTA frontal image with improved image quality. Note that the ROI may be set by the region detection unit 302 as described above.

Finally, in step S704, the blend processing unit 304 or the BC adjustment unit 305 adjusts the ROI in the high-quality medical image so that the difference between the pixel value of the ROI and the pixel value of the area other than the ROI widens. Apply a second image processing. At this time, the blend processing unit 304 or the BC adjustment unit 305 adjusts the ROI in the high-quality medical image so that the pixel value of the ROI is lower than the pixel value of the ROI before the second image processing. 2 image processing is applied. For example, the blend processing unit 304 performs the second image processing on the ROI in the high-quality medical image, the medical image acquired in step S701 and the high-quality medical image acquired in step S702. Blend processing is performed. The blending process may be performed using any known method, and the blending ratio may be a predetermined ratio, or may be set according to an operator's instruction. By performing the blending process, the image before overcorrection and the image with overcorrection are blended for the area where overcorrection has occurred due to the image quality improvement process using the trained model, and the overcorrection is performed. The difference between the pixel values of the area where the . In other words, by performing the blending process, the pixel values of the overcorrected area are lowered. In such processing, pixel values can be reduced only in areas where overcorrection has occurred without reducing the pixel values of the entire image. Therefore, for example, it is possible to suppress overcorrection in areas such as NPA and FAZ.

In addition, the BC adjustment unit 305 performs BC adjustment processing as second image processing on the ROI in the high-quality medical image. The BC adjustment process may be performed using any known technique, for example, the brightness in the ROI may be set to a predetermined value or increased/decreased by a predetermined value. Also, in the ROI, the contrast may be adjusted using, for example, a tone curve, a gamma curve, or the like. Furthermore, the BC adjustment process may set at least one of the brightness and contrast in the ROI to a value according to the operator's instruction, or may increase or decrease the value by that value. For example, the BC adjustment unit 305 may determine a brightness correction value or set values such as a tone curve used for contrast adjustment according to an operator's instruction. By performing the BC adjustment process, the pixel values of the over-corrected area and the pixel values of the other areas are corrected for the area where the over-correction is caused by the image quality improvement process using the learned model. At least one of brightness and contrast is adjusted to widen the difference. In other words, by performing the BC adjustment process, the pixel value of the area where overcorrection occurs is lowered. In such processing, pixel values can be reduced only in areas where overcorrection has occurred without reducing pixel values for the entire image. Therefore, for example, it is possible to suppress overcorrection in areas such as NPA and FAZ.

In this embodiment, the blend processing unit 304 performs blend processing of the OCTA frontal image and the high-quality OCTA frontal image on the NPA set as the ROI in the high-quality OCTA frontal image. Further, the BC adjustment unit 305 performs BC adjustment processing of the OCTA frontal image with improved image quality for the NPA set as the ROI in the OCTA frontal image with improved image quality. As described above, either one of the processing by the blend processing unit 304 and the processing by the BC adjustment unit 305 may be performed, or both may be performed. The high-quality image subjected to the second image processing by the blend processing unit 304 and the BC adjustment unit 305 is displayed on the output unit 103 by the display control unit 101-5, or is output to an external device or the like by the output unit 103. , the storage unit 101-3 or the external storage device 102. FIG.

By performing image processing according to the present embodiment in this way, an area in which overcorrection occurs due to image processing of a medical image using a trained model is detected, and an image in which the overcorrection is reduced for the detected area is obtained. can be treated. For example, it is possible to detect the NPA in the OCTA frontal image and perform blending processing and BC adjustment processing to reduce overcorrection for the NPA in the high-quality OCTA frontal image acquired using the trained model.

As mentioned above, NPA is of particular importance in diagnosing fundus images. By identifying NPA, it is possible to determine whether there is no blood flow where there should be blood vessels, or whether there is some blood flow where there should be no blood vessels (such as neovascularization). In particular, regarding FAZ and NPA, by leaving the original state before image quality enhancement to some extent, it is possible to assist the operator (examiner)'s judgment regarding discrimination between noise and blood vessels in image diagnosis.

On the other hand, by setting the brightness and contrast to a desired state and actively removing noise and the like from the NPA, it is possible to make it more NPA-like and improve the visibility. In this case as well, in image diagnosis, it is possible to assist the operator's judgment regarding the distinction between noise and blood vessels.

In addition, since diagnosis is generally performed in combination with other examinations, the second image processing method may be set in advance or may be selected as appropriate. Also, the second image processing method may be changed depending on the area to be classified. Furthermore, it is also possible to change the setting depending on the eye to be examined. Also, the depth of the OCTA front image, that is, the setting may be changed between the shallow layer and the deep layer. These settings may be stored for each user or subject.

As described above, the image processing apparatus 101 according to this embodiment includes the image quality enhancement processing unit 301 , the area detection unit 302 , the blend processing unit 304 and the BC adjustment unit 305 . The image quality enhancement processing unit 301 uses an image quality enhancement model (image quality enhancement engine) obtained by learning using the medical image of the subject as learning data to perform high quality processing on the first image of the medical image of the subject. An image quality improvement process is performed to obtain a second high-quality medical image of the subject. The region detection unit 302 detects a target region in the medical image (first image or second image) of the subject. At least one of the blend processing unit 304 and the BC adjustment unit 305 adjusts the target region in the second image such that the difference between the pixel values of the target region and the pixel values of the regions other than the target region increases. is lower than the pixel value of the target area before image processing.

As image processing, the blend processing unit 304 combines the first image and the second image so that the difference between the pixel values of the target region and the pixel values of the regions other than the target region widens and the pixel values of the target region become lower. Blend processing is performed to blend with the image of In addition, as image processing, the BC adjustment unit 305 adjusts the brightness and contrast so that the difference between the pixel values of the target region and the pixel values of the regions other than the target region widens and the pixel values of the target region become lower. BC adjustment processing is performed to correct at least one of them. The medical image of the subject can be, for example, a motion contrast image of the subject's eye, and the target region can include, for example, at least one of a no-perfusion region, a foveal vessel region, and an optic disc region. In addition, the region detection unit 302 can detect a target region using a learned model obtained by learning using a medical image of a subject as learning data.

With such a configuration, the image processing apparatus according to the present embodiment detects an area such as an NPA where overcorrection occurs due to image processing of a medical image using a trained model, and performs the overcorrection on the detected area. image processing can be applied to reduce the In such processing, pixel values can be reduced only in areas where overcorrection has occurred without reducing pixel values for the entire image. Therefore, overcorrection due to image processing of medical images using a trained model can be reduced. Accordingly, in image diagnosis, it is possible to assist the operator's judgment regarding the distinction between noise and blood vessels.

Various modifications are possible for the series of image processing according to the present embodiment. A specific example of a series of image processing according to this embodiment will be described in detail below with reference to FIGS. 8 to 13 . For each example, the description of the same processing as the above-described image processing shown in FIG. 7 will be omitted.

<First example of image processing method>
A first example of the image processing method will be described with reference to FIG. In this example, an image processing method for suppressing overcorrection by the image quality improvement engine for FAZ and NPA will be described.

Since the processes in steps S801 and S802 are the same as those in steps S701 and S702, description thereof is omitted. In step S802, when the high-quality medical image is obtained, the process proceeds to step S803.

In step S803, the area detection unit 302 uses the area detection engine to detect the FAZ and NPA areas in the medical image (input image) acquired in step S801. FAZ and NPA may be detected separately, or may be detected collectively as one region. Alternatively, only one of the regions may be detected, or regions other than the FAZ and NPA may be detected as perfusion regions. After the region detection unit 302 outputs the information indicating the detected region, the ROI setting unit 303 sets the ROI in the high-quality medical image in step S802 based on the output information. Note that the pixel positions of the detected regions in the medical image can correspond to each other in the medical image and the high-quality medical image.

Finally, in step S804, the blend processing unit 304 uses the input image and the high-quality medical image acquired in step S802 to perform blend processing on the detected region in the high-quality medical image. conduct. Here, the known alpha blending process or the like may be used for the blending process. Also, the blend ratio may be a fixed value, or the blend processing unit 304 may change the blend ratio for each pixel according to the reliability (likelihood, probability) of the attribute information output by the area detection engine. Specifically, for pixels having attribute information of non-perfused regions, the α blend ratio can be set such that the higher the reliability, the higher the blend ratio of the input image.

In addition, when attribute information such as FAZ and NPA is detected separately, the respective blend ratios may be changed according to the distinguished attribute information. Furthermore, the input image may be appropriately combined with pixels around the target area such as FAZ and NPA, for example, areas other than FAZ and NPA. This makes it possible to moderate sharp changes at the boundary of the area.

Note that the blend ratio may be instructed by the operator via the UI. For example, a GUI such as a slide bar may be used to set the blend ratio corresponding to the suppression intensity of overcorrection.

According to this example, with respect to FAZ and NPA, the original state before image quality improvement can be left to some extent. Therefore, in image diagnosis, it is possible to assist the operator's judgment regarding discrimination between noise and blood vessels.

When detecting a target region in a medical image (first image) that is an input image, the region detection unit 302 can use a region detection engine configured by a trained model as the region detection engine. . In this case, as learning data for the region detection engine, a medical image, which is an input image, is used as input data, and a region label image having, as a pixel value, information indicating the label of each region in the medical image, which is an input image, is used as output data. be able to. The region detection unit 302 can also use a region detection engine configured using a rule-based algorithm or the like based on the structure of the subject. In this case as well, the region detection unit 302 detects target regions such as FAZ and NPA from the characteristic portions of the medical image in which image quality has not been improved using the trained model, in other words, in which overcorrection has not occurred. can be detected.

<Second example of image processing method>
A second example of the image processing method will be described with reference to FIG. In this example, an image processing method for emphasizing image quality improvement processing for NPA will be described.

If the performance of the area detection engine is high, the area detection unit 302 can more appropriately detect areas such as FAZ and NPA. For example, FAZ and NPA are naturally devoid of blood vessels. Therefore, if the performance of the region detection engine is high, these regions are processed by the image quality enhancement engine and then further darkened by BC adjustment to obtain an image more suitable for diagnosis. The processing of this example, which can obtain images more suitable for diagnosis using such image processing, will be described in more detail below.

Since the processes in steps S901 and S902 are the same as those in steps S701 and S702, description thereof is omitted. When the high-quality medical image is obtained in step S902, the process proceeds to step S903.

In step S903, the area detection unit 302 uses the area detection engine to detect the FAZ and NPA areas in the medical image (input image) acquired in step S901. FAZ and NPA may be detected separately, or may be collected and detected as one region. Alternatively, only one of the regions may be detected, or regions other than the FAZ and NPA may be detected as perfusion regions. After the region detection unit 302 outputs the information indicating the detected region, the ROI setting unit 303 sets the ROI in the high-quality medical image in step S902 based on the output information. Note that the pixel positions of the detected regions in the medical image can correspond to each other in the medical image and the high-quality medical image.

Finally, in step S904, the BC adjustment unit 305 performs BC adjustment processing on pixels in the detected regions (FAZ and NPA regions) in the high-quality medical image acquired using the image quality enhancement engine in step S902. I do. The BC adjustment may be performed so as to darken the luminance value (brightness) of the pixel value. For example, the luminance value may simply be lowered or set to zero. Also, image processing according to the original pixel values, such as gamma correction, may be applied. Note that the strength of correction may be changed between FAZ and NPA, respectively, and if the attribute and reliability are held as information indicating the detected area, BC adjustment may be performed according to the reliability.

The method and intensity of BC adjustment may be instructed by the operator via the UI. For example, a GUI such as a radio button may be used to select the method, and a GUI such as a slide bar may be used to specify the intensity.

According to this example, it is also possible to improve the visibility of areas such as the NPA and FAZ. In this case as well, it is possible to assist the doctor's judgment regarding the distinction between noise and blood vessels in image diagnosis. Furthermore, the suppression process shown in the first example and the enhancement process shown in the second example may be selected for each attribute of the area. For example, FAZ may be emphasized and NPA may be suppressed. In this case, it is possible to generate an image more suitable for diagnosis according to each region, and to assist the doctor's judgment.

<Third example of image processing method>
A third example of the image processing method will be described with reference to FIG. In this example, image processing for detecting areas such as FAZ and NPA using an area detection engine from a medical image whose image quality has been enhanced using an image quality enhancement engine and suppressing overcorrection in these areas will be described.

Since the processing in steps S1001 and S1002 is the same as that in steps S701 and S702, the description thereof is omitted. In step S1002, when the high-quality medical image is obtained, the process proceeds to step S1003.

In step S1003, the region detection unit 302 uses the region detection engine to detect the FAZ and NPA regions in the medical image whose image quality has been improved in S1002. FAZ and NPA may be detected separately from each other, or may be detected collectively as one region. Alternatively, only one of the regions may be detected, or regions other than the FAZ and NPA may be detected as perfusion regions. After the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the high-quality medical image in step S1002 based on the output information.

Finally, in step S1004, the blend processing unit 304 uses the high-quality medical image and the medical image acquired in step S1001 to perform blend processing on the detected region in the high-quality medical image. conduct. The blending process may be the same as the process described in the first example.

In the case of this example, as in the first example, the original state before image quality improvement can be left to some extent with respect to FAZ and NPA. Therefore, in image diagnosis, it is possible to assist the operator's judgment regarding the distinction between noise and blood vessels.

Note that when detecting a target region in a high-quality medical image, the region detection unit 302 can use a region detection engine configured by a trained model. In this case, as learning data for the region detection engine, a high-quality medical image (second image) is used as input data, and information indicating the label of each region in the medical image (first image) that is the input image. can be used as output data. Here, the high-quality medical image that is the input data is a medical image that has been high-quality enhanced using an image-quality enhancement engine, and is a high-quality image in which overcorrection occurs in a target area such as NPA. be able to. On the other hand, the region label image to be output data may be an image obtained by labeling a medical image before image quality enhancement using the image quality enhancement engine. In this case, the area detection engine configured by the trained model can output an area label image including the label of the overcorrected target area from the input high-quality image according to the tendency of learning. . By using high-quality images as input, it is expected that the region detection engine configured with trained models will be able to extract features in images more appropriately, and output region label images with higher accuracy. be done.

<Fourth example of image processing method>
A fourth example of the image processing method will be described with reference to FIG. In this example, image processing that detects areas such as FAZ and NPA using an area detection engine from a medical image that has been enhanced using the image quality enhancement engine and emphasizes image quality enhancement processing for these areas will be described. do.

Since the processing in steps S1101 and S1102 is the same as that in steps S701 and S702, description thereof will be omitted. When the high-quality medical image is obtained in step S1102, the process proceeds to step S1103.

In step S1103, the region detection unit 302 uses the region detection engine to detect the FAZ and NPA regions in the medical image whose image quality has been enhanced in step S1102. FAZ and NPA may be detected separately, or may be detected collectively as one region. Alternatively, only one of the regions may be detected, or regions other than the FAZ and NPA may be detected as perfusion regions. After the region detection unit 302 outputs the information indicating the detected region, the ROI setting unit 303 sets the ROI in the high-quality medical image in step S1102 based on the output information.

Finally, in step S1104, the BC adjustment unit 305 performs BC adjustment processing on pixels in the detected area (FAZ or NPA area) in the high-quality medical image. The BC adjustment process may be the same process as the process described in the second example.

In the case of this example as well, the visibility of the regions such as the NPA and FAZ can be improved as in the second example. In this case as well, it is possible to support the doctor's judgment regarding the distinction between noise and blood vessels in image diagnosis. Furthermore, the suppression processing shown in the first and third examples and the enhancement processing shown in the fourth example may be selected for each attribute of the area. For example, FAZ may be emphasized and NPA may be suppressed. In this case, it is possible to generate an image more suitable for diagnosis according to each region, and to assist the doctor's judgment.

<Fifth example of image processing method>
A fifth example of the image processing method will be described with reference to FIG. In this example, regions such as FAZ and NPA are detected using a region detection engine for a medical image before image quality enhancement, and BC adjustment is performed on these regions, and then the image quality enhancement engine is used. Image processing that applies image quality enhancement processing will be described.

Since the processing in step S1201 is the same as that in step S701, description thereof is omitted. In step S1202, the area detection unit 302 uses the area detection engine to detect the FAZ and NPA areas in the medical image (input image) acquired in step S1201. After the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the input image based on the output information.

Next, in step S1203, the BC adjustment unit 305 performs BC adjustment processing on pixels in the detected area (the FAZ or NPA area) in the input image. Note that the BC adjustment process may be the same process as the process described in the second example.

Finally, in step S1204, the image quality enhancement processing unit 301 uses the image quality enhancement engine to perform image quality enhancement processing on the medical image on which the BC adjustment processing has been performed in step S1203. Acquire medical images. Blend processing and BC adjustment processing based on the detected area may be further added to the medical image after the image quality enhancement processing by the image quality enhancement processing unit 301 . The blending process and the BC adjustment process may be similar to the processes described in the first and second examples, respectively.

As described above, in this example, the region detection unit 302 detects the target region in the first medical image of the subject. In addition, the BC adjusting unit 305 adjusts the target region in the first medical image of the subject so that the difference between the pixel values of the target region and the pixel values of the regions other than the target region widens, and The BC adjustment process is performed so that the pixel value is lower than the pixel value of the target area before the BC adjustment process. Furthermore, the image quality improvement processing unit 301 performs image quality improvement processing on the first image that has undergone BC adjustment processing, using an image quality improvement model obtained by learning using medical images of the subject as learning data. to obtain a high-quality second medical image of the subject.

In the case of this example, the BC adjustment process is performed to perform the image quality improvement process on the medical image in which the pixel values of the region where overcorrection occurs are lowered. In such processing, the pixel values can be lowered only in areas where overcorrection occurs without lowering the pixel values of the entire image. Therefore, for example, overcorrection in areas such as NPA and FAZ can be suppressed, and the visibility of areas such as NPA and FAZ can be improved. In this case as well, it is possible to assist the doctor's judgment regarding the distinction between noise and blood vessels in image diagnosis.

<Sixth example of image processing method>
A sixth example of the image processing method will be described with reference to FIG. In this example, the area detection engine is used to perform area detection on a medical image whose image quality has been enhanced using the image quality enhancement engine. After that, regarding the medical image before the image quality enhancement, BC adjustment is performed based on the area detected by the above-described area detection, and then image processing that applies image quality enhancement processing using the image quality enhancement engine. explain.

Since the processes in steps S1301 and S1302 are the same as those in steps S701 and S702, description thereof will be omitted. When the high-quality medical image is acquired in step S1302, the process proceeds to step S1303.

In step S1303, the area detection unit 302 uses an area detection engine to perform area detection processing on the image whose image quality has been enhanced in step S1302. Note that the areas to be detected may be FAZ, NPA, etc., as in the first to fifth examples. By applying the region detection engine to the high-quality medical image in this way, it can be expected that the detection performance can be improved. After the area detection unit 302 outputs the information indicating the detected area, the ROI setting unit 303 sets the ROI in the input image based on the output information. Note that the pixel positions of the detected regions in the high-quality medical image can correspond to the medical image and the high-quality medical image.

Next, in step S1304, the BC adjustment unit 305 performs BC adjustment processing on the medical image (input image) acquired in step S1301 based on the area detected in step S1303. Note that the BC adjustment process may be the same process as the process described in the second example.

Finally, in step S1305, the image quality enhancement processing unit 301 uses the image quality enhancement engine to perform image quality enhancement processing on the medical image on which the BC adjustment processing has been performed in step S1303. acquire a medical image. Blending processing and BC adjustment based on region classification may be further added to the image after image quality enhancement processing by the image quality enhancement processing unit 301 in step S1305. Blend processing and BC adjustment may be processing similar to the processing described in the first and second examples, respectively.

As described above, in this example, the image quality enhancement processing unit 301 uses the image quality enhancement model obtained by learning using the medical image of the subject as learning data to obtain the first image of the medical image of the subject. A high-quality image of the first medical image of the subject is acquired from the first image. The region detection unit 302 detects a target region in an image obtained by enhancing the image quality of the first medical image of the subject. The BC adjusting unit 305 adjusts the pixel values of the target region in the first medical image of the subject so that the difference between the pixel values of the target region and the pixel values of the regions other than the target region widens. is lower than the pixel value of the target area before the BC adjustment process. Furthermore, the image quality improvement processing unit 301 uses the image quality improvement model to perform image quality improvement processing on the first image on which the BC adjustment processing has been performed, and obtains a high quality second image of the medical image of the subject. Get an image of Note that the region detection unit 302 detects the target region using an image obtained by enhancing the image quality of the first medical image of the subject for which the BC adjustment is not performed by the BC adjustment unit 305 .

In the case of this example, the BC adjustment process is performed to perform the image quality improvement process on the medical image in which the pixel values of the region where overcorrection occurs are lowered. In such processing, the pixel values can be lowered only in areas where overcorrection occurs without lowering the pixel values of the entire image. Therefore, for example, overcorrection in areas such as NPA and FAZ can be suppressed, and the visibility of areas such as NPA and FAZ can be improved. In this case as well, it is possible to assist the doctor's judgment regarding the distinction between noise and blood vessels in image diagnosis.

As described above, various modifications are possible for the image processing using the image quality improvement engine and the area detection engine. These illustrated examples are capable of many variations and are not intended to be construed as limiting the present disclosure. For example, when performing follow-up observation of medical images, blending processing and BC adjustment processing based on the detected region may be matched to the same conditions, or these conditions may be stored and managed for each subject.

(Second embodiment)
An image processing system including an image processing apparatus according to the second embodiment of the present invention will be described in detail with reference to FIG. FIG. 14 is a flow chart showing the flow of a series of image processing according to this embodiment. Note that the configuration of the image processing system according to this embodiment is the same as that of the image processing system according to the first embodiment, so the same reference numbers are used and the description is omitted. However, the region detection unit 302, the ROI setting unit 303, the blend processing unit 304, and the BC adjustment unit 305 may not be provided in the image quality enhancement unit according to this embodiment. The image processing system according to the present embodiment will be described below, focusing on differences from the image processing system according to the first embodiment.

In the first embodiment, a method for obtaining a more suitable high-quality image by combining an image quality enhancement engine using deep learning and an area detection engine and partially performing additional image correction has been described. . On the other hand, in the present embodiment, a configuration for performing image quality enhancement processing using an image quality enhancement engine constructed by applying these methods will be described.

The machine learning algorithm of the image quality enhancement engine according to this embodiment may be the same as the machine learning algorithm of the image quality enhancement engine according to the first embodiment. However, the high-quality image finally generated by any of the methods described in the first embodiment is used as the output data of the learning data of the image quality improvement engine according to the present embodiment. These processing flows are applied to a predetermined number of medical images to output high-quality images, which are then paired to construct a learning data set. It can be expected that an image quality improvement engine obtained by performing learning using such learning data can generate a high quality image in which overcorrection is suppressed according to the tendency of learning. In addition, by collectively performing machine learning including partially corrected results, it is expected that the image processing load during inference can be reduced.

A processing procedure of the image processing apparatus 101 of this embodiment will be described with reference to FIG. First, in step S1401, the acquisition unit 101-1 acquires a medical image. The processing of step S1401 may be the same as the processing of step S701.

Next, in step S1402, the image quality enhancement processing unit 301 uses the image quality enhancement engine obtained by learning using the learning data as described above to enhance the image quality of the medical image acquired in step S1401. Process and obtain a high-quality image. The high-quality image acquired in this way becomes a high-quality image in which overcorrection is suppressed according to the learning tendency of the image quality improvement engine.

As described above, the image processing apparatus according to this embodiment includes the acquisition unit 101-1 and the image quality enhancement processing unit 301. FIG. Acquisition unit 101-1 acquires a first medical image of a subject. The image quality enhancement processing unit 301 performs image quality enhancement processing using a first trained model obtained by learning using a medical image of the subject as learning data, and blend processing or brightness enhancement processing in a target region of the medical image of the subject. Using a second trained model obtained by learning using a medical image of a subject that has undergone correction processing for at least one of brightness and contrast as learning data, image quality enhancement processing is performed on the first image. A second image of the medical image of the subject is obtained. Also in this case, it is possible to generate a high-quality image in which overcorrection is suppressed.

By performing various types of image analysis using a medical image whose image quality has been enhanced by any of the methods according to the first and second embodiments, more accurate analysis can be performed. Also, for disease screening using an artificial intelligence engine, high-quality images can be used as input images, and more accurate processing can be expected.

[Modification 1]
In the first embodiment, a plurality of front images corresponding to a plurality of depth ranges can be used as learning data when the region detection unit 302 detects a target region using a trained model. For example, a plurality of OCTA frontal images corresponding to a plurality of depth ranges can be used as input data for learning data of an area detection engine using a trained model. As the output data of the learning data, the area label image corresponding to the OCTA front image used as the input data may be used. The region detection unit 302 uses a common learned model obtained by learning such learning data to detect an object corresponding to an arbitrary depth range, for example, a depth range selected according to an instruction from the examiner. A two-dimensional target region can be detected from the front image of the specimen. Note that the depth range may include, for example, the superficial layer, the deep layer, the outer layer, the choroidal vascular network, and different depth ranges with different offset values from the reference layer. Further, the front image is not limited to the OCTA front image, and may be an En-Face image.

Also, a plurality of trained models corresponding to depth ranges may be prepared using learning data for each depth range. In this case, the area detection unit 302 selects a plurality of learned models obtained by learning using a plurality of front images of the subject corresponding to a plurality of depth ranges as learning data, and selects the A trained model corresponding to the selected depth range may be selected, and a two-dimensional region of interest may be detected from the frontal image of the subject using the selected trained model. Further, the region detection unit 302 selects a trained model corresponding to the depth range of the medical image used for region detection processing from among the plurality of such trained models, and uses the selected trained model to detect the subject. A two-dimensional target area may be detected from the front image of the specimen.

Furthermore, as learning data, a plurality of front images of different depth ranges may be combined and used (for example, a plurality of front images may be divided into the surface layer side and the deep layer side). Also in this case, the region detection unit 302 selects a learned model corresponding to the depth range selected according to an instruction from the examiner or the depth range of the medical image used for region detection processing, and selects the selected learned model. can be used to detect a two-dimensional region of interest from the frontal image of the subject.

Furthermore, a three-dimensional image may be used as input data for learning data of an area detection engine using a trained model. For example, a three-dimensional tomographic image or a three-dimensional motion contrast image is used as input data for learning data, and a three-dimensional region label image obtained by labeling the three-dimensional image is used as output data for learning data. be able to. In this case, the region detection unit 302 can detect the three-dimensional target region from the three-dimensional image of the subject using a trained model obtained by learning using the three-dimensional image of the subject as learning data.

Similarly, the image quality enhancement engine may also perform learning using learning data in which a three-dimensional image is input data and a three-dimensional image with enhanced image quality is output data. In this case, the image quality enhancement processing unit 301 uses an image quality enhancement engine obtained by learning using a three-dimensional medical image of the subject as learning data to obtain a three-dimensional image of the three-dimensional medical image of the subject. A dimensional image can be acquired. Furthermore, the blend processing unit 304 and the BC adjustment unit 305 can be configured to perform various types of processing on a three-dimensional target region in a three-dimensional medical image. In this case, the same effect as in the first embodiment can be obtained by processing a three-dimensional medical image. Similarly, a three-dimensional image may be used as learning data for the image quality enhancement engine according to the second embodiment. In this case, the same effect as in the second embodiment can be obtained by processing a three-dimensional medical image.

[Modification 2]
Perform learning to adjust (tune) the trained model used for image quality improvement processing and region detection processing (learned model for image quality improvement, trained model for region detection) for each subject, A dedicated trained model may be generated. For example, transfer of a general-purpose trained model for generating high-quality medical images and a general-purpose trained model for detecting regions using medical images acquired in past examinations of the subject. Training can be performed to generate a trained model specific to that subject. By linking the trained model dedicated to the subject with the ID of the subject and storing it in the storage unit 101-3 or an external device such as a server, the image processing apparatus 101 can perform the current examination of the subject. is performed, a trained model dedicated to the subject can be specified and used based on the ID of the subject. By using the trained model dedicated to the subject, it is possible to improve the accuracy of the image quality enhancement process and the area detection process.

[Modification 3]
An image quality improvement engine may be prepared for each type of image that is input data. For example, an image quality enhancement model for anterior eye images, an image quality enhancement model for SLO images, an image quality enhancement model for tomographic images, an image quality enhancement model for OCTA frontal images, and the like may be prepared. Further, for the OCTA front image and the En-Face image, a high image quality model may be prepared for each depth range for generating the image. For example, a high image quality model for surface layers and a high image quality model for deep layers may be prepared. Furthermore, the image quality improvement model may be obtained by learning an image for each imaging site (for example, the center of the macula and the center of the optic papilla), or may be obtained by learning regardless of the imaging site. .

At this time, for example, the image quality of the fundus OCTA frontal image is improved using a high image quality model obtained by learning the fundus OCTA frontal image as learning data, and the anterior eye OCTA frontal image is learned as learning data. The image quality improvement model may be used to improve the image quality of the anterior ocular OCTA front image. Also, the high image quality model may be one that has undergone learning regardless of the imaging region. Here, for example, the fundus OCTA front image and the anterior eye OCTA front image may be relatively similar in distribution of blood vessels to be imaged. In this way, in a plurality of types of medical images in which the states of imaging targets are relatively similar to each other, feature amounts may be relatively similar to each other. Therefore, for example, using a high image quality model obtained by learning the fundus OCTA front image as learning data, not only the image quality of the fundus OCTA front image can be improved, but also the anterior eye OCTA front image can be improved. may be In addition, for example, it is possible to improve not only the image quality of the anterior eye OCTA front image but also the fundus OCTA front image by using a high image quality model obtained by learning the anterior eye OCTA front image as learning data. may be configured. That is, using a high image quality model obtained by learning at least one type of frontal image of the fundus OCTA frontal image and the anterior eye OCTA frontal image as learning data, the fundus OCTA frontal image and the anterior eye OCTA frontal image are obtained. At least one type of front image may be configured to be capable of high image quality.

Here, consider a case where an OCT apparatus capable of photographing the fundus can also photograph the anterior eye. At this time, for the OCTA En-Face image, for example, the fundus OCTA front image may be applied in the fundus imaging mode, and the anterior eye OCTA front image may be applied in the anterior segment imaging mode. At this time, when the image quality enhancement button is pressed, for example, in the fundus imaging mode, in the display area of the OCTA En-Face image, the image of the low-quality fundus OCTA front image and the high-quality fundus OCTA front image is displayed. The display of one may be configured to change to the display of the other. Further, when the image quality enhancement button is pressed, for example, in the anterior segment imaging mode, a low-quality anterior OCTA front image and a high-quality anterior OCTA front image are displayed in the display area of the OCTA En-Face image. may be configured such that the display of one of them is changed to the display of the other.

An OCT apparatus capable of photographing the fundus may be configured so that an anterior eye adapter can be attached when the anterior eye can also be photographed. Alternatively, the optical system of the OCT apparatus may be configured to be movable by a distance approximately equal to the axial length of the subject's eye without using the anterior eye adapter. At this time, the focus position of the OCT apparatus may be configured so as to be able to be greatly changed to the normal vision side to the extent that an image is formed on the anterior eye.

For the tomographic image, for example, a fundus OCT tomographic image may be applied in the fundus imaging mode, and an anterior eye OCT tomographic image may be applied in the anterior segment imaging mode. Further, the image quality enhancement processing for the fundus OCTA front image and the anterior eye OCTA front image described above can also be applied as image quality enhancement processing for the fundus OCT tomographic image and the anterior eye OCT tomographic image, for example. At this time, when the image quality enhancement button is pressed, for example, in the fundus imaging mode, one of the low-quality fundus OCT tomographic image and the high-quality fundus OCT tomographic image is displayed in the tomographic image display area. It may be configured to change to the other display. Further, when the image quality enhancement button is pressed, for example, in the anterior segment imaging mode, one of the low image quality anterior eye OCT tomographic image and the high image quality anterior eye OCT tomographic image is displayed in the tomographic image display area. may be configured such that the display of one is changed to the display of the other.

For the tomographic image, for example, a fundus OCTA tomographic image may be applied in the fundus imaging mode, and an anterior eye OCTA tomographic image may be applied in the anterior segment imaging mode. Further, the image quality enhancement processing of the fundus OCTA front image and the anterior eye OCTA front image described above can also be applied as the image quality enhancement processing of the fundus OCTA tomographic image and the anterior eye OCTA tomographic image, for example. At this time, for example, in the fundus imaging mode, in the display area of the tomographic image, information indicating a blood vessel region (for example, motion contrast data equal to or greater than a threshold value) in the fundus OCTA tomographic image is superimposed on the fundus OCT tomographic image at the corresponding position. may be configured to be displayed as Further, for example, in the anterior segment imaging mode, information indicating the blood vessel region in the anterior eye OCTA tomographic image may be superimposed and displayed on the anterior eye OCT tomographic image at the corresponding position in the tomographic image display region. .

In this way, for example, when the feature values (appearance of an object to be imaged) of a plurality of types of medical images are considered to be relatively similar to each other, at least one of the plurality of types of medical images A high image quality model obtained by learning medical images as learning data may be used to improve the image quality of at least one type of medical image among a plurality of types of medical images. As a result, for example, a common learned model (common image quality improvement model) can be used to enable the image quality improvement of multiple types of medical images.

The display screen in the fundus imaging mode and the display screen in the anterior segment imaging mode may have the same display layout, or may have display layouts corresponding to the respective imaging modes. Various conditions such as imaging conditions and analysis conditions may be the same or different between the fundus imaging mode and the anterior segment imaging mode.

Here, the target image of the image quality enhancement process may be, for example, a plurality of OCTA front images (OCTA En-Face images, motion contrast En-Face images) (corresponding to a plurality of depth ranges). Also, the target image for image quality enhancement processing may be, for example, one OCTA front image corresponding to one depth range. In addition, instead of the OCTA front image, the target image of the image quality improvement process is, for example, a luminance front image (luminance En-Face image), an OCT tomographic image that is a B-scan image, or a tomographic image of motion contrast data ( OCTA tomographic image). In addition, the target image of the image quality improvement process is not only the OCTA front image, but also various medical images such as OCT tomographic images that are luminance front images and B-scan images and tomographic images of motion contrast data (OCTA tomographic images). It may be an image. That is, the image to be subjected to the image quality enhancement process may be, for example, at least one of various medical images displayed on the display screen of the output unit 103 . At this time, for example, since the feature amount of an image may differ for each type of image, a trained model for image quality improvement corresponding to each type of image to be subjected to image quality improvement processing may be used. For example, when the image quality improvement button is pressed in response to an instruction from the examiner, the OCTA frontal image is not only processed to improve the image quality using the trained model for image quality improvement corresponding to the OCTA frontal image, but also The OCT tomographic image may also be configured to perform image quality enhancement processing using a trained model for image quality enhancement corresponding to the OCT tomographic image. Further, for example, when an image quality enhancement button is pressed in response to an instruction from the examiner, a high-quality OCTA frontal image generated using a trained model for image quality enhancement corresponding to the OCTA frontal image is displayed. , and may be configured to display a high-quality OCT tomographic image generated using a trained model for improving image quality corresponding to the OCT tomographic image. At this time, a line indicating the position of the OCT tomographic image may be superimposed on the OCTA front image. Further, the line may be configured to be movable on the OCTA front image according to an instruction from the examiner. Further, when the display of the image quality improvement button is in an active state, after the line is moved, a high image quality OCT tomogram obtained by performing image quality improvement processing on the OCT tomographic image corresponding to the position of the current line It may be configured to be changed to display an image. Further, by displaying an image quality improvement button for each image to be subjected to the image quality improvement process, the image quality improvement process may be performed independently for each image.

Information indicating a blood vessel region (for example, motion contrast data equal to or greater than a threshold) in an OCTA tomographic image may be superimposed and displayed on an OCT tomographic image, which is a B-scan image of the corresponding position. At this time, for example, when the quality of the OCT tomographic image is improved, the quality of the OCTA tomographic image at the corresponding position may be improved. Information indicating the blood vessel region in the OCTA tomographic image obtained with high image quality may be superimposed and displayed on the OCT tomographic image obtained with high image quality. The information indicating the blood vessel region may be any information as long as it is identifiable information such as color. In addition, superimposed display and non-display of information indicating a blood vessel region may be configured to be changeable according to an instruction from the examiner. Further, when the line indicating the position of the OCT tomographic image is moved on the OCTA front image, the display of the OCT tomographic image may be updated according to the position of the line. At this time, since the OCTA tomographic image at the corresponding position is also updated, the superimposed display of the information indicating the blood vessel region obtained from the OCTA tomographic image may be updated. As a result, for example, it is possible to effectively confirm the three-dimensional distribution and state of the blood vessel region while easily confirming the positional relationship between the blood vessel region and the region of interest at an arbitrary position. Further, the improvement of the image quality of the OCTA tomographic image may be the image quality improvement processing such as averaging processing of a plurality of OCTA tomographic images acquired at the corresponding position instead of using the trained model for image quality improvement. . Also, the OCT tomographic image may be a pseudo OCT tomographic image reconstructed as a cross section at an arbitrary position in OCT volume data. Also, the OCTA tomographic image may be a pseudo OCTA tomographic image reconstructed as a cross section at an arbitrary position in the OCTA volume data. The arbitrary position may be at least one arbitrary position, and may be configured to be changeable according to an instruction from the examiner. At this time, a plurality of pseudo tomographic images corresponding to a plurality of positions may be reconstructed.

Note that only one tomographic image (for example, an OCT tomographic image or an OCTA tomographic image) may be displayed, or a plurality of tomographic images may be displayed. When a plurality of tomographic images are displayed, the tomographic images obtained at different positions in the sub-scanning direction may be displayed. When displayed, images in different scanning directions may be displayed. For example, when displaying a plurality of tomographic images obtained by radial scanning or the like with high image quality, a plurality of partially selected tomographic images (for example, two tomographic images at mutually symmetrical positions with respect to the reference line) ) may be displayed respectively. In addition, multiple tomographic images are displayed on the display screen for follow-up observation (display screen for follow-up), and instructions for higher image quality and analysis results (for example, the thickness of a specific layer, etc.) are obtained by the same method as the above method. ) may be displayed. At this time, the plurality of tomographic images to be displayed may be a plurality of tomographic images of a predetermined portion of the subject's eye obtained at different dates and times, or may be a plurality of tomographic images obtained at different times on the same examination date. good too. Further, image quality enhancement processing may be performed on the tomographic image based on the information stored in the database by a method similar to the method described above.

Similarly, when the SLO image is displayed with high image quality, for example, the SLO image displayed on the same display screen may be displayed with high image quality. Furthermore, in the case of displaying the front image with luminance in a higher quality, for example, the front image with luminance displayed on the same display screen may be displayed with a higher image quality. Furthermore, a plurality of SLO images and a front image of brightness are displayed on the display screen for follow-up observation, and instructions for improving image quality and analysis results (e.g., thickness of a specific layer, etc.) are displayed by the same method as the above method. may be performed. Further, image quality enhancement processing may be performed on the SLO image or the luminance front image based on the information stored in the database by a technique similar to the above-described method. It should be noted that the display of the tomographic image, the SLO image, and the luminance front image is an example, and these images may be displayed in any manner according to the desired configuration. Also, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the luminance front image may be displayed with high image quality by a single instruction.

With such a configuration, the display control unit 101-5 can cause the output unit 103 to display a high-quality image obtained by the high-quality image processing. Note that if at least one of a plurality of conditions related to the display of high-quality images, the display of analysis results, the depth range of the front image to be displayed, etc. is selected, even if the display screen transitions, the selection It may be configured such that the specified conditions are maintained. The display control unit 101-5 may control the display of various high-quality images, information indicating the lines and blood vessel regions, and the like.

Also, the high image quality model may be used for at least one frame of the live moving image on the preview screen displayed on the output unit 103 by the display control unit 101-5. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the learned model corresponding to each live moving image may be used. For example, an anterior eye image used for alignment processing may be an image whose image quality has been enhanced using an image quality enhancement model for an anterior eye image. Similarly, for various images used in the process of detecting a predetermined region in various images, images whose image quality has been enhanced using the image quality enhancement model for each image may be used.

At this time, for example, when the image quality enhancement button is pressed in response to an instruction from the examiner, a plurality of different types of live moving images (for example, anterior eye images, SLO images, tomographic images) are displayed ( at the same time), the display may be changed to display a high-quality moving image obtained by the high-quality image processing. At this time, the display of the high-quality moving image may be continuous display of high-quality images obtained by subjecting each frame to high-quality image processing. Further, for example, since the feature amount of an image may differ for each type of image, a trained model for image quality enhancement corresponding to each type of target image for image quality enhancement processing may be used. For example, when an image quality enhancement button is pressed in response to an instruction from the examiner, the image quality enhancement model corresponding to the anterior eye image is used to not only perform image quality enhancement processing for the anterior eye image, but also to correspond to the SLO image. The SLO image may also be configured to perform the image quality enhancement process using the image quality enhancement model. Further, for example, when an image quality enhancement button is pressed in response to an instruction from the examiner, the display is changed to a high quality anterior eye image generated using an image quality enhancement model corresponding to the anterior eye image. In addition, the display may be changed to a high-quality SLO image generated using a high-quality model corresponding to the SLO image. Further, for example, when the image quality enhancement button is pressed in response to an instruction from the examiner, not only is the image quality enhancement processing performed on the SLO image using the image quality enhancement model corresponding to the SLO image, but also the tomographic image is processed. The tomographic image may also be configured to perform image quality enhancement processing using the image quality enhancement model. Further, for example, when the image quality enhancement button is pressed in response to an instruction from the examiner, the display is simply changed to display a high image quality SLO image generated using an image quality enhancement model corresponding to the SLO image. Instead, the display may be changed to display a high-quality tomographic image generated using a high-quality model corresponding to the tomographic image. At this time, a line indicating the position of the tomographic image may be superimposed on the SLO image. Also, the line may be configured to be movable on the SLO image according to an instruction from the examiner. Further, when the display of the image quality improvement button is in an active state, after the line is moved, a high quality tomographic image obtained by performing image quality improvement processing on the tomographic image corresponding to the position of the current line is displayed. It may be configured to change the display. Further, by displaying an image quality improvement button for each image to be subjected to the image quality improvement process, the image quality improvement process may be performed independently for each image.

As a result, the processing time can be shortened even for a live moving image, for example, so that the examiner can obtain highly accurate information before the start of imaging. For this reason, for example, when the operator corrects the alignment position while checking the preview screen, it is possible to reduce the failure of re-imaging and the like, so that the accuracy and efficiency of diagnosis can be improved. In addition, the image processing apparatus 101 performs the above-described rescanning operation so that a partial area such as an artifact area obtained by segmentation processing or the like is re-captured (rescanned) during or at the end of imaging in response to an instruction regarding the start of imaging. The scanning means may be driven and controlled. Depending on the state of the subject's eye, such as the movement of the subject's eye, it may not be possible to capture an image successfully with one rescan. At this time, the rescan may be terminated even during the predetermined number of rescans according to an instruction from the operator (for example, after pressing the photographing cancel button). At this time, the imaging data may be stored until the rescan is completed in accordance with an instruction from the operator. Note that, for example, a confirmation dialog may be displayed after the photography cancel button is pressed, and it may be possible to select whether to save the photography data or discard the photography data according to an instruction from the operator. Also, for example, after pressing the shooting cancel button, the next rescan is not executed (although the current rescan is executed until it is completed), and the system waits until there is an instruction (input) from the operator in the confirmation dialog. It may be configured as Further, for example, when the information indicating the certainty of the object detection result (for example, the numerical value indicating the ratio) regarding the part of interest exceeds the threshold value, each adjustment, the start of imaging, etc. may be automatically performed. good. Also, for example, when the information indicating the certainty of the object detection result (for example, the numerical value indicating the ratio) regarding the target part exceeds the threshold, each adjustment and the start of imaging can be executed according to the instructions from the examiner. It may be configured to change to a normal state (cancel the execution prohibition state).

Here, during the auto-alignment, there is a possibility that the object to be imaged, such as the retina of the subject's eye 200, has not yet been successfully imaged. For this reason, there is a possibility that a high-quality image cannot be obtained with high accuracy due to the large difference between the medical image input to the trained model and the medical image used as learning data. Therefore, when an evaluation value such as image quality evaluation of a tomographic image (B-scan image) exceeds a threshold value, display of high-quality moving images (continuous display of high-quality frames) may be automatically started. Further, when the evaluation value of the image quality evaluation of the tomographic image exceeds the threshold value, the image quality improvement button may be changed to a state in which the examiner can designate (active state). The image quality enhancement button is a button for specifying execution of image quality enhancement processing. Of course, the high image quality button may be a button for instructing display of a high quality image.

Alternatively, different image quality enhancement models may be prepared for different image capturing modes with different scan patterns, etc., and a trained model for image quality enhancement corresponding to the selected image capturing mode may be selected. Also, one image quality enhancement model obtained by learning learning data including various medical images obtained in different imaging modes may be used.

Here, in an ophthalmologic apparatus, for example, an OCT apparatus, the scan pattern of the light flux used for measurement and the imaging region differ for each imaging mode. For this reason, with respect to the trained model that uses tomographic images as input data, a trained model is prepared for each imaging mode, and the trained model corresponding to the selected imaging mode is selected according to the operator's instruction. may be configured. In this case, the imaging modes may include, for example, a retinal imaging mode, an anterior segment imaging mode, a vitreous imaging mode, a macular imaging mode, an optic disc imaging mode, an OCTA imaging mode, and the like. Also, the scan pattern may include 3D scan, radial scan, cross scan, circle scan, raster scan, Lissajous scan (scan along a Lissajous curve), and the like. In the OCTA imaging mode, the imaging control unit 101-2 controls the above-described scanning unit so that the same region (same position) of the subject's eye is scanned multiple times with the measurement light. Even in the OCTA imaging mode, for example, raster scan, radial scan, cross scan, circle scan, Lissajous scan, etc. can be set as scan patterns. Also, with respect to a trained model that uses tomographic images as input data, it is possible to perform learning using tomographic images corresponding to cross sections in different directions as learning data. For example, learning may be performed using a tomographic image of a cross section in the xz direction, a tomographic image of a cross section in the yz direction, or the like as learning data.

It should be noted that the necessity of execution of image quality enhancement processing by the image quality enhancement model (or display of a high quality image obtained by image quality enhancement processing) is determined by the operator using the image quality enhancement button provided on the display screen. It may be performed in accordance with an instruction, or may be performed in accordance with settings stored in advance in storage unit 101-3. Note that the image quality improvement process using the trained model (image quality improvement model) may be displayed in the active state of the image quality improvement button, etc., or may be displayed as a message on the display screen to that effect. good. Further, the execution state of the image quality improvement process may be maintained in the execution state at the time of the previous activation of the ophthalmologic apparatus, or may be maintained in the execution state at the time of the previous examination for each subject.

Further, moving images to which various learned models such as high-quality models can be applied are not limited to live moving images, and may be, for example, moving images stored (saved) in storage unit 101-3. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving images of the fundus stored (saved) in the storage unit 101-3 may be displayed on the display screen. For example, when the vitreous body is desired to be properly observed, first, a reference frame may be selected based on conditions such as the presence of as much vitreous body as possible on the frame. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, high-quality images (high-quality frames) sequentially generated by the high-quality image engine for each at least one frame of the moving image may be continuously displayed.

As a method of aligning between frames described above, the same method may be applied to the method of aligning in the X direction and the method of aligning in the Z direction (depth direction), or different methods may be used. may be applied. Also, the alignment in the same direction may be performed multiple times by different techniques. For example, fine alignment may be performed after performing rough alignment. Alignment methods include, for example, alignment using a retinal layer boundary obtained by segmentation processing of a tomographic image (B-scan image) (rough in the Z direction), and a plurality of images obtained by dividing a tomographic image. Alignment (precise in the X direction and Z direction) using correlation information (similarity) between the region and the reference image, and one-dimensional projection image generated for each tomographic image (B scan image) (X direction ) alignment, alignment (in the X direction) using a two-dimensional front image, and the like. Also, it may be configured such that after rough alignment is performed in units of pixels, fine alignment is performed in units of sub-pixels.

Further, the high image quality model may be updated by additional learning using the value of the ratio set (changed) according to the instruction from the examiner as the learning data. For example, when the input image is relatively dark, if the examiner tends to set a high ratio of the input image to the high-quality image, the learned model undergoes additional learning so as to achieve such a tendency. As a result, for example, it can be customized as a trained model that can obtain a combination ratio that suits the examiner's taste. At this time, a button may be displayed on the display screen for determining whether or not to use the set (changed) ratio value as learning data for additional learning in accordance with an instruction from the examiner. Alternatively, the ratio determined using the trained model may be set as the default value, and thereafter the ratio value may be changed from the default value in accordance with an instruction from the examiner. Also, the high image quality model may be a trained model obtained by additionally learning learning data including at least one high quality image generated using the high image quality model. At this time, whether or not to use the high-quality image as learning data for additional learning may be selectable by an instruction from the examiner.

[Modification 4]
The image feature acquisition unit 101-44, the extraction unit 101-462, and the region detection unit 302 may generate label images using a trained model for image segmentation and perform image segmentation processing. Here, the label image means a label image in which a region label is attached to each pixel of the tomographic image. Specifically, it is an image in which arbitrary regions are divided by a group of identifiable pixel values (hereinafter referred to as label values) from among the regions drawn in the acquired image. Here, the specified arbitrary region includes a region of interest, a volume of interest (VOI), and the like.

By specifying a coordinate group of pixels having arbitrary label values from the image, it is possible to specify a coordinate group of pixels that render a corresponding region such as a retinal layer in the image. Specifically, for example, when the label value indicating the ganglion cell layer that constitutes the retina is 1, a coordinate group having a pixel value of 1 among the pixel groups of the image is specified, and the coordinates corresponding to the coordinate group are identified from the image. Extract the pixel group that Thereby, the region of the ganglion cell layer in the image can be specified.

Note that the image segmentation process may include a process of reducing or enlarging the label image. At this time, the image interpolation processing method used to reduce or enlarge the label image shall use the nearest neighbor method, etc., so as not to erroneously generate an undefined label value or a label value that should not exist at the corresponding coordinates. .

The image segmentation process is a process of specifying a region called ROI (Region Of Interest) or VOI, such as an organ or lesion depicted in an image, for use in image diagnosis or image analysis. For example, according to the image segmentation process, it is possible to specify a region group of a group of layers forming the retina from an image obtained by OCT imaging of the posterior segment of the eye. Note that the number of specified regions is 0 if the region to be specified is not rendered in the image. Also, if a plurality of region groups to be specified are drawn in the image, the number of specified regions may be plural, or even if there is only one region surrounding the region group. good.

The identified region group is output as information that can be used in other processing. Specifically, for example, it is possible to output a group of coordinates of a group of pixels forming each of the identified region groups as a group of numerical data. Further, for example, a group of coordinates indicating a rectangular area, an elliptical area, a rectangular parallelepiped area, an ellipsoidal area, etc., including each of the identified area groups can be output as a numerical data group. Furthermore, for example, it is possible to output a group of coordinates indicating a straight line, a curve, a plane, a curved surface, or the like, which is the boundary of the specified area group, as a numerical data group. Also, for example, it is possible to output a label image indicating the identified region group.

Here, for example, a convolutional neural network (CNN) can be used as a machine learning model for image segmentation. It should be noted that the configuration of the CNN used in this modification is the function of an encoder consisting of multiple layers including multiple downsampling layers, and the function of a decoder consisting of multiple layers including multiple upsampling layers. It can be a machine learning model of type. In the U-net type machine learning model, position information (spatial information) obscured in multiple layers configured as encoders is converted to the same dimensional layers (mutually corresponding layers) in multiple layers configured as decoders. ) (eg, using a skip connection).

Further, as a modification of the configuration of the CNN, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolutional layer. Through these steps of CNN, the features of the captured image can be extracted.

As the machine learning model according to this modification, for example, CNN (U-net type machine learning model), a model combining CNN and LSTM, FCN (Fully Convolutional Network), or SegNet can be used. . A machine learning model or the like that performs object recognition can also be used depending on the desired configuration. As a machine learning model for object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, it is also possible to use a machine learning model that performs object recognition on a region-by-region basis. YOLO (You Only Look Once) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used as a machine learning model that performs object recognition on a region-by-region basis.

In addition, learning data for a machine learning model for image segmentation uses a tomographic image obtained by OCT as input data, and outputs a labeled image in which a region is labeled for each pixel of the tomographic image. Label images include, for example, the inner limiting membrane (ILM), nerve fiber layer (NFL), ganglion cell layer (GCL), photoreceptor inner segment outer segment junction (ISOS), retinal pigment epithelium layer (RPE), Bruch Label images with labels such as membrane (BM) and choroid can be used. Other regions include, for example, vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner plexiform layer (IPL), inner nuclear layer (INL), cornea, anterior chamber, iris, and labeled images such as lenses may be used.

Input data for a machine learning model for image segmentation is not limited to tomographic images. It may be an anterior eye image, an SLO image, an OCTA image, or the like. In this case, the learning data can use various images as input data and output data as label images in which each pixel of each image is labeled with a region name or the like. For example, if the input data of the learning data are SLO images, the output data may be images labeled with the periphery of the optic disc, Disc, Cup, and the like.

The label image used as the output data may be an image in which each region in the tomographic image is labeled by a doctor or the like, or an image in which each region is labeled by rule-based region detection processing. There may be. However, if machine learning is performed using labeled images that have not been properly labeled as output data for training data, images obtained using a trained model that has been trained using the training data will also be properly labeled. There is a possibility that it will be a label image that has not been Therefore, by removing pairs including such label images from the training data, it is possible to reduce the possibility that inappropriate label images are generated using the trained model. Here, the rule-based area detection processing refers to detection processing using known regularity such as the regularity of the shape of the retina.

The image feature acquisition unit 101-44, the extraction unit 101-462, and the area detection unit 302 perform image segmentation processing using such a trained model for image segmentation, thereby identifying specific areas of various images. High-speed and accurate detection can be expected. A trained model for image segmentation may also be prepared for each type of image that is input data. Also, for the OCTA front image and the En-Face image, a trained model may be prepared for each depth range for generating the image. Furthermore, the trained model for image segmentation may be one that has been trained on images for each imaging region (for example, the center of the macula, the center of the optic papilla), or one that has been trained regardless of the imaging region. may

In addition, with respect to the trained model for image segmentation, additional learning may be performed using data manually corrected in accordance with an operator's instruction as learning data. In addition, determination of necessity of additional learning and determination of whether or not to transmit data to the server may be made in the same manner. In these cases as well, it is expected that the accuracy of each process can be improved, and that processes can be performed according to the tendency of the examiner's preferences.

Furthermore, when the image processing apparatus 101 detects a partial region (for example, a region of interest, an artifact region, an abnormal region, etc.) of the subject's eye 200 using a trained model, the image processing apparatus 101 obtains a predetermined image for each detected partial region. It can also be processed. As an example, a case of detecting at least two partial regions of the vitreous body region, the retinal region, and the choroidal region will be described. In this case, when image processing such as contrast adjustment is performed on at least two detected partial regions, by using different image processing parameters, adjustment suitable for each region can be performed. By displaying an image adjusted appropriately for each region, the operator can more appropriately diagnose a disease or the like for each partial region. Note that the configuration using different image processing parameters for each detected partial region is similarly applied to the partial region of the eye 200 to be examined that is obtained by detecting the partial region of the eye 200 to be examined without using the learned model. may

[Modification 5]
The display control unit 101-5 in the various embodiments and modifications described above may display analysis results such as desired layer thicknesses and various blood vessel densities on the report screen of the display screen after tomographic imaging. . Also, optic nerve head, macula, blood vessel area, capillary area, artery area, vein area, nerve fiber bundle, vitreous area, macular area, choroid area, scleral area, cribriform plate area, retinal layer boundary, retina Values (distribution) of parameters related to the site of interest including at least one of layer boundary edge, photoreceptor, blood cell, blood vessel wall, blood vessel inner wall boundary, blood vessel outer boundary, ganglion cell, corneal area, angle area, Schlemm's canal, etc. may be displayed as the analysis result. Here, the site of interest may be, for example, a vortex vein or the like, which is an outflow port of a blood vessel in the Haller layer (an example of a blood vessel in a partial depth range of the choroidal region) to the outside of the eye. At this time, the parameters related to the region of interest include, for example, the number of vortex veins (for example, the number for each region), the distance from the optic disc to each vortex vein, and the angle at which each vortex vein is positioned around the optic disc. may be As a result, for example, various diseases related to Pachychoroid (thickened choroid) (for example, choroidal neovascular disease) can be accurately diagnosed. Further, for example, by analyzing medical images to which various types of artifact reduction processing have been applied, the various analysis results described above can be displayed as highly accurate analysis results. Artifacts include, for example, false image areas caused by light absorption by blood vessel areas, projection artifacts, and strip-shaped artifacts in the front image that occur in the main scanning direction of the measurement light due to the state of the subject's eye (movement, blinking, etc.). There may be. Also, the artifact may be anything, for example, as long as it is an image failure area that occurs randomly in each imaging on a medical image of a predetermined region of the subject. In addition, the display control unit 101-5 may cause the output unit 103 to display the values (distribution) of the parameters regarding the area including at least one of the various artifacts (imaging area) as described above as the analysis result. Also, parameter values (distribution) relating to an area including at least one of drusen, new blood vessels, vitiligo (hard vitiligo), and abnormal sites such as pseudodrusen may be displayed as the analysis result. In addition, comparison results obtained by comparing standard values and standard ranges obtained using a standard database with analysis results may be displayed.

Also, the analysis result may be displayed as an analysis map, a sector indicating a statistical value corresponding to each divided area, or the like. Note that the analysis results may be generated using a trained model (analysis result generation engine, trained model for generating analysis results) obtained by learning analysis results of medical images as learning data. . At this time, the trained model is learned using learning data including medical images and analysis results of the medical images, or learning data including medical images and analysis results of medical images of a different type from the medical images. It may be obtained by

Also, the learning data for image analysis may include labeled images generated using a trained model for image segmentation processing and analysis results of medical images using the labeled images. In this case, the image processing apparatus 101 can function as an example of an analysis result generation unit that generates an analysis result of a tomographic image from the result of image segmentation processing using a trained model for generating analysis results, for example. can. Furthermore, the trained model is training data including input data that is a set of a plurality of different types of medical images of a predetermined site, such as brightness En-Face images and motion contrast frontal images (OCTA En-Face images). may be obtained by learning using

Further, it may be configured to display analysis results obtained using a high-quality image generated using a high-quality image model. In this case, the input data included in the learning data may be a high-quality image generated using a trained model for improving image quality, or a set of a low-quality image and a high-quality image. good too. Note that the learning data may be an image obtained by manually or automatically correcting at least a part of an image whose image quality has been improved using a trained model.

In addition, the learning data includes, for example, an analysis value obtained by analyzing the analysis area (e.g., average value, median value, etc.), a table containing the analysis value, an analysis map, the position of the analysis area such as a sector in the image, etc. It may be data obtained by labeling (annotating) input data with information including one as correct data (for supervised learning). Note that the analysis results obtained using the learned model for generating analysis results may be displayed according to instructions from the operator.

Further, the display control unit 101-5 in the above-described embodiment and modification may display various diagnostic results such as diabetic retinopathy, glaucoma, age-related macular degeneration, etc. on the report screen of the display screen. At this time, for example, by analyzing a medical image to which various artifact reduction processes as described above have been applied, it is possible to display a highly accurate diagnosis result. Further, the diagnosis result may display the position of the identified abnormal site or the like on an image, or may display the state of the abnormal site or the like in characters or the like. Further, classification results (for example, Curtin classification) such as abnormal sites may be displayed as diagnosis results. Further, as the classification result, for example, information indicating the likelihood of each abnormal site (for example, a numerical value indicating a ratio) may be displayed. Information necessary for the doctor to confirm the diagnosis may also be displayed as the diagnosis result. As the necessary information, for example, advice such as additional photographing can be considered. For example, when an abnormal site is detected in a blood vessel region in an OCTA image, it may be displayed that fluorescence imaging using a contrast agent that enables observation of blood vessels in more detail than OCTA is additionally performed. Further, the diagnosis result may be information related to the subject's future medical treatment policy and the like. In addition, the diagnosis results include, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, findings (interpretation findings, etc.), the basis for the diagnosis name (positive (negative medical support information, etc.) and grounds for denying the diagnosis (negative medical support information). At this time, for example, a diagnosis result that is more likely than the diagnosis result such as a diagnosis name input in response to an instruction from the examiner may be displayed as the medical support information. In addition, when a plurality of types of medical images are used, for example, the types of medical images that can serve as the basis for the diagnosis result may be displayed in an identifiable manner. In addition, as the basis for the diagnosis result, a map (attention map, activation map) that visualizes the feature amount extracted by the trained model, for example, a color map (heat map) that shows the feature amount in color good. At this time, for example, a heat map may be superimposed on the medical image used as the input data. The heat map is, for example, Grad-CAM (Gradient-weighted Class Activation Mapping) or Guided Grad - can be obtained using CAM or the like;

The diagnosis result may be generated using a trained model (a diagnosis result generating engine, a trained model for generating diagnosis results) obtained by learning the diagnosis results of medical images as learning data. . In addition, the trained model is obtained by learning using learning data including medical images and diagnostic results of the medical images, learning data including medical images and diagnostic results of medical images of a different type from the medical images, and the like. It may be obtained.

Also, the learning data may include labeled images generated using a region recognition engine or a trained model for segmentation processing, and diagnostic results of medical images using the labeled images. In this case, the image processing apparatus 101 can function as an example of a diagnostic result generation unit that generates a diagnostic result of a tomographic image from the result of image segmentation processing using a trained model for generating a diagnostic result. can.

Furthermore, it may be configured to display the diagnosis result obtained using the high-quality image generated using the high-quality image engine. In this case, the input data included in the learning data may be a high-quality image generated using a high-quality image engine, or may be a set of a low-quality image and a high-quality image. Note that the learning data may be an image obtained by manually or automatically correcting at least a part of an image whose image quality has been improved using a trained model.

In addition, the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, findings (interpretation findings, etc.), the basis for the diagnosis name (positive The input data is labeled (annotated) as correct data (supervised learning) that includes at least one of (such as medical support information) and grounds for denying the diagnosis (negative medical support information). Data may be used. Note that the diagnostic results obtained using the learned model for generating diagnostic results may be displayed according to instructions from the examiner.

Note that a trained model may be prepared for each piece of information used as input data or for each type of information, and a diagnosis result may be acquired using the trained model. In this case, the information output from each trained model may be statistically processed to determine the final diagnostic result. For example, the ratio of information output from each trained model may be added for each type of information, and information with a higher total ratio than other information may be determined as the final diagnosis result. Statistical processing is not limited to calculation of the total, and may be calculation of an average value, a median value, or the like. Further, for example, among the information output from each trained model, information with a higher percentage than other information (information with the highest percentage) may be used to determine the diagnosis result. Similarly, out of the information output from each trained model, the information of the ratio of the threshold value or more may be used to determine the diagnostic result.

Further, it may be possible to determine (approve) whether the determined diagnosis result is good or bad according to the operator's instruction (selection). Further, the diagnosis result may be determined from the information output from each learned model according to the instruction (selection) of the operator. At this time, for example, the display control unit 101-5 may cause the output unit 103 to display the information output from each trained model and the ratio thereof side by side. The selected information may be determined as the diagnosis result by the operator selecting information with a higher percentage than other information, for example. Furthermore, a diagnosis result may be determined using a machine learning model from the information output from each trained model. In this case, the machine learning algorithm may be a different type of machine learning algorithm than the machine learning algorithm used to generate the diagnosis, such as neural networks, support vector machines, Adaboost, Bayesian networks, or random Forrest or the like may be used.

The learning of the various trained models described above may be not only supervised learning (learning using labeled learning data) but also semi-supervised learning. In semi-supervised learning, for example, after multiple discriminators (classifiers) perform supervised learning, they identify (classify) unlabeled learning data, and according to the reliability of the classification result (classification result) This is a method of automatically labeling (annotating) (for example, identification results whose certainty is greater than a threshold) and performing learning using the labeled learning data. Semi-supervised learning may be, for example, Co-Training (or Multiview). At this time, the trained model for generating the diagnosis result uses, for example, a first classifier that identifies a medical image of a normal subject and a second classifier that identifies a medical image containing a specific lesion. It may also be a trained model obtained by semi-supervised learning (eg, co-training). It should be noted that the purpose is not limited to diagnosis, and may be, for example, an imaging support or the like. In this case, the second discriminator may, for example, discriminate a medical image including a partial region such as a region of interest or an artifact region.

Further, the display control unit 101-5 according to the various embodiments and modifications described above displays the object recognition result ( object detection results) or segmentation results may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, a color or the like may be superimposed on the object in the image and displayed. The object recognition results and segmentation results are the trained models (object recognition engine, object recognition engine) obtained by learning learning data in which medical images are labeled (annotated) with information indicating object recognition and segmentation as correct data. A trained model, a segmentation engine, a trained model for segmentation) may be used. Note that the analysis result generation and diagnosis result generation described above may be obtained by using the object recognition result and segmentation result described above. For example, analysis result generation and diagnosis result generation processing may be performed on a region of interest obtained by object recognition or segmentation processing.

When detecting an abnormal site, the image processing apparatus 101 may use a generative adversarial network (GAN) or a variational auto-encoder (VAE). For example, a DCGAN (Deep Convolutional GAN) consisting of a generator obtained by learning to generate medical images and a discriminator obtained by learning to discriminate between new medical images generated by the generator and real medical images. can be used as a machine learning model.

In the case of using DCGAN, for example, the discriminator encodes an input medical image into a latent variable, and the generator generates a new medical image based on the latent variable. After that, the difference between the input medical image and the generated new medical image can be extracted (detected) as an abnormal site. When VAE is used, for example, an input medical image is encoded by an encoder to generate a latent variable, and a decoder decodes the latent variable to generate a new medical image. After that, the difference between the input medical image and the generated new medical image can be extracted as an abnormal site.

Furthermore, the image processing device 101 may detect an abnormal site using a convolutional auto-encoder (CAE). When CAE is used, the same medical image is learned as input data and output data during learning. As a result, when a medical image with an abnormal portion is input to CAE at the time of estimation, a medical image without an abnormal portion is output according to the tendency of learning. After that, the difference between the medical image input to CAE and the medical image output from CAE can be extracted as an abnormal site.

In these cases, the image processing apparatus 101 converts information about the difference between the medical image obtained using the hostile generation network or the autoencoder and the medical image input to the hostile generation network or the autoencoder into information about the abnormal site. can be generated as As a result, the image processing apparatus 101 can be expected to detect an abnormal site at high speed and with high accuracy. For example, even if it is difficult to collect a large number of medical images containing abnormal regions as learning data in order to improve the detection accuracy of abnormal regions, medical images of normal subjects, which are relatively easy to collect, can be used as learning data. can be used. For this reason, for example, learning for accurately detecting an abnormal site can be performed efficiently. Here, autoencoders include VAE, CAE, and the like. Also, at least a part of the generation unit of the adversarial generation network may be composed of VAEs. As a result, for example, a relatively clear image can be generated while reducing the phenomenon of generating similar data. For example, the image processing apparatus 101 obtains information about the difference between a medical image obtained from various medical images using an adversarial generation network or an autoencoder and a medical image input to the adversarial generation network or the autoencoder. , can be generated as information about the abnormal site. Further, for example, the display control unit 101-5 controls the display of a medical image obtained from various medical images using a hostile generation network or an autoencoder and a medical image input to the hostile generation network or the autoencoder. Information about the difference can be displayed on the output unit 103 as information about the abnormal site.

In addition, the trained model for generating diagnostic results in particular may be a trained model obtained by learning using learning data including input data that is a set of a plurality of different types of medical images of a predetermined region of a subject. good. At this time, the input data included in the learning data may be, for example, input data that is a set of a motion contrast front image and a luminance front image (or a luminance tomographic image) of the fundus. As input data included in the learning data, for example, input data such as a set of a fundus tomographic image (B-scan image) and a color fundus image (or a fluorescent fundus image) can be considered. Moreover, the multiple medical images of different types may be acquired by different modalities, different optical systems, or different principles.

In addition, the trained model for diagnosis result generation in particular may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of different parts of the subject. At this time, as the input data included in the learning data, for example, input data that is a set of a tomographic image (B-scan image) of the fundus and a tomographic image (B-scan image) of the anterior segment can be considered. In addition, as input data included in the learning data, for example, input data such as a set of a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic papilla of the fundus. is also conceivable.

The input data included in the learning data may be a plurality of medical images of different regions and different types of the subject. At this time, the input data included in the learning data may be, for example, input data that is a set of a tomographic image of the anterior segment and a color fundus image. Further, the above-described trained model may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of a predetermined part of the subject with different imaging angles of view. The input data included in the learning data may be obtained by pasting together a plurality of medical images obtained by time-dividing a predetermined site into a plurality of regions, such as a panorama image. At this time, by using a wide-angle image such as a panoramic image as training data, it is possible to acquire the feature amount of the image with high accuracy because the amount of information is larger than that of a narrow-angle image. results can be improved. Also, the input data included in the learning data may be input data that is a set of a plurality of medical images of a predetermined part of the subject taken on different dates.

Further, the display screen on which at least one of the above-described analysis result, diagnosis result, object recognition result, and segmentation result is displayed is not limited to the report screen. Such a display screen is, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). may be displayed in For example, by displaying at least one result obtained using the above-described learned model on the photographing confirmation screen, the operator can confirm a highly accurate result even immediately after photographing.

Further, for example, when a specific object is recognized, a frame surrounding the recognized object may be superimposed on the live moving image. At this time, if the information indicating the certainty of the object recognition result (for example, a numerical value indicating the ratio) exceeds the threshold value, the color of the frame surrounding the object is changed, for example. good. This allows the examiner to easily identify the object on the live video.

In addition, a trained model for correct data generation for generating correct data such as labeling (annotation) may be used to generate correct data used for learning of the various trained models described above. At this time, the trained model for correct data generation may be obtained by (sequentially) additionally learning the correct data obtained by labeling (annotating) by the examiner. In other words, the trained model for correct data generation may be obtained by additionally learning learning data in which data before labeling is used as input data and data after labeling is used as output data. In addition, in a plurality of continuous frames such as a moving image, it is configured to correct the result of a frame determined to have low accuracy in consideration of the results of object recognition, segmentation, etc. of the preceding and succeeding frames. good too. At this time, the corrected results may be used as correct data for additional learning in accordance with instructions from the examiner. In addition, for example, for a medical image with a low result accuracy, the examiner can visualize the feature extracted by the trained model on the medical image (attention map, activation map). may be configured such that additional learning is performed using labeled (annotated) images as input data while confirming a color map (heat map) showing in color. For example, in the heat map on the layer immediately before outputting the results of the trained model, if the points of interest differ from the intention of the examiner, label (annotate) the points that the examiner thinks should be noted. Additional learning may be performed on the medical images that have been acquired. As a result, for example, the trained model prioritizes the feature amount of a partial region that is a partial region on a medical image and has a relatively large effect on the output result of the trained model over other regions ( weighted) can be additionally learned.

Here, the various trained models described above can be obtained by machine learning using learning data. Machine learning includes, for example, deep learning consisting of multilevel neural networks. Also, for example, a convolutional neural network can be used for at least part of the multi-layered neural network. Also, at least a part of the multi-layered neural network may employ a technology related to an autoencoder. Also, a technique related to back propagation (error backpropagation method) may be used for learning. Also, for learning, a method (dropout) of randomly inactivating each unit (each neuron or each node) may be used. Also, for learning, a method (batch normalization) of normalizing data transmitted to each layer of a multi-layer neural network before an activation function (for example, ReLu function) is applied may be used. However, machine learning is not limited to deep learning, and any learning using a model capable of extracting (expressing) feature amounts of learning data such as images by learning can be used. Here, the machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. Also, a trained model is a model that has been trained (learned) in advance using appropriate learning data for a machine learning model based on an arbitrary machine learning algorithm. However, it is assumed that the trained model is not one that does not perform further learning, and that additional learning can be performed. Also, learning data is composed of a pair of input data and output data (correct data). Here, learning data may be referred to as teacher data, or correct data may be referred to as teacher data.

Note that the GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing using a GPU. Therefore, in this modification, the GPU is used in addition to the CPU for processing by the image processing apparatus 101, which is an example of a learning unit (not shown). Specifically, when a learning program including a learning model is executed, the CPU and the GPU cooperate to perform calculations for learning. Note that the processing of the learning unit may be performed by only the CPU or GPU. Also, a processing unit (estimating unit) that executes processing using various learned models described above may also use a GPU, like the learning unit. Also, the learning unit may include an error detection unit and an updating unit (not shown). The error detection unit obtains an error between correct data and output data output from the output layer of the neural network according to input data input to the input layer. The error detector may use a loss function to calculate the error between the output data from the neural network and the correct data. Also, the updating unit updates the weighting coefficients for coupling between nodes of the neural network based on the error obtained by the error detecting unit so as to reduce the error. This updating unit updates the connection weighting coefficients and the like using, for example, the error backpropagation method. The error backpropagation method is a method of adjusting the connection weighting coefficients and the like between nodes of each neural network so as to reduce the above error.

In addition, the machine learning model used for object recognition, segmentation, high image quality, etc. described above includes an encoder function consisting of multiple layers including multiple downsampling layers, and multiple layers including multiple upsampling layers. A U-net type machine learning model having a decoder function consisting of is applicable. In the U-net type machine learning model, position information (spatial information) obscured in multiple layers configured as encoders is converted to the same dimensional layers (mutually corresponding layers) in multiple layers configured as decoders. ) (eg, using a skip connection).

As a machine learning model used for object recognition, segmentation, image quality improvement, and the like, for example, FCN (Fully Convolutional Network), SegNet, or the like can be used. Also, a machine learning model that performs object recognition on a region-by-region basis according to a desired configuration may be used. As a machine learning model for object recognition, RCNN (Region CNN), fastRCNN, or fasterRCNN, for example, can be used. Furthermore, YOLO (You Only Look Once) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used as a machine learning model for recognizing objects in units of regions.

Also, the machine learning model may be, for example, a Capsule Network (CapsNet). Here, in a general neural network, each unit (each neuron or each node) is configured to output a scalar value, for example, the spatial positional relationship (relative position) between features in an image is configured to reduce spatial information about As a result, for example, learning can be performed in which the effects of local distortion, translation, and the like of an image are reduced. On the other hand, in a capsule network, each unit (each capsule) is configured to output spatial information as a vector, thereby retaining spatial information. As a result, for example, learning can be performed in consideration of the spatial positional relationship between features in the image.

[Modification 6]
The preview screens in the various embodiments and modifications described above may be configured so that the various learned models described above are used for at least one frame of the live moving image. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the learned model corresponding to each live moving image may be used. As a result, the processing time can be shortened even for a live moving image, for example, so that the examiner can obtain highly accurate information before the start of imaging. For this reason, for example, failures in re-imaging can be reduced, so that accuracy and efficiency of diagnosis can be improved.

Note that the plurality of live moving images may be, for example, a moving image of the anterior segment for alignment in the XYZ directions, and a front moving image of the fundus for focus adjustment of the fundus oculi observation optical system and OCT focus adjustment. Also, the plurality of live moving images may be, for example, tomographic moving images of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length). When such a preview image is displayed, the above-described various adjustments are performed so that the region detected using the above-described trained model for object recognition and the trained model for segmentation satisfies predetermined conditions. The image processing apparatus 101 may be configured as follows. For example, a value (e.g., contrast value or intensity value) related to a predetermined retinal layer such as a vitreous region or RPE detected using a trained model for object recognition or a trained model for segmentation exceeds a threshold ( Alternatively, it may be configured such that various adjustments such as OCT focus adjustment are performed so as to achieve a peak value. Further, for example, the OCT is performed so that a predetermined retinal layer such as the vitreous region and the RPE detected using a trained model for object recognition or a trained model for segmentation is positioned at a predetermined position in the depth direction. Coherence gate adjustments may be configured to occur.

In these cases, the image processing apparatus 101 can generate a high-quality moving image by performing image quality enhancement processing on the moving image using the learned model. Further, the imaging control unit 101-2 moves the reference mirror 221 so that the partial region such as the region of interest obtained by the segmentation process or the like is positioned at a predetermined position in the display region while the high-quality moving image is being displayed. An optical member for changing the imaging range can be driven and controlled. In such a case, the imaging control unit 101-2 can automatically perform alignment processing so that the desired area is at a predetermined position in the display area based on highly accurate information. The optical member that changes the imaging range may be, for example, an optical member that adjusts the coherence gate position, and more specifically, the reference mirror 221 that reflects the reference light. Also, the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measurement optical path length and the reference optical path length. , or the like. Note that the optical member that changes the imaging range may be, for example, the stage section 100-2. In addition, the imaging control unit 101-2 performs scanning so that a partial area such as an artifact area obtained by segmentation processing or the like is re-captured (rescanned) during or at the end of imaging in response to an instruction regarding the start of imaging. You may drive-control a means. Further, for example, when the information indicating the certainty of the object recognition result (for example, the numerical value indicating the ratio) regarding the target site exceeds the threshold value, various adjustments and the start of imaging may be automatically performed. good. Also, for example, when the information indicating the certainty of the object recognition result (for example, the numerical value indicating the ratio) regarding the target part exceeds the threshold, each adjustment and the start of imaging can be executed according to the instructions from the examiner. It may be configured to change to a normal state (cancel the execution prohibition state).

Further, moving images to which various learned models described above can be applied are not limited to live moving images, and may be, for example, moving images stored (saved) in storage unit 101-3. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving images of the fundus stored (saved) in the storage unit 101-3 may be displayed on the display screen. For example, when the vitreous body is desired to be properly observed, first, a reference frame may be selected based on conditions such as the presence of as much vitreous body as possible on the frame. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, high-quality images (high-quality frames) sequentially generated by a trained model for improving image quality may be continuously displayed for each at least one frame of a moving image.

As a method for aligning between frames described above, the same method may be applied to the method of aligning in the X direction and the method of aligning in the Z direction (depth direction), or different methods may be used. may be applied. Also, the alignment in the same direction may be performed multiple times by different techniques. For example, fine alignment may be performed after performing rough alignment. Alignment techniques include, for example, alignment using a retinal layer boundary obtained by segmentation processing of a tomographic image (B-scan image) (rough in the Z direction), and a plurality of images obtained by dividing a tomographic image. Alignment (precise in the X and Z directions) using correlation information (similarity) between the region and the reference image, and one-dimensional projection images generated for each tomographic image (B scan image) (X direction ) alignment, alignment (in the X direction) using a two-dimensional front image, and the like. Also, it may be configured such that after rough alignment is performed in units of pixels, fine alignment is performed in units of sub-pixels.

Here, during various adjustments, there is a possibility that the object to be imaged, such as the retina of the subject's eye, has not yet been successfully imaged. For this reason, there is a possibility that a high-quality image cannot be obtained with high accuracy due to the large difference between the medical image input to the trained model and the medical image used as learning data. Therefore, when an evaluation value such as image quality evaluation of a tomographic image (B scan) exceeds a threshold value, display of high-quality moving images (continuous display of high-quality frames) may be automatically started. Further, when an evaluation value such as image quality evaluation of a tomographic image (B-scan) exceeds a threshold value, the image quality improvement button may be changed to a state (active state) in which the examiner can designate.

In addition, for example, a different trained model for improving image quality is prepared for each imaging mode with a different scan pattern, etc., and the trained model for improving image quality corresponding to the selected imaging mode is selected. may be Alternatively, one trained model for improving image quality obtained by learning learning data including various medical images obtained in different imaging modes may be used.

[Modification 7]
In the above-described embodiments and modifications, when various trained models are undergoing additional learning, it may be difficult to output (estimate/predict) using the trained models themselves that are undergoing additional learning. be. Therefore, it is preferable to prohibit the input of medical images other than learning data to a trained model during additional learning. Also, another trained model that is the same as the trained model before execution of additional learning may be prepared as a backup trained model. At this time, it is preferable to be configured so that input of medical images other than learning data to the preliminary trained model can be executed during execution of additional learning. Then, after the additional learning is completed, the trained model after the execution of the additional learning is evaluated, and if there is no problem, the spare trained model can be replaced with the trained model after the execution of the additional learning. Also, if there is a problem, a backup trained model may be used.

As for the evaluation of the trained model after execution of additional learning, for example, a trained model for classification is used to classify high-quality images obtained by the trained model for high-quality images from other types of images. may be used. For the trained model for classification, for example, multiple images including high-quality images and low-quality images obtained by the trained model for high image quality are input data, and the types of these images are labeled (annotated). It may be a trained model obtained by learning learning data that includes the obtained data as correct data. At this time, the type of image of the input data at the time of estimation (prediction) is displayed together with information indicating the likelihood of each type of image included in the correct data at the time of learning (for example, a numerical value indicating the ratio). good too. As input data for the trained model for classification, in addition to the above images, superimposition processing of multiple low-quality images (for example, averaging processing of multiple low-quality images obtained by alignment), etc. A high-quality image that has undergone high-contrast, noise reduction, or the like may be included. In addition, for evaluation of the trained model after execution of additional learning, for example, the trained model after execution of additional learning and the trained model before execution of additional learning (preliminary trained model) are used respectively. A plurality of high-quality images obtained from the images may be compared, or analysis results of the plurality of high-quality images may be compared. At this time, for example, a comparison result of the plurality of high-quality images (an example of change due to additional learning) or a comparison result of the analysis results of the plurality of high-quality images (an example of change due to additional learning) is within a predetermined range. It may be determined whether or not, and the determination result may be displayed.

Also, a learned model obtained by learning for each imaging region may be selectively used. Specifically, a first trained model obtained using learning data including a first imaging region (for example, an anterior segment, a posterior segment, etc.) and a second model different from the first imaging region. It is possible to prepare a plurality of trained models, including a second trained model obtained using learning data including the imaging part. The image processing apparatus 101 may have selection means for selecting one of these learned models. At this time, the image processing apparatus 101 may have control means for performing additional learning on the selected trained model. In response to an instruction from the examiner, the control means searches for data paired with an imaging region corresponding to the selected learned model and a photographed image of the imaging region, and uses the retrieved data as learning data. can be performed as additional learning on the selected trained model. Note that the imaged region corresponding to the selected learned model may be obtained from information in the header of the data or manually input by the examiner. Also, data retrieval may be performed via a network from, for example, a server of an external facility such as a hospital or research institute. As a result, additional learning can be efficiently performed for each imaging part using the photographed image of the imaging part corresponding to the learned model.

Note that the selection means and the control means may be configured by a software module executed by a processor such as the CPU or MPU of the image processing apparatus 101 . Also, the selection means and the control means may be configured by a circuit such as an ASIC that performs a specific function, an independent device, or the like.

In addition, when acquiring learning data for additional learning from a server of an external facility such as a hospital or research institute via a network, it is necessary to reduce reliability deterioration due to falsification and system troubles during additional learning. is useful. Therefore, the correctness of the learning data for additional learning may be detected by confirming the matching by digital signature or hashing. Thereby, learning data for additional learning can be protected. At this time, if the validity of the learning data for additional learning cannot be detected as a result of confirming the match by digital signature or hashing, a warning to that effect is issued and additional learning is performed using the learning data. Make it not exist. It should be noted that the server may take any form such as a cloud server, a fog server, an edge server, etc., regardless of its installation location. In addition, when configuring a network within a facility, within a site that includes a facility, within an area that includes multiple facilities, etc., for wireless communication, for example, Reliability of the network may be improved by configuring to use radio waves of a dedicated wavelength band. Alternatively, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and multiple simultaneous connections.

Moreover, the protection of data by confirming consistency as described above is applicable not only to learning data for additional learning but also to data including medical images. In addition, the image management system may be configured such that transactions of data including medical images between servers at multiple facilities are managed by a distributed network. In addition, the image management system may be configured to chronologically connect a plurality of blocks in which the transaction history and the hash value of the previous block are recorded together. As a technology for confirming consistency, cryptography that is difficult to calculate even using a quantum computer such as a quantum gate system (e.g., lattice cryptography, quantum cryptography with quantum key distribution, etc.) may be used. good. Here, the image management system may be a device or system that receives and stores an image captured by an image capturing device or an image that has undergone image processing. In addition, the image management system can transmit images in response to requests from connected devices, perform image processing on stored images, and request other devices to perform image processing. can. The image management system can include, for example, a picture archival communication system (PACS). The image management system also includes a database capable of storing various types of information such as subject information and imaging time associated with the received images. Also, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with saved images in response to requests from other devices. .

When performing additional learning on various trained models, the GPU can be used to perform high-speed processing. GPUs can perform efficient calculations by processing more data in parallel, so when learning models such as deep learning are used for multiple times, GPUs can be used for processing. It is valid. Note that the additional learning process may be performed in cooperation with the GPU and the CPU.

[Modification 8]
In the various embodiments and modifications described above, the instruction from the examiner may be an instruction by voice or the like in addition to a manual instruction (for example, an instruction using a user interface or the like). At this time, for example, a machine learning model including a speech recognition model (speech recognition engine, trained model for speech recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by character input using a keyboard, touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (a character recognition engine, a learned model for character recognition) obtained by machine learning may be used. Also, the instruction from the examiner may be an instruction by a gesture or the like. At this time, a machine learning model including a gesture recognition model (a gesture recognition engine, a trained model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be the sight line detection result of the examiner on the display screen of the output unit 103, or the like. The line-of-sight detection result may be, for example, a pupil detection result using a moving image of the examiner captured from the periphery of the display screen in the output unit 103 . At this time, the object recognition engine as described above may be used for pupil detection from moving images. Further, the instructions from the examiner may be instructions based on brain waves, weak electrical signals flowing through the body, or the like.

In such a case, for example, as the learning data, character data or voice data (waveform data) indicating instructions for displaying the results of the processing of the various learned models as described above may be used as input data. The learning data may be learning data in which correct data is an execution command for actually displaying the result of model processing on the output unit 103 . Further, as the learning data, for example, even if it is learning data having correct data such as an execution command for whether or not to automatically set shooting parameters and an execution command for changing the button for that command to an active state, etc. good. Furthermore, as the learning data, for example, an execution command for whether or not to perform image quality enhancement processing in order to obtain a high quality image, an execution command for changing the button for the command to an active state, etc. are used as correct data. It may be learning data. Any learning data may be used as long as the contents of instructions indicated by character data or voice data and the contents of execution commands correspond to each other. Alternatively, speech data may be converted into character data using an acoustic model, a language model, or the like. Also, waveform data obtained by a plurality of microphones may be used to perform processing for reducing noise data superimposed on audio data. Further, it may be configured such that an instruction by text, voice, or the like and an instruction by a mouse, a touch panel, or the like can be selected according to an instruction from the examiner. Moreover, it may be configured such that ON/OFF of instructions by text, voice, or the like can be selected according to instructions from the examiner.

Here, machine learning includes deep learning as described above, and RNN, for example, can be used for at least part of the multi-layered neural network. Here, as an example of the machine learning model according to this modification, an RNN, which is a neural network that handles time-series information, will be described with reference to FIGS. 15(a) and 15(b). Also, a long short-term memory (hereinafter referred to as LSTM), which is a type of RNN, will be described with reference to FIGS. 16(a) and 16(b).

FIG. 15(a) shows the structure of RNN, which is a machine learning model. RNN 1520 has a loop structure in the network, receives data x ^t 1510 at time t, and outputs data h ^t 1530 . Since the RNN 1520 has a loop function in the network, it is possible to take over the state of the current time to the next state, so it can handle time-series information. FIG. 15B shows an example of input and output of parameter vectors at time t. The data x ^t 1510 includes N (Params1 to ParamsN) data. Data h ^t 1530 output from the RNN 1520 includes N pieces of data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information during error backpropagation, LSTM is sometimes used. The LSTM can learn long-term information by having a forget gate, an input gate, and an output gate. Here, the structure of LSTM is shown in FIG. 16(a). In LSTM 1640, the information that the network takes over at the next time t is the internal state c ^t-1 of the network called a cell and the output data h ^t-1 . Note that the lower case letters (c, h, x) in the figure represent vectors.

Next, the details of the LSTM 1640 are shown in FIG. 16(b). In FIG. 16(b), a forget gate network FG, an input gate network IG and an output gate network OG are shown, each being a sigmoid layer. Therefore, a vector in which each element is a value between 0 and 1 is output. The forget gate network FG determines how much past information is retained, and the input gate network IG determines which values are updated. Also, in FIG. 16(b), the cell update candidate network CU is shown, and the cell update candidate network CU is the activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gating network OG selects elements of the cell candidates and how much information to convey the next time.

Note that the above-described LSTM model is a basic model, so it is not limited to the network shown here. Coupling between networks may be changed. QRNN (Quasi Recurrent Neural Network) may be used instead of LSTM. Furthermore, machine learning models are not limited to neural networks, and boosting, support vector machines, and the like may be used. In addition, when the instruction from the examiner is input by text, voice, or the like, a technique related to natural language processing (for example, sequence to sequence) may be applied. At this time, as a technique related to natural language processing, for example, a model that is output for each sentence that is input may be applied. Moreover, the various learned models described above may be applied not only to instructions from the examiner but also to output to the examiner. Also, a dialogue engine (dialogue model, learned model for dialogue) that responds to the examiner with text or voice output may be applied.

Also, as a technique related to natural language processing, a trained model obtained by pre-learning document data by unsupervised learning may be used. Also, as a technique related to natural language processing, a trained model obtained by performing transfer learning (or fine-tuning) on a trained model obtained by pre-learning may be used according to the purpose. Also, as a technology related to natural language processing, for example, BERT (Bidirectional Encoder Representations from Transformers) may be applied. Also, as a technique related to natural language processing, a model that can extract (express) the context (feature value) by itself by predicting a specific word in a sentence from both the left and right contexts may be applied. Also, as a technique related to natural language processing, a model that can determine the relationship (continuity) between two sequences (sentences) in input time-series data may be applied. In addition, as a technique related to natural language processing, a model in which Encoder of Transformer is used in the hidden layer and a sequence of vectors is input and output may be applied.

Here, the instructions from the examiner to which this modification can be applied include changing the display of various images and analysis results as described in the various embodiments and modifications described above, and generating an En-Face image. Selection of depth range, selection of whether to use as learning data for additional learning, selection of trained models, output (display, transmission, etc.) and storage of results obtained using various trained models Any at least one instruction will suffice. Further, the instructions from the examiner to which this modification can be applied may be not only instructions after imaging but also instructions before imaging. , may be an instruction regarding the start of shooting. Further, the instruction from the examiner to which this modified example can be applied may be an instruction regarding change of the display screen (screen transition).

The machine learning model may be a machine learning model that combines a machine learning model for images such as CNN and a machine learning model for time-series data such as RNN. With such a machine learning model, for example, it is possible to learn the relationship between the feature amount related to images and the feature amount related to time-series data. When the input layer side of the machine learning model is CNN and the output layer side is RNN, for example, a medical image is used as input data, and sentences related to the medical image (for example, the presence or absence of a lesion, the type of lesion, recommendations for the next examination etc.) may be used as output data for learning. As a result, for example, the medical information related to the medical image is automatically explained in sentences, so that even an examiner with little medical experience can easily understand the medical information related to the medical image. In addition, when the input layer side of the machine learning model is RNN and the output layer side is CNN, for example, medical sentences such as lesions, findings, and diagnoses are input data, and medical images corresponding to the medical sentences are output. Learning may be performed using learning data as data. As a result, for example, the examiner can easily search for medical images related to the case that the examiner wants to check.

In addition, even if a machine translation engine (machine translation model, trained model for machine translation) that machine-translates sentences such as letters and sounds into any language is used for instructions from the examiner and output to the examiner good. Any language may be configured to be selectable according to instructions from the examiner. Also, any language may be automatically selected by using a trained model that automatically recognizes the type of language. Further, the automatically selected language type may be configured to be modifiable according to instructions from the examiner. For example, the above-described natural language processing technology (for example, sequence to sequence) may be applied to the machine translation engine. For example, after a text input to a machine translation engine is machine-translated, the machine-translated text may be input to a character recognition engine or the like. Further, for example, the sentences output from the various learned models described above may be input to a machine translation engine, and the sentences output from the machine translation engine may be output.

Also, the various learned models described above may be used in combination. For example, characters corresponding to instructions from the examiner are input to a character recognition engine, and speech obtained from the input characters is input to another type of machine learning engine (for example, a machine translation engine, etc.). may be Alternatively, for example, characters output from another type of machine learning engine may be input to a character recognition engine, and voice obtained from the input characters may be output. Also, for example, a voice corresponding to an instruction from an examiner is input to a voice recognition engine, and characters obtained from the input voice are input to another type of machine learning engine (for example, a machine translation engine, etc.). may be configured to Further, for example, the speech output from another type of machine learning engine may be input to the speech recognition engine, and characters obtained from the input speech may be displayed on the output unit 103 . At this time, the output to the examiner may be configured to be selectable between character output and voice output according to instructions from the examiner. In addition, it may be configured such that input by text or input by voice as an instruction from the examiner can be selected according to the instruction from the examiner. Also, the above-described various configurations may be adopted by selection based on instructions from the examiner.

[Modification 9]
A label image, a high-quality image, and the like related to the image acquired by the actual photographing may be stored in the storage unit 101-3 according to an instruction from the operator. At this time, for example, after an instruction from the operator to save a high-quality image, when registering a file name, any part of the file name (for example, the first part, or last part), the file name containing information (for example, characters) indicating that it is an image generated by processing using a trained model for high image quality (high image quality processing) is entered by the operator. It may be displayed in an editable state according to instructions. Similarly, for the label image and the like, a file name including information that the image is generated by processing using the trained model may be displayed.

Also, when displaying a high-quality image on the output unit 103 on various display screens such as a report screen, it is necessary to confirm that the displayed image is a high-quality image generated by processing using a high-quality image model. A descriptive indication may be displayed along with the high quality image. In this case, the operator can easily identify from the display that the displayed high-quality image is not the image itself acquired by photography, thereby reducing misdiagnosis and improving diagnostic efficiency. be able to. It should be noted that the display indicating that the image is a high-quality image generated by processing using the high-quality image model may be any mode as long as the input image and the high-quality image generated by the processing are identifiable. may be of In addition to the processing using the high image quality model, the processing using various trained models as described above is also shown to be the result generated by the processing using the type of trained model. A display may be displayed with the results. For example, when displaying the analysis result of the segmentation result using the trained model for image segmentation processing, the display indicating that the analysis result is based on the result of using the trained model for image segmentation May be displayed with results.

At this time, the display screen such as the report screen may be stored as image data in the storage unit 101-3 according to an instruction from the operator. For example, a report screen is stored in storage unit 101-3 as a single image in which high-quality images, etc., and an indication indicating that these images are images generated by processing using a learned model are arranged. good too.

In addition, regarding the display indicating that the image is a high-quality image generated by processing using the high-quality model, the display indicating what kind of learning data the high-quality model has learned is displayed on the output unit. 103 may be displayed. The display may include an explanation of the types of input data and correct data of the learning data, and any display related to correct data such as imaging regions included in the input data and correct data. It should be noted that, for example, for processing using the above-described various trained models such as image segmentation processing, the output unit 103 displays information indicating what kind of learning data the trained model of that type has learned. may be displayed.

Further, information (for example, characters) indicating that the image is generated by processing using a trained model may be displayed or stored in a state superimposed on the image or the like. At this time, the portion to be superimposed on the image may be any region (for example, the end of the image) that does not overlap the region where the region of interest to be imaged is displayed. Alternatively, a non-overlapping region may be determined and superimposed on the determined region. In addition to processing using the image quality enhancement model, images obtained by processing using the above-described various trained models, such as image segmentation processing, may be similarly processed.

In addition, if the initial display screen of the report screen is set by default so that the high image quality processing button, etc. is in an active state (high image quality processing is turned on), high It may be configured such that a report image corresponding to the report screen including the quality image and the like is transmitted to the server. Also, if the button is set to be active by default, at the end of the examination (for example, when the imaging confirmation screen or preview screen is changed to the report screen in response to instructions from the examiner) Alternatively, the report image corresponding to the report screen including the high-quality image and the like may be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, whether or not the analysis map is superimposed, whether or not it is a high-quality image, the display screen for follow-up observation setting regarding at least one of whether or not the report image generated based on the setting is transmitted to the server. It should be noted that the same processing may be performed even when the button indicates switching of the image segmentation processing.

[Modification 10]
In the above-described embodiments and modifications, among the various trained models as described above, an image obtained by the first type of trained model (for example, a high-quality image, an image showing an analysis result such as an analysis map, An image showing a region recognition result, an image showing a segmentation result) may be input to a second type of trained model different from the first type. At this time, it may be configured such that a result (for example, an analysis result, a diagnosis result, a region recognition result, a segmentation result) by processing the second type of trained model is generated.

Further, among the various learned models as described above, the results of processing of the first type of learned model (for example, analysis results, diagnosis results, region recognition results, segmentation results) are used to obtain the first type An image to be input to a second type of trained model different from the first type may be generated from the image input to the trained model of . At this time, the generated image is highly likely to be an image suitable for processing using the second type of trained model. For this reason, images obtained by inputting the generated images into the second type of trained model (for example, high-quality images, images showing analysis results such as analysis maps, images showing region recognition results, segmentation results) image shown) can be improved.

By inputting a common image to the first type of trained model and the second type of trained model, it is possible to generate (or display) each processing result using these trained models. may be configured to execute At this time, for example, the generation (or display) of each processing result using these learned models may be collectively (in conjunction with) executed according to an instruction from the examiner.

Also, the type of image to be input (for example, high-quality image, region recognition result, object recognition result, segmentation result, similar case image), the type of processing result to be generated (or displayed) (for example, high-quality image, region recognition result) , diagnosis results, analysis results, object recognition results, segmentation results, similar case images), input types and output types (e.g. text, voice, language), etc. can be selected according to instructions from the examiner. may be Furthermore, the type of input may be configured to be automatically selectable by using a trained model that automatically recognizes the type of input. Also, the type of output may be configured to be automatically selectable so as to correspond (eg, be the same type) as the type of input. Furthermore, the automatically selected type may be configured to be modifiable according to instructions from the examiner. At this time, at least one trained model may be selected according to the selected type. At this time, when a plurality of trained models are selected, the method of combining the plurality of trained models (for example, the order of inputting data, etc.) may be determined according to the selected type. Note that, for example, the type of image to be input and the type of processing result to be generated (or displayed) may be configured to be selectable differently, or if they are the same, they may be selected differently. It may be configured to output prompting information to the examiner.

Also, each trained model may be executed at any location. For example, some of the trained models may be used in a cloud server, and others may be used in another server such as a fog server or an edge server. In addition, when configuring a network within a facility, within a site that includes a facility, within an area that includes multiple facilities, etc., for wireless communication, for example, Reliability of the network may be improved by configuring to use radio waves of a dedicated wavelength band. Alternatively, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and multiple simultaneous connections. As a result, for example, vitreous body, cataract, glaucoma, corneal refractive correction, extraocular surgery, laser photocoagulation, and other treatments can be supported in real time even remotely. At this time, for example, information obtained by using at least one of various learned models by a fog server or an edge server that wirelessly receives at least one of various medical images obtained by these devices related to surgery and treatment. may be configured to be wirelessly transmitted to a surgical or therapeutic device. Further, for example, the information received wirelessly by a device related to surgery or treatment may be the amount of movement (vector) of the optical system or optical member as described above. In this case, the device related to surgery or treatment may be automatically controlled. It may be configured as Further, for example, for the purpose of assisting operations by the examiner, automatic control (semi-automatic control) accompanied by permission of the examiner may be configured.

Also, similar case image retrieval may be performed using an external database stored in a server or the like, using the analysis results, diagnosis results, and the like obtained by processing the learned model as described above as retrieval keys. Further, a similar case image search may be performed using an external database stored in a server or the like using object recognition results, segmentation results, and the like obtained by processing various learned models as described above as search keys. In addition, when multiple medical images stored in a database are already managed in a state in which the feature values of each of the multiple medical images are attached as incidental information by machine learning, etc., the medical images themselves A similar case image search engine (similar case image search model, trained model for similar case image search) may be used as a search key. For example, the image processing apparatus 101 uses a trained model for similar case image retrieval (different from a trained model for high image quality) to retrieve similar case images related to the medical image from various medical images. It can be performed.

Further, for example, the display control unit 101-5 can cause the output unit 103 to display similar case images obtained from various medical images using a trained model for similar case image retrieval. At this time, the similar case image is, for example, an image with a feature amount similar to the feature amount of the medical image input to the trained model. In addition, for example, when a partial region such as an abnormal site is included in the medical image input to the trained model, the similar case image is an image with a feature amount similar to the feature amount of the partial region such as the abnormal site. . For this reason, for example, it is possible not only to efficiently perform learning for accurately retrieving similar case images, but also to efficiently diagnose an abnormal site when an abnormal site is included in the medical image. be able to. Also, a plurality of similar case images may be retrieved, and the plurality of similar case images may be displayed such that the order in which the feature amounts are similar can be identified. In addition, the trained model for similar case image retrieval is additionally trained using learning data including an image selected according to an instruction from the examiner and the feature amount of the image from among the plurality of similar case images. may be configured to be

In addition, the learning data of various trained models is not limited to data obtained using the ophthalmologic apparatus itself that actually performs imaging. Data or the like obtained using an ophthalmologic apparatus may be used.

Note that various trained models according to the above-described embodiments and modifications can be provided in the image processing apparatus 101 . A trained model may be configured by, for example, a software module or the like executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured by a circuit or the like that performs a specific function such as an ASIC. Also, these learned models may be provided in a device such as another server connected to the image processing device 101 . In this case, the image processing apparatus 101 can use the trained model by connecting to a server or the like having the trained model via an arbitrary network such as the Internet. Here, the server provided with the learned model may be, for example, a cloud server, a fog server, an edge server, or the like. In addition, when configuring a network within a facility, within a site that includes a facility, within an area that includes multiple facilities, etc., for wireless communication, for example, Reliability of the network may be improved by configuring to use radio waves of a dedicated wavelength band. Alternatively, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and multiple simultaneous connections.

[Modification 11]
Medical images processed by the image processing apparatus 101 according to the various embodiments and modifications described above include images acquired using any modality (imaging device, imaging method). Medical images to be processed can include medical images acquired by any imaging device or the like, and images created by a medical image processing device or a medical image processing method.

Furthermore, the medical image to be processed is an image of a predetermined region of a subject (subject), and the image of the predetermined region includes at least part of the predetermined region of the subject. Also, the medical image may include other parts of the subject. Also, the medical image may be a still image or a moving image, and may be a black-and-white image or a color image. Further, the medical image may be an image representing the structure (morphology) of a predetermined site, or an image representing its function. Images representing functions include, for example, images representing blood flow dynamics (blood flow, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. In addition, the predetermined part of the subject may be determined according to the object to be imaged. Includes optional parts such as legs and arms.

Further, the medical image may be a tomographic image or a front image of the subject. The front image is, for example, an SLO image of the fundus or the anterior segment of the eye, a fundus image obtained by fluorescence photography, or data acquired by OCT (three-dimensional OCT data), and data of at least a part of the range in the depth direction of the object to be photographed. Contains En-Face images generated using The En-Face image is an OCTA En-Face image (motion contrast frontal image) generated using data of at least a partial range in the depth direction of the object to be imaged for three-dimensional OCTA data (three-dimensional motion contrast data). ). Three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Here, the motion contrast data is data indicating changes between a plurality of volume data obtained by controlling the measurement light to scan the same region (same position) of the subject's eye a plurality of times. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data indicating changes between a plurality of tomographic images obtained at approximately the same position at each different position. Note that the motion contrast frontal image is also called an OCTA frontal image (OCTA En-Face image) related to OCT angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also called OCTA data. Motion contrast data can be obtained, for example, as a decorrelation value between two tomographic images or their corresponding interference signals, a variance value, or a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value). , may be determined by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to scan the same region (same position) of the subject's eye a plurality of times. When controlling the scanning means so that the measurement light scans approximately the same position a plurality of times, the time interval between one scan (one B scan) and the next scan (next B scan) ) may be changed (determined). As a result, for example, even if the blood flow velocity varies depending on the state of the blood vessel, the blood vessel region can be visualized with high accuracy. At this time, for example, the time interval may be configured to be changeable according to an instruction from the examiner. Further, for example, any motion contrast image may be selectable from a plurality of motion contrast images corresponding to a plurality of preset time intervals according to an instruction from the examiner. Further, for example, the time interval when the motion contrast data is acquired and the motion contrast data may be associated with each other and stored in the storage unit 101-3. Further, for example, the display control unit 101-5 may cause the output unit 103 to display the time interval when the motion contrast data is acquired and the motion contrast image corresponding to the motion contrast data. Also, for example, the time interval may be determined automatically, or at least one candidate for the time interval may be determined. At this time, for example, a machine learning model may be used to determine (output) the time interval from the motion contrast image. Such a machine learning model, for example, uses a plurality of motion contrast images corresponding to a plurality of time intervals as input data, and corrects the difference from the plurality of time intervals to the time interval when the desired motion contrast image is acquired. It can be obtained by learning learning data to be data.

An En-Face image is, for example, a front image generated by projecting the data of the range between two layer boundaries in the XY directions. At this time, the front image is at least a partial depth range of volume data (three-dimensional tomographic image) obtained using optical interference, and is data corresponding to the depth range determined based on the two reference planes. is generated by projecting or integrating on a two-dimensional plane. The En-Face image is a front image generated by projecting data corresponding to a depth range determined based on the detected retinal layers out of the volume data onto a two-dimensional plane. As a method of projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a representative value of data within the depth range is used as a pixel value on the two-dimensional plane. method can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within the range in the depth direction of the area surrounded by the two reference planes. Further, the depth range of the En-Face image is, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of two layer boundaries regarding the detected retinal layer. good too. Further, the depth range related to the En-Face image is, for example, a range changed (offset) according to the operator's instruction from the range between two layer boundaries related to the detected retinal layer. good.

Also, the imaging device is a device for capturing an image used for diagnosis. The imaging device is, for example, a device that obtains an image of a predetermined portion of a subject by irradiating the predetermined portion of the subject with radiation such as light, X-rays, electromagnetic waves, or ultrasonic waves, or detects radiation emitted from the subject. It includes a device for obtaining an image of a predetermined site by means of More specifically, the imaging apparatus according to the various embodiments and modifications described above includes at least an X-ray imaging apparatus, a CT apparatus, an MRI apparatus, a PET apparatus, a SPECT apparatus, an SLO apparatus, an OCT apparatus, an OCTA apparatus, and a fundus. Including cameras and endoscopes. Note that the configurations according to the above-described embodiments and modifications can be applied to these imaging devices. In this case, the movement of the subject corresponding to the movement of the subject's eye to be predicted may be, for example, the movement of the face or body, the movement of the heart (heartbeat), or the like.

The OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Fourier-domain OCT devices may also include spectral-domain OCT (SD-OCT) devices and wavelength-swept OCT (SS-OCT) devices. Also, the OCT apparatus may include a Line-OCT apparatus using line light (or an SS-Line-OCT apparatus). Also, the OCT apparatus may include a Full Field-OCT apparatus using area light (or an SS-Full Field-OCT apparatus). Also, the OCT device may include a Doppler-OCT device. Further, the SLO device and the OCT device may include a wavefront compensation SLO (AO-SLO) device and a wavefront compensation OCT (AO-OCT) device using a wavefront compensation optical system. In addition, the SLO device and the OCT device may include a polarization SLO (PS-SLO) device and a polarization OCT (PS-OCT) device for visualizing information on polarization phase difference and depolarization. Also, the SLO device and the OCT device may include a pathological microscope SLO device, a pathological microscope OCT device, and the like. Also, the SLO device and the OCT device may include a handheld SLO device, a handheld OCT device, and the like. Also, the SLO device and the OCT device may include a catheter SLO device, a catheter OCT device, and the like. Further, the SLO device and the OCT device may include a head-mounted SLO device, a head-mounted OCT device, and the like. Also, the SLO device and the OCT device may include a binocular SLO device, a binocular OCT device, and the like. Also, the SLO device and the OCT device may be of a configuration capable of optically varying the magnification so that the imaging angle of view can be changed. In addition, the SLO device can acquire a color image or a fluorescence image by using each light source of RGB and by a configuration in which light is received by one light receiving element in a time division manner or by a configuration in which light is received by a plurality of light receiving elements at the same time. good too.

Further, in the above-described embodiments and modifications, the image processing apparatus 101 is configured as part of the OCT apparatus, but the image processing apparatus 101 may be configured separately from the OCT apparatus. In this case, the image processing apparatus 101 may be connected to the imaging apparatus 100 of the OCT apparatus via the Internet or the like. Also, the configuration of the OCT apparatus is not limited to the configuration described above, and a part of the configuration included in the OCT apparatus, for example, the SLO imaging unit may be configured separately from the OCT apparatus.

Note that in trained models for voice recognition, character recognition, gesture recognition, etc., the inclination between input continuous time-series data values is extracted as part of the feature quantity and used for estimation processing. it is conceivable that. Such a trained model is expected to be able to perform accurate estimation by using the effects of temporal changes in specific numerical values in estimation processing. In addition, even in the trained models for image quality improvement, region recognition, segmentation processing, image analysis, and diagnosis result generation according to the above-described embodiments and modifications, the magnitude of the luminance value of the tomographic image, the bright portion, and the It can be considered that the order, inclination, position, distribution, continuity, etc. of the dark areas are extracted as part of the feature amount and used for the estimation process.

(Other embodiments)
The present invention supplies software (program) that implements one or more functions of the various embodiments and modifications described above to a system or device via a network or a storage medium, and the computer of the system or device executes the program. It can also be realized by a process of reading and executing. A computer has one or more processors or circuits and may include separate computers or a network of separate processors or circuits for reading and executing computer-executable instructions.

As such, the processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuitry may include a digital signal processor (DSP), data flow processor (DFP), or neural processing unit (NPU).

Although the present invention has been described with reference to the embodiments and modifications, the present invention is not limited to the above embodiments and modifications. Inventions modified within the scope of the present invention and inventions equivalent to the present invention are also included in the present invention. Moreover, the above-described embodiments and modifications can be appropriately combined within the scope of the present invention.

Claims (27)

A trained model obtained by learning with learning data including input data that is a medical image having a first image size, and is a medical image having a second image size that is larger than the first image size. An image processing apparatus comprising an image processing unit that outputs a second image having the second image size as output data by inputting the first image as input data to the trained model. The image processing unit uses, as input data, a plurality of medical images having the second image size obtained by dividing a medical image having a third image size larger than the second image size into a plurality of images. 2. The image processing apparatus according to claim 1, wherein a plurality of images corresponding to said plurality of medical images are output as output data by inputting to said trained model. 3. The image processing apparatus according to claim 2, wherein said image processing unit generates an image of said third image size by synthesizing said plurality of output images. 4. The image processing apparatus according to claim 1, wherein said trained model includes an Encoder-Decoder type Transformer. The image processing unit processes the target region in the first image or the second image so that the difference between the pixel value of the target region and the pixel value of the region other than the target region widens, and the pixels of the target region 5. The image processing apparatus according to any one of claims 1 to 4, wherein image processing is performed so that the value becomes lower. The image processing is performed on the first image and the second image so that the difference between the pixel values of the target region and the pixel values of the regions other than the target region widens and the pixel values of the target region become lower. 6. The image processing apparatus according to claim 5, further comprising a process of blending between and. The image processing unit detects a target region in a medical image using the first image or the second image, and detects a target region in the second image corresponding to the detected target region in the first image. 7. The image processing apparatus according to claim 5, wherein said image processing is performed on a region. The image processing device according to any one of claims 1 to 7, wherein the image processing unit detects a target area using the trained model. The medical image is a frontal image,
The image processing unit detects a two-dimensional target region using the learned model obtained by learning using a plurality of front images of the subject corresponding to a plurality of depth ranges as learning data. 8. The image processing device according to any one of 7. The medical image is a frontal image,
The image processing unit selects according to an instruction from an examiner from among a plurality of learned models obtained by learning using a plurality of front images of the subject corresponding to a plurality of depth ranges as learning data. 8. The image processing device according to any one of claims 1 to 7, wherein a trained model corresponding to a depth range is selected, and the selected trained model is used to detect a two-dimensional target region. The medical image is a three-dimensional medical image,
8. The image processing unit detects a three-dimensional target region using the learned model obtained by learning using a three-dimensional medical image of a subject as learning data. The image processing device according to . The image processing apparatus according to any one of claims 1 to 7, wherein the image processing unit uses the first image to detect the target region by rule-based processing based on the structure of the subject. 13. The image processing according to any one of claims 1 to 12, wherein the image processing includes processing for correcting at least one of brightness and contrast such that a difference between pixel values of the target area and pixel values of areas other than the target area widens. The image processing device according to . the medical image of the subject is a motion contrast image of the subject's eye,
14. The image processing apparatus according to any one of claims 1 to 13, wherein the target region includes at least one of an aperfused region, a foveal vessel region, and an optic disc region. The output unit performs image quality improvement processing using a first trained model obtained by learning using the medical image of the subject as learning data, and blend processing or brightness and contrast in the target region of the medical image of the subject. Using a second trained model obtained by learning using the medical image of the subject that has undergone at least one of the correction processing as learning data, performing image quality improvement processing on the first image, 15. The image processing apparatus according to any one of claims 1 to 14, which outputs a second medical image of the subject. The image processing apparatus according to any one of claims 1 to 15, further comprising a display control unit that causes a display unit to display a live moving image of the subject. The display control unit is a high-quality image generated by using a trained model obtained by learning learning data including an image related to a subject, and is a high-quality image obtained by inputting the medical image of the subject. 17. The image processing apparatus according to claim 16, wherein an image is displayed on said display unit. The display control unit causes the display unit to display the anterior eye image generated as the high-quality image as the live moving image, and generates the SLO image generated as the high-quality image as the high-quality image. an SLO image on which a line indicating the position of the obtained tomographic image is superimposed is displayed on the display unit as the live moving image, and the tomographic image corresponding to the position of the line on the SLO image is used as the live moving image. 18. The image processing device according to claim 17, which is displayed on said display unit. The display control unit displays information indicating a blood vessel region in the tomographic image corresponding to the position of the line, which is the tomographic image generated as the high-quality image, superimposed on the tomographic image corresponding to the position of the line. 19. The image processing device according to claim 18, wherein The display control unit is an analysis result generated using a learned model for generating analysis results obtained by learning learning data including an image related to the subject, and obtained by inputting an image related to the subject. 20. The image processing apparatus according to any one of claims 16 to 19, wherein the analysis result is displayed on the display unit. The display control unit is a diagnostic result generated using a trained model for generating a diagnostic result obtained by learning learning data including an image related to the subject, and obtained by inputting an image related to the subject. 21. The image processing apparatus according to any one of claims 16 to 20, wherein a diagnosis result is displayed on said display unit. The display control unit is an image generated using a hostile generation network or an autoencoder, the image obtained by inputting an image related to the subject, and the image input to the hostile generation network or the autoencoder. 22. The image processing apparatus according to any one of claims 16 to 21, wherein the display unit displays information about the difference from the image of the subject as information about the abnormal site. The display control unit inputs an image of a subject, which is a similar case image retrieved using a trained model for similar case image retrieval obtained by learning learning data including an image of the subject. The image processing apparatus according to any one of claims 16 to 22, wherein the obtained similar case images are displayed on the display unit. The display control unit is an object recognition result or a segmentation result generated using a trained model for object recognition or a trained model for segmentation obtained by learning learning data including an image related to a subject, 24. The image processing apparatus according to any one of claims 16 to 23, wherein an object recognition result or a segmentation result obtained by inputting an image of a subject is displayed on the display unit. The operator's instruction for obtaining the second image uses at least one trained model among a trained model for character recognition, a trained model for speech recognition, and a trained model for gesture recognition. 25. The image processing apparatus according to any one of claims 1 to 24, wherein the information is obtained by A trained model obtained by learning with learning data including input data that is a medical image having a first image size, and is a medical image having a second image size that is larger than the first image size. An image processing method, comprising a step of outputting a second image having the second image size as output data by inputting the first image as input data to the trained model. A program that, when executed by a computer, causes the computer to perform each step of the image processing method according to claim 26.

Images