JP2021069667A - Image processing device, image processing method and program - Google Patents

Abstract

To effectively reduce noise in a medical image, such as a tomographic image or a front image, which is obtained by OCT or the like.SOLUTION: An image processing device comprises: means for detecting a position of noise in a first medical image obtained by imaging a subject; means for generating a second medical image in which a pixel value at the position of the noise detected in the first medical image is reduced, and an estimation image in which it is estimated that the noise in the first medical image is reduced; means for changing the estimation image using an objective function including a first term which concerns a difference between the second medical image and the estimation image, and a second term which concerns the estimation image; and means for outputting, as a noise reduction image, the changed estimation image, which is obtained so that the object function satisfies a predetermined condition.SELECTED DRAWING: Figure 4

Description

The disclosed techniques relate to image processing devices, image processing methods and programs.

An optical coherence tomography (OCT) is a device that photographs a tomographic image using the interference of light, and the subject may be any substance that transmits light. In particular, an optical interference tomometer for ophthalmology is called a tomographic imaging device because the subject is the eye and it is possible to observe the state inside the retinal layer three-dimensionally. This tomographic imaging device has been attracting attention in recent years because it is useful for more accurately diagnosing diseases. As a form of OCT, there is TD-OCT (Time domain OCT) which is a combination of a wide band light source and a Michelson interferometer. This is because the interference light between the backward scattered light and the reference light generated when the light is incident on the subject is measured, and the reflection intensity increases when the optical path length of the reference light and the optical path length of the backward scattered light match. Is configured to obtain information in the depth direction of the eye to be inspected. With TD-OCT, only one point in the depth direction can be obtained by one measurement. Therefore, the TD-OCT needs to acquire information on a plurality of points in the depth direction while changing the optical path length of the reference light by mechanically moving the reference light mirror that reflects the reference light. Therefore, FD-OCT (Fourier domain OCT) has been developed as a method for acquiring images at higher speed. With FD-OCT, all information in the depth direction can be obtained by one measurement. Therefore, in FD-OCT, mechanical scanning in the depth direction becomes unnecessary, and measurement can be performed at a higher speed than TD-OCT. Two types of OCT are known for FD-OCT. One is SD-OCT (Spectral domain OCT), which acquires an interferogram with a spectroscope using a wideband light source and obtains information in Fourier space. The other is SS-OCT (Swept Source OCT), in which light wave interference is performed in Fourier space by changing the transmission wavelength of the light source at high speed. The FD-OCT tomographic image is an absolute value component of complex data obtained by Fourier transforming the observed signal. The signal of this absolute value component is called a luminance image or an intensity image.

Here, as a method for reducing noise in an OCT image, there is a method using a convolution filter (for example, a method of applying a Gaussian filter having a size of 3x3 to an image). This method is a method of applying noise reduction to a local image area. Specifically, by sequentially applying this noise reduction to individual local parts of the entire image, noise reduction is performed for the entire image. Further, as a method for reducing noise of an OCT image, a method called a total variation noise reduction method (total variation noise removal method) is known (Patent Document 1). This method is a method of reducing noise in consideration of the amount of variable components in the entire image. That is, noise reduction is performed so that the amount of variation component of the entire image after noise reduction becomes a certain value. Here, the variable component is an absolute value of the difference from the adjacent pixel. For example, the case where the amount of variation component of the entire image is zero is the case where all the pixel values of the image are the same value. By controlling the amount of variable components in the entire image, the strength of the noise reduction effect is determined. When noise reduction is performed so that the amount of fluctuating components is reduced, fluctuating components such as edge components and random noise components are reduced, and the output image looks "flat". On the contrary, when the noise is reduced so that the amount of the fluctuating component is large, the edge component and the random noise component are not reduced, so that the noise reduction effect is weakened.

Special Table 2018-531773

By the way, it has been found that even if the conventional total variation noise reduction method is applied to the OCT image as it is, it is difficult to effectively reduce the black spot noise, so that the noise may not be effectively reduced. .. Here, the black spot noise is as shown in FIG. 7 (b). Note that FIG. 7A is a tomographic image of OCT, and FIG. 7B is an enlarged view of a part of the tomographic image. As shown in FIG. 7B, many black spots are scattered (hereinafter referred to as black spots), which is one of the factors for reducing the image quality. In particular, there are black spots on the retinal layer, which is a structure, and when the doctor observes the retinal layer, the black spots interfere with the diagnosis, which may make the diagnosis difficult.

One of the purposes of the disclosed technique is to effectively reduce the noise of medical images such as tomographic images and frontal images obtained by OCT or the like, which was made in view of the above problems.

It should be noted that not only the above-mentioned purpose but also the action and effect derived by each configuration shown in the embodiment for carrying out the invention described later, and the action and effect which cannot be obtained by the conventional technique can be obtained. It can be positioned as one.

One of the disclosed image processing devices is
A means for detecting the position of noise in a first medical image obtained by imaging a subject, and
A means for generating a second medical image in which the pixel value of the detected noise position in the first medical image is reduced, and an estimated image in which noise in the first medical image is presumed to be reduced. When,
A means for changing the estimated image by using an objective function including a first term relating to the difference between the second medical image and the estimated image and a second term relating to the estimated image.
It has means for outputting an estimated image obtained by changing the objective function so as to satisfy a predetermined condition as a noise reduction image.

According to one of the disclosed techniques, it is possible to effectively reduce the noise of a medical image such as a tomographic image or a frontal image obtained by OCT or the like.

The figure which shows the structure of an image processing system. The figure for demonstrating the structure of an eye part and a tomographic image and a fundus image. A flowchart showing a processing flow in an image processing system. The block diagram for demonstrating the noise reduction part in an image processing apparatus. An example of a GUI that changes the ratio λ. A block diagram for explaining a noise reduction part when predicting the position of a black spot with a random number. The figure for demonstrating the black spot noise. The figure for demonstrating an example of a high image quality processing. An example of the configuration of a neural network used as a machine learning model. An example of the configuration of a neural network used as a machine learning model.

(First Embodiment)
Hereinafter, the image processing system including the image processing apparatus according to the first embodiment, the structure of the eye, and the image of the eye acquired by the image processing system will be described in detail with reference to the drawings.

First, FIG. 2 is a diagram showing an eye structure and an image acquired by an image processing system. FIG. 2A shows a schematic view of the eyeball. In FIG. 2A, C is the cornea, CL is the crystalline lens, V is the vitreous body, M is the macula (the central part of the macula represents the fovea centralis), and D represents the optic nerve head. The tomographic imaging apparatus 200 according to the present embodiment mainly describes a case where the posterior pole of the retina including the vitreous body, the macula, and the optic nerve head is photographed. Although not described in this embodiment, the tomographic imaging apparatus 200 can also image the anterior segment of the cornea and the crystalline lens.

Further, FIG. 1 is a diagram showing a configuration of an image processing system 100 including an image processing device 300 according to the present embodiment. As shown in FIG. 1, in the image processing system 100, the image processing device 300 has a tomographic image capturing device (also referred to as OCT) 200, a fundus image capturing device 400, an external storage unit 500, a display unit 600, and an input via an interface. It is configured by being connected to the unit 700.

Here, the tomographic image capturing apparatus 200 images an eye, which is a subject (not shown), and generates a tomographic image of the ocular retinal layer. The tomographic image is an example of a first medical image obtained by imaging a subject. At this time, the tomographic imaging apparatus 200 is configured to receive the interference light due to the return light from the eye to be examined irradiated with the measurement light and the reference light corresponding to the measurement light. Further, as the tomographic imaging apparatus 200, a Fourier domain type OCT apparatus is preferably used. Further, the tomographic imaging apparatus 200 is composed of, for example, an SD-OCT in which a light receiving means for receiving interference light is composed of a line sensor or the like, or an SS-OCT in which a wavelength sweeping light source is used. Further, although the tomographic image capturing apparatus 200 is communicably connected to the image processing apparatus 300, the tomographic imaging apparatus 200 may be configured to be incorporated inside the tomographic imaging apparatus 200. Since the tomographic image capturing apparatus 200 is a known apparatus, detailed description thereof will be omitted, and here, the imaging of the tomographic image performed according to the instruction from the image processing apparatus 300 will be described.

First, in FIG. 1, the galvanometer mirror 201, which is an example of the scanning means, is for scanning the fundus of the measurement light, and defines the imaging range of the fundus by OCT. Further, the drive control unit 202 controls the drive range and speed of the galvano mirror 201 to define the imaging range and the number of scanning lines (scanning speed in the plane direction) in the plane direction on the fundus of the eye. Further, the drive control unit 202 controls the scanning means so as to scan the measurement light at the same position of the eye to be inspected. Here, the galvano mirror is shown as one unit for the sake of simplicity, but it is actually composed of a mirror for X scan and two mirrors for Y scan, and a desired range can be scanned with measurement light on the fundus. ..

Further, the focus 203 is for focusing on the retinal layer of the fundus through the anterior segment of the eye which is the subject. The measurement light is focused on the retinal layer of the fundus through the anterior segment of the eye, which is the subject, by a focus lens (not shown). The measurement light that irradiates the fundus is reflected and scattered by each retinal layer and returned. When observing the vitreous body in detail, the focus is moved to the anterior segment side of the retinal layer to focus on the vitreous body.

Further, the internal fixation lamp 204 includes a display unit 241 and a lens 242. A display unit 241 in which a plurality of light emitting diodes (LDs) are arranged in a matrix is used. The lighting position of the light emitting diode is changed according to the part to be photographed by the control of the drive control unit 202. The light from the display unit 241 is guided to the eye to be inspected through the lens 242. The light emitted from the display unit 241 is 520 nm, and the drive control unit 202 displays a desired pattern.

Further, the coherence gate stage 205 is controlled by the drive control unit 202 in order to cope with the difference in the axial length of the eye to be inspected. The coherence gate represents a position where the optical distances of the measurement light and the reference light in OCT are equal. Furthermore, by controlling the position of the coherence gate as an imaging method, it is possible to control imaging on the retinal layer side or a deeper side than the retinal layer. Here, the tomographic image acquired by the tomographic image capturing apparatus 200 will be described with reference to FIG. 2B. In FIG. 2B, V represents the vitreous body, M represents the macula, and D represents the optic nerve head. L1 is the boundary between the inner limiting membrane (ILM) and the nerve fiber layer (NFL), L2 is the boundary between the nerve fiber layer and the ganglion cell layer (GCL), and L3 is the junction between the inner segment and the outer segment of the photoreceptor (ISOS). ), L4 represents the retinal pigment epithelial layer (RPE), L5 represents the Bruch's membrane (BM), and L6 represents the choroid. In the tomographic image, the horizontal axis (main scanning direction of OCT) is the x-axis, and the vertical axis (depth direction) is the z-axis.

Further, the fundus image capturing device 400 is a device that captures a fundus image of the eye portion, and examples of the device include a fundus camera, an SLO (Scanning Laser Ophothermoscope), and the like. FIG. 2C shows a fundus image of the eye portion. In FIG. 2 (c), M represents the macula, D represents the optic nerve head, and the thick curve represents the blood vessels of the retina. In the fundus image, the horizontal axis (the main scanning direction of the OCT) is the x-axis, and the vertical axis (the sub-scanning direction of the OCT) is the y-axis. The device configuration of the tomographic image capturing device 200 and the fundus imaging device 400 may be an integrated type or a separate type.

Further, the image processing device 300 includes an image acquisition unit 301, a storage unit 302, an image processing unit 303, an instruction unit 304, and a display control unit 305. The image acquisition unit 301 includes a tomographic image generation unit 311, acquires signal data of a tomographic image taken by the tomographic image capturing apparatus 200, and generates a tomographic image by performing signal processing. In addition, the fundus image data captured by the fundus image capturing device 400 is acquired. Then, the generated tomographic image and fundus image are stored in the storage unit 302. The image processing unit 303 includes an alignment unit 331, a detection unit 332, a calculation unit 333, and a noise reduction unit 334. The alignment unit 331 performs tomographic image alignment between a plurality of tomographic images and alignment of the tomographic image and the fundus image. The detection unit 332 detects the vitreous boundary and the vitreous region. The calculation unit 333 quantifies the characteristics of the region defined by the vitreous boundary and the upper layer of the retina. The noise reduction unit 334 specifies an area for calculation by the calculation unit 333.

In addition, the external storage unit 500 holds information about the eye to be inspected (patient's name, age, gender, etc.) in association with captured image data, imaging parameters, image analysis parameters, and parameters set by the operator. There is.

Further, the input unit 700 is, for example, a mouse, a keyboard, a touch operation screen, etc., and the operator gives an instruction to the image processing device 300, the tomographic image capturing device 200, and the fundus image capturing device 400 via the input unit 700. Do.

Next, the processing procedure of the image processing apparatus 300 of the present embodiment is shown with reference to FIGS. 3 and 4. FIG. 3 is a flowchart showing the flow of operation processing of the entire system in the present embodiment. FIG. 4 is a block diagram relating to the noise reduction unit 334 of the image processing unit 303 in the present embodiment.

<Step S301>
In step S301, the eye-examined information acquisition unit (not shown) acquires the subject identification number from the outside as information for identifying the eye to be examined. Then, based on the subject identification number, the information about the eye to be inspected held by the external storage unit 500 is acquired and stored in the storage unit 302.

<Step S302>
In step S302, the eye to be inspected is scanned and an image is taken. When the operator selects to start scanning (not shown), the tomographic imaging apparatus 200 controls the drive control unit 202 and operates the galvano mirror 201 to scan the eye to be inspected. The galvano mirror 201 is composed of an X scanner for the horizontal direction and a Y scanner for the vertical direction. Therefore, if the orientations of these scanners are changed, scanning can be performed in each of the horizontal direction (X) and the vertical direction (Y) in the device coordinate system. Then, by changing the directions of these scanners at the same time, it is possible to scan in the combined direction of the horizontal direction and the vertical direction, so that it is possible to scan in any direction on the fundus plane.

Adjust various shooting parameters when shooting. Specifically, the position of the internal fixation lamp, the scan range, the scan pattern, the coherence gate position, and the focus are set. The drive control unit 202 controls the light emitting diode of the display unit 241 to control the position of the internal fixation lamp 204 so as to take an image at the center of the macula or the optic nerve head. As the scan pattern, a scan pattern such as a raster scan for capturing a three-dimensional volume, a radial scan, or a cross scan is set. This embodiment is a C scan. The coherence gate position is on the vitreous side, and the focus is also on the vitreous body.

After the adjustment of these shooting parameters is completed, the operator selects the shooting start (not shown) to perform shooting. When shooting is started, an interference signal is output for each A scan.

<Step S303>
In step S303, a tomographic image, which is an example of the first medical image obtained by imaging the subject, is generated. The tomographic image generation unit 311 generates a tomographic image by performing a general reconstruction process on each interference signal.

First, the tomographic image generation unit 311 removes fixed pattern noise from the interference signal. Fixed pattern noise removal is performed by extracting fixed pattern noise by averaging a plurality of detected A scan signals and subtracting this from the input interference signal.

Next, the tomographic image generation unit 311 performs a desired window function processing in order to optimize the depth resolution and the dynamic range, which are in a trade-off relationship when the Fourier transform is performed in a finite interval. Next, an absolute value tomographic image is obtained by performing an absolute value calculation on the result of the FFT process. The tomographic image of the OCT is obtained by performing logarithmic calculation on the tomographic image of the absolute value, and is called a luminance image or an intensity image.

<Step S304>
In step S304, the noise reduction unit 334 reduces the noise of the tomographic image. FIG. 4 shows a block diagram of the noise reduction unit 334. The noise reduction unit uses a total variation noise reduction method (total variation de-noise) in consideration of black spots. A black dot is a pixel that is extremely dark compared to surrounding pixels.

Here, a conventional total variation noise reduction method will be described. The conventional total variation noise reduction method is expressed by the following equation.

In Equation 1, V is the evaluation function, g (x, y) is the input image, and f (x, y) is the output image. The output image f (x, y) is an image in which the noise of the input image g (x, y) is reduced. The size of the input image g (x, y) and the size of the output image f (x, y) are the same. Σ () in Equation 1 calculates the sum in parentheses for each pixel to calculate the sum for the entire image, d / dx calculates the derivative in the x direction, and d / dy calculates the derivative in the y direction. λ is a ratio for determining the ratio of the first term and the second term. Of the evaluation functions V, f (x, y) is a variable, and the others are fixed values. The value of the evaluation function V is a value of zero or more.

The total variation noise reduction method is a method of obtaining an output image f (x, y) that minimizes the evaluation function V when an input image g (x, y) is given. Here, the meaning of the evaluation function V will be mentioned. The first term is the square of the difference between the input image and the output image, so if the output image is similar to the input image, the first term will be smaller, and the second term will be the magnitude of the derivative of the output image, so the output. If the image is smooth, the second term becomes smaller. From this, reducing the evaluation function V requires that "the output image is similar to the input image, but not only is it similar, but the output image is smooth". The ratio of the first term and the second term is determined by the size of λ, and if λ is small, the ratio of the first term to the whole becomes large, so it is strongly requested that the output image resembles the input image. Become. If λ is large, the ratio of the second term to the whole becomes large, so that it is strongly requested that the output image be smooth.

If noise is to be removed, it seems that only the second term is sufficient, but if the evaluation function is composed of only the second term, f (x, y) = 5, f (x, y) = 7, etc. , The case where all the pixels of f (x, y) have constant values is solved. This is because when all the pixels have constant values, the second term becomes zero, and as a result, the evaluation function becomes zero and becomes the minimum value. When all the pixels have constant values, it does not make sense to reduce noise in the input image. Therefore, both the first term and the second term are necessary. The first term is called the error term, and the second term is called the total variation regularization term. λ is a parameter for changing the ratio of the total variation regularization term and the error term.

As an algorithm for obtaining the output image f (x, y), a steepest descent method, an ADMM (Alternating Direction Method of Multipliers), a master-dual proximity separation method, an Iteratively Reduced Last Squares method, and the like are known. All of these are iterative methods. Since it is an iterative method, the evaluation function is gradually minimized. At this time, the output image f (x, y) in the state of being toward minimization is called an estimated image of the noise reduction image. The output image f (x, y) in the minimized state is called a noise-reduced image or a noise-reduced image. The estimated image of the noise reduction image is generated by the objective function minimization unit 3344 of FIG. The minimization of the objective function is an example of a predetermined condition that the objective function should satisfy. At this time, an estimated image obtained by changing the objective function so as to satisfy the predetermined condition is output as a noise reduction image. ..

The total variation noise reduction method works effectively to reduce noise in portraits and landscapes. However, when a tomographic image is given to the input image g (x, y) of the above formula 1 and the output image f (x, y) is solved, noise is not sufficiently removed from the output image. The reason is that the tomographic image contains not only Gaussian noise but also many black spots. The first term requests that the output image and the input image are similar, but the black spots of the input image request that the black spots exist at the same positions of the output image. Therefore, it is preferable to adopt the following evaluation function V1 which is an improvement of the equation 1. In this embodiment, this V1 is used as an objective function.

Here, h (x, y) is a first medical image which is an example of a tomographic image or a frontal image obtained by imaging a subject by an OCT device, and is a pixel of a noise position in the first medical image. It is an image in which the value is deleted (reduced), and is an example of a second medical image (defective image). In the present embodiment, it is an image in which the pixel at the position of the black dot in g (x, y) is set to zero. A (x, y) is an image in which the pixel at the black dot position is zero and the other pixels are 1, and is a mask image in which the pixel value is masked with respect to the noise position. In this embodiment, A (x, y) is called an observation matrix.

Equation 2 is the same in the following equation.

With the introduction of A (x, y), the first term of the evaluation function V1 prevents the value of the evaluation function from increasing when the black spot of the output image does not exist at the same position as the black spot of the input image. That is, regarding the position of the detected noise (black spot), it is possible not to consider the difference in the pixel value for each pixel between the input image and the output image. Thanks to this, there are no black spots in the output image, or the number of black spots is reduced, and the noise reduction effect of the tomographic image is higher than that of the conventional total variation noise reduction method. That is, it is possible to generate an output image in which the pixel value at the noise (black spot) position is improved (the pixel values other than the noise position are similar, but the pixel values at the noise position are not similar). .. The generation of A (x, y) is performed by the observation matrix generation unit 3342 of the internal block of the noise reduction unit 334.

In the present embodiment, the output image f (x, y) considers an image obtained by performing an absolute value calculation on the result of the FFT process in step S303 and further performing a logarithmic calculation. An image that is not performed may be given. In this case, it is preferable to use V2 as the evaluation function.

In the evaluation function V2, δ is a constant used when performing a logarithmic operation. When the evaluation function V2 is used, the output image f (x, y) is obtained by the solution algorithm, and then logarithmic calculation is performed to obtain a noise-reduced image of the tomographic image. In particular

を演算して対数画像を得て、断層画像のノイズ低減画像を得る。

Is calculated to obtain a logarithmic image, and a noise-reduced image of a tomographic image is obtained.

The image processing system 100 inputs the tomographic image generated in step S303 and the imaging parameters in step S302 to the noise reduction unit 334. The noise reduction unit 334 inputs a tomographic image and imaging parameters to the noise detection unit 3341. Further, the noise reduction unit 334 inputs the tomographic image to the objective function minimization unit 3344.

The noise detection unit 3341 detects black spots, which are a type of noise, from the tomographic image and identifies the pixel positions of the black spots. As a method of detecting black spots, there is a method of creating an average image and comparing the average image with the tomographic image. Specifically, a pixel whose pixel value of the tomographic image is 70% or less of the pixel value of the average image is determined to be a black dot. The number 70% is referred to as the number A for convenience of explanation. The number A of 70% is the case of this embodiment, and as another embodiment, the number A may be larger or smaller than 70%. Since the occurrence of black spots is related to the shooting parameters, the noise detection unit 3341 may automatically increase the number A in the case where it can be determined from the shooting parameters that many black spots are generated. For example, in the case where the number A has an initial value and it can be determined from the shooting parameters that many black spots occur, there is a method of making it larger than the initial value. This method is a case of relative setting. Alternatively, the noise detection unit 3341 may set the number A in a case where it can be determined from the photographing parameters that many black spots are generated. This method is a case of absolute setting. In cases where it can be determined from the imaging parameters that there are few black spots, a method of automatically reducing the number A may be adopted. That is, the number A is a threshold value for detecting a black spot.

Alternatively, the noise detection unit 3341 may searchly determine the number A so that the ratio of black spots is 20% of the entire pixel. This number of 20% is referred to as the number B for convenience of explanation. A large number B means that many black spots are detected, and a small number B means that few black spots are detected. Since the pixel ratio is determined when the number A is determined, exploratory determination of the number A so that the ratio of black spots is 20% of the entire pixel is to gradually reduce the number A from 100% while gradually reducing the number A. It means to find the number A in which the number B is about 20%. In this embodiment, the number B is 20%, but in other embodiments, it may be larger or smaller than 20%. As described above, the occurrence of black spots is related to the shooting parameters. Therefore, in the case where it can be determined from the shooting parameters that many black spots are generated, the noise detection unit 3341 may automatically increase the number B. Good. In this case, the number A is exploratoryly determined, and the number A becomes large. That is, the number B is also a threshold value for detecting a black spot.

After the noise detection unit 3341 specifies the pixel positions x and y of the black point, the noise detection unit 3341 inputs the pixel position of the black point to the observation matrix generation unit 3342. The observation matrix generation unit 3342 uses the input pixel positions of the black dots to generate A (x, y) of the evaluation functions V1 and V2. As for the value of A (x, y), the value at the position of the black dot is zero, and the other values are 1. The noise reduction unit 334 inputs A (x, y) generated by the observation matrix generation unit 3342 to the objective function minimization unit 3344.

Further, the noise reduction unit 334 inputs the shooting parameters to the ratio λ generation unit 3343. The ratio λ generation unit 3343 is a block that generates the ratio λ of the evaluation functions V1 and V2. The ratio λ is a parameter that determines the strength of the noise reduction effect, and the larger the λ, the greater the noise reduction effect. Since the amount of noise is related to the shooting parameter, the ratio λ generation unit 3343 generates the ratio λ in relation to the shooting parameter. As another embodiment, the generation of the ratio λ does not have to depend on the imaging parameters. The noise reduction unit 334 inputs the λ generated by the ratio λ generation unit 3343 to the objective function minimization unit 3344.

The objective function minimization unit 3344 is a block that minimizes the objective function. The objective function of this embodiment is the evaluation function V1. The objective function minimization unit 3344 uses the input tomographic image as the input image g (x, y) and uses the input A (x, y) and the input λ to minimize the objective function. The objective function minimization unit 3344 determines that the objective function has been minimized when it detects that the value of the objective function does not change even if the iteration is repeated. The minimization algorithm of this embodiment uses ADMM, but it does not have to be this algorithm, and an algorithm capable of minimizing the objective function may be used. The output image f (x, y) obtained by minimization is a tomographic image with noise reduced. The noise reduction unit 334 outputs the noise-reduced tomographic image generated by the objective function minimization unit 3344 to the outside of the noise reduction unit 334. The image processing system 100 stores the noise-reduced tomographic image in the storage unit 302.

The ratio λ is not automatically determined by the image processing system 100, but may be changed by a person who operates the image processing system. FIG. 5 is a GUI in which the display unit and the input unit are combined. This GUI has an image display unit 6001 and a ratio λ input unit 7001. The image display unit 6001 is an area for displaying a noise-reduced tomographic image, and the ratio λ input unit 7001 is a slider for continuously or discretely changing the value of the ratio λ. Although the present embodiment is a slider, it may be a GUI for directly inputting a button or a numerical value instead of the slider. FIG. 5 is an example of a GUI in which when the ratio λ is changed by the ratio λ input unit, the noise reduction unit 334 operates and the noise-reduced tomographic image is displayed on the image display unit 6001.

Further, in the present embodiment, the noise detection unit 3341 detects a black spot, but as another embodiment, the noise detection unit 3341 detects the position of the overexposed pixel and the observation matrix generation unit 3342 masks the position of the overexposed pixel. A (x, y) may be generated. Here, the overexposed pixel is a pixel whose signal level is extremely higher than that in the vicinity of the periphery. Alternatively, as another embodiment, the noise detection unit 3341 detects the positions of both the black spots and the whiteout pixels, and the observation matrix generation unit 3342 masks the positions of the black spots and the whiteout pixels A (x, y). It may be a configuration to be generated.

<Step S305>
In step S305, the detection unit 332 detects the layer boundary of the retinal layer from the tomographic image generated in step S303 or the noise-reduced tomographic image generated in step S304, and outputs the layer boundary line. The present embodiment uses the noise-reduced tomographic image generated in step S304. The detection unit 332 first performs an edge detection calculation on the noise-reduced tomographic image in step S304, and then applies a line enhancement filter. The line enhancement filter includes, for example, an enhancement filter based on the Hessian matrix eigenvalues. The layer detection unit 332 determines the position of the layer boundary by using the result of the edge detection calculation and the result of the line emphasis filter.

<Step S307>
In step S307, the display control unit 305 causes the display unit 600 to display the noise-reduced tomographic image. That is, the display control unit 305 causes the display unit 600 to display the noise reduction image. At this time, the layer boundary line detected by the layer detection unit may be superimposed on the tomographic image and displayed on the display unit 600.

<Step S308>
In step S308, the instruction acquisition unit (not shown) acquires an instruction from the outside whether or not to end the acquisition of the tomographic image by the image processing system 100. This instruction is input by the operator using the input unit 700. When the instruction to end the process is acquired, the image processing system 100 ends the process. On the other hand, if shooting is to be continued without ending the processing, the processing is returned to step S302 to continue shooting.

As described above, the processing of the image processing system 100 is performed.

(Second Embodiment)
Hereinafter, the second embodiment will be described. The image processing system including the image processing device according to the present embodiment has some parts in common with the first embodiment. In the following, only the differences will be described.

In the first embodiment, one tomographic image was scanned. In the second embodiment, scanning is repeated a plurality of times at the same location to obtain a plurality of tomographic images, and these are added and averaged to reduce noise and reduce noise in the noise reduction unit 334. The process by addition averaging is an example of a composition process for synthesizing a plurality of estimated images, and the addition averaging image obtained by addition averaging is an example of a composition image. First, in addition to step S302 of the first embodiment, in the present embodiment, the number N (N ≧ 2) to be added and averaged is also set when adjusting various shooting parameters in performing shooting. Further, regarding the scan pattern of step S302 of the first embodiment, the place where scan patterns such as raster scan, radial scan, and cross scan for capturing a three-dimensional volume are set is the same, but in this embodiment, each scan is set. The difference is that the patterns are repeatedly photographed on one line. The number N to be added and averaged is the number of times of repeated shooting. Further, in step S303 of the first embodiment, one C-scan tomographic image is generated, but in this embodiment, N C-scan tomographic images are generated. Further, step S304 of the first embodiment performs noise reduction, but in this embodiment, N (N ≧ 2) C-scan tomographic images generated in step S203 are used before noise reduction. The alignment unit 331 aligns the absolute tomographic image. In the present embodiment, an absolute tomographic image is used, but an image obtained by performing logarithmic calculation on the absolute tomographic image may be used. As the alignment process, for example, an evaluation function representing the similarity between the two tomographic images is defined in advance, and the absolute tomographic image is deformed so that the value of this evaluation function becomes the best. As an evaluation function, a method of evaluating by a pixel value can be mentioned (for example, a method of evaluating by using a correlation coefficient can be mentioned).

Evaluating the similarity between tomographic images of the absolute value of C scan takes a long time because of the large amount of data. In order to speed up the processing time, a plurality of typical B-scan tomographic images are extracted from the C-scan absolute tomographic images to create an absolute tomographic image set, and the similarity is obtained for each absolute tomographic image set. To calculate.

Equation 6 shows the formula when the correlation coefficient is used as the evaluation function representing the degree of similarity.

In Equation 6, the region of the first absolute value tomographic image is f (x, z), and the region of the second absolute value tomographic image is g (x, z).

は、それぞれ領域ｆ（ｘ，ｚ）と領域ｇ（ｘ，ｚ）の平均を表す。なお数式６のｆとｇは、数式１〜４のｆとｇとは別物である。ここで領域とは位置合わせに用いるための画像領域であり、通常断層画像のサイズ以下の領域が設定され、眼の断層画像においては網膜層領域を含むように設定される事が望ましい。画像の変形処理としては、アフィン変換を用いて並進や回転を行ったり、拡大率を変化させたりする処理が挙げられる。次に、画像処理システム１００は、位置合わせ部３３１が位置合わせした断層画像Ｎ個に対し、ノイズ低減部３３４で１枚ずつノイズ低減を行う。ノイズ低減部３３４は、Ｎ個のノイズ低減された断層画像を出力する。その後、画像処理システム１００が、Ｎ個のノイズ低減された断層画像について、同じ座標のピクセル同士の加算平均を行い、加算平均をした断層画像を生成する。これらの処理を各ラインにおける複数の絶対値の断層画像に対して実行する。加算平均処理を行うことで、ノイズを減らし硝子体や網膜の信号を強調した高画質な画像を生成する事が出来る。

Represents the average of the region f (x, z) and the region g (x, z), respectively. Note that f and g in Equation 6 are different from f and g in Equations 1 to 4. Here, the region is an image region to be used for alignment, and it is desirable that a region smaller than the size of a tomographic image is usually set, and a region including a retinal layer region is included in a tomographic image of the eye. Examples of the image transformation process include a process of performing translation and rotation using an affine transformation, and a process of changing the enlargement ratio. Next, in the image processing system 100, the noise reduction unit 334 reduces noise one by one with respect to N tomographic images aligned by the alignment unit 331. The noise reduction unit 334 outputs N noise-reduced tomographic images. After that, the image processing system 100 performs addition averaging of pixels having the same coordinates on N noise-reduced tomographic images, and generates a tomographic image obtained by the addition averaging. These processes are performed on a plurality of absolute tomographic images in each line. By performing the averaging process, it is possible to generate a high-quality image that reduces noise and emphasizes the signals of the vitreous body and the retina.

According to this embodiment, not only the noise reduction effect of addition averaging but also the noise reduction effect by the noise reduction unit 334 can be obtained. Therefore, the same effect as the noise reduction effect obtained only by the addition average of the number of scans of N times can be obtained with the number of scans less than N. When the number of scans is small, the time for examining the patient is shortened, which has the effect of reducing the burden on the patient. In this embodiment, the scanning method is C scan, but as another embodiment, cross scan or multi-cross scan may be used. Further, in the present embodiment, the noise is reduced by the noise reduction unit 334 after the alignment, but the noise may be reduced by the noise reduction unit 334 and then aligned and added and averaged.

In the present embodiment, noise is reduced by the noise reduction unit 334 with respect to N tomographic images corresponding to the number of scans of N times. However, there may be an embodiment in which noise reduction is performed for only a part of the N pieces instead of noise reduction for all of them. When the execution processing time of the noise reduction unit 334 is long, the total processing time can be shortened by performing noise reduction on a part of N noises rather than on all of them. Further, in the present embodiment, the noise detection unit 3341 has detected the position of the black spot, but in order to save the time for detecting the position of the black spot, the position of the black spot may be predicted by a random number. In this case, since the position generated by the random number is not always the black dot, the random number is generated a plurality of times and the noise is reduced, and the results are added and averaged to output as a noise-reduced tomographic image. FIG. 6 is a block diagram thereof. Image 1 of FIG. 6 is the first process, and is an image obtained by determining the position of the black spot by random number generation and minimizing the evaluation function in the objective function minimization unit as described in the first embodiment. Is. Image 2 of FIG. 6 is the second process, in which the position of the black spot is determined by random number generation as in the first process, and the evaluation function is minimized by the objective function minimization unit as described in the first embodiment. It is an image obtained by. This is repeated M times (repeated a plurality of times) to obtain images 1 to M. Finally, images 1 to M are added and averaged and output as a noise-reduced tomographic image. In this embodiment, a random number is generated when predicting the position of noise, and at this time, the ratio of noise to the total number of pixels is set as a fixed value. As another embodiment, the ratio of noise to the total number of pixels may be changed according to the control parameters at the time of photographing by the OCT apparatus.

(Third Embodiment)
Hereinafter, the third embodiment will be described. The image processing system including the image processing device according to the present embodiment has a part in common with the first embodiment and the second embodiment. In the following, only the differences will be described.

In the first and second embodiments, the objective function used in the objective function minimization unit 3344 is V1 of Equation 2, but in this embodiment, instead of the output image f (x, y), the field of deep learning It is generated using a CAE (Functional Autoencoder) known as. The input given to the CAE is the input image g (x, y). As another embodiment, an image generation function that inputs an image and outputs an image may be used instead of CAE, and for example, AE (Autoencoder) can be considered. For the following explanation, CAE and the above-mentioned image generation function are collectively referred to as a function h (Image, x, y).

The function h is a function that inputs an image image and generates an image having the same size as the input image g (x, y) of the evaluation function V1. When the generated image is called an image h_out for convenience, the pixel value at the positions x and y of the image h_out is h (Image, x, y). The pixel values at the positions x and y of the image h_out depend on the plurality of pixel values of the image image and are calculated by a nonlinear function using the plurality of pixels of the image image, and this nonlinear function changes the shape of the function. Has multiple parameters for. As described above, the function h has a plurality of parameters, and when the parameters are changed, the image h_out is changed. The objective function minimization unit in the case of the present embodiment does not change the pixels of the output image f (x, y) as in the first and second embodiments, but changes the parameters constituting the function h. This indirectly changes the pixels of the output image f (x, y). The output images f (x, y) of Equations 2 to 4 store the values related to the positions x and y, and the objective function minimization unit 3344 of the first and second embodiments stores the stored values. The objective function was minimized by changing it. On the other hand, in the present embodiment, the objective function is minimized by changing the parameters constituting the function h. The method of minimizing using CAE is known in the field of deep learning, and there are various methods, but this embodiment uses the steepest descent method. According to the present embodiment, by using CAE, it is possible to add a non-negative constraint condition, that is, a condition that h (Image, x, y) ≥ 0. For example, it can be realized by using a sigmoid function for the activation function of the final layer of CAE. Since learning in deep learning is to minimize the objective function, it can be said that this embodiment is also learning in that sense, but at least the following things are different. First, the present embodiment does not perform learning by giving a large amount of teacher data as is known in deep learning. In this embodiment, there is only one data corresponding to the teacher data of deep learning, which is the input image g (x, y). Secondly, deep learning has a learning processing step and an inference processing step, but this embodiment has only a learning processing step and no inference processing step. In this embodiment, the evaluation function V1 is used as the objective function, and the function h (Image, x, y) is used instead of the output image f (x, y). It may be used as a function and generated by using the function h (Image, x, y) instead of the output image f (x, y).

(Fourth Embodiment)
The noise reduction unit 334 in the image processing apparatus according to the present embodiment is an example of a high image quality improvement means (noise reduction means) for improving the image quality (noise reduction) of the tomographic image, and is a process by addition averaging in the above-described embodiment (noise reduction means). Instead of (synthesis processing), high image quality processing by machine learning is applied. The noise reduction unit 334 is also referred to as a high image quality unit. At this time, the high image quality section inputs low image quality tomographic images generated from a small number of tomographic images into the machine learning model, so that the image quality is equivalent to that obtained from a large number of tomographic images (low noise). And generate a tomographic image (with high contrast). Here, the machine learning model according to the present embodiment is input data which is a low-quality image acquired under predetermined shooting conditions assumed as a processing target, and output data which is a high-quality image corresponding to the input data. It refers to a function generated by performing machine learning using training data composed of pairs of (correct answer data). The predetermined shooting conditions include a shooting portion, a shooting method, a shooting angle of view, an image size, and the like.

Here, the low-quality tomographic image is acquired as follows, for example. First, when the operator operates the input unit 700 and presses the shooting start (Capture) button on the shooting screen (preview screen), OCT shooting under the shooting conditions set according to the instruction from the operator is started. Will be done. At this time, the instruction unit 304 instructs the tomographic imaging apparatus 200 to perform OCT imaging based on the setting instructed by the operator, and the tomographic imaging apparatus 200 acquires the corresponding OCT tomographic image. The tomographic image capturing apparatus 200 also acquires an SLO image and executes tracking processing based on the SLO moving image. Here, the setting of the imaging conditions is, for example, 1) registration of the Macular Disease inspection set, 2) selection of the OCT scan mode, 3) setting of the following imaging parameters, and the like. The shooting parameters include, for example, 3-1) scanning pattern: 300A scan (book) x 300B scan (sheets), 3-2) scanning area size: 6 x 6 mm, 3-3) main scanning direction: horizontal direction. , Etc. are set. Further, as imaging parameters, for example, 3-4) scanning interval: 0.01 mm, 3-5) fixation lamp position: macula (fovea centralis), 3-6) coherence gate position: glass side, 3- 7) Default display report type: Monocular examination report, etc. are set. Then, the display control unit 305 causes the display unit 600 to display the acquired tomographic image, information on the imaging conditions, and the like. At this time, in the present embodiment, the operator operates the input unit 700 and presses a high image quality button (not shown) on the display screen (report screen for displaying the analysis result or the like), thereby causing the high image quality unit. Shall carry out high image quality processing on the tomographic image. That is, the high image quality button is a button for instructing the execution of the high image quality processing. Of course, the high image quality button may be a button for instructing the display of the high image quality image (generated before pressing the high image quality button).

In the present embodiment, the input data used as the training data is a low-quality tomographic image generated from a single cluster having a small number of tomographic images. The output data (correct answer data) used as the training data is a high-quality tomographic image obtained by adding and averaging a plurality of aligned tomographic images. The output data used as the training data is not limited to this, and may be, for example, a high-quality tomographic image generated from a single cluster composed of a large number of tomographic images. Further, the output data used as the training data may be a high-quality tomographic image obtained by setting a tomographic image having a higher resolution (higher magnification) than the input image to the same resolution (same magnification) as the input image. The pair of the input image and the output image used for training the machine learning model is not limited to the above, and any known image combination may be used. For example, an image obtained by adding the first noise component to a tomographic image acquired by the tomographic image capturing device 200 or another device is used as an input image, and the tomographic image (acquired by the tomographic imaging device 200 or another device) is used as an input image. An image to which a second noise component (different from the first noise component) is added may be used as an output image for training a machine learning model. That is, the image quality improving unit may improve the image quality of the tomographic image input as the input image by using the trained model for high image quality obtained by learning the learning data including the tomographic image of the fundus. Anything is fine.

FIG. 8 shows a configuration example of a machine learning model in the image quality improving unit according to the present embodiment. The machine learning model is a convolutional neural network (CNN), and is composed of a plurality of layers responsible for processing input value groups and outputting them. The types of layers included in the above configuration include a convolution layer, a downsampling layer, an upsampling layer, and a composite layer. The convolution layer is a layer that performs convolution processing on the input value group according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image. The downsampling layer is a process of reducing the number of output value groups to be smaller than the number of input value groups by thinning out or synthesizing the input value groups. Specifically, for example, there is a Max Polling process. The upsampling layer is a process of increasing the number of output value groups to be larger than the number of input value groups by duplicating the input value group or adding the interpolated value from the input value group. Specifically, for example, there is a linear interpolation process. The composite layer is a layer in which a value group such as an output value group of a certain layer or a pixel value group constituting an image is input from a plurality of sources, and the processing is performed by concatenating or adding them. In such a configuration, the value group in which the pixel value group constituting the input image 1301 is output through the convolution processing block and the pixel value group constituting the input image 1301 are combined in the composite layer. After that, the combined pixel value group is formed into a high-quality image 1302 in the final convolution layer. Although not shown, as an example of changing the configuration of the CNN, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolution layer. You may. Although the image to be processed is described as a two-dimensional image in FIG. 8 for the sake of simplicity, the present invention is not limited to this. For example, the present invention also includes a case where a three-dimensional low-quality tomographic image is input to the high-quality image-enhancing unit and a three-dimensional high-quality tomographic image is output.

Here, the GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed a plurality of times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in the present embodiment, the GPU is used in addition to the CPU for the processing by the image processing unit 303, which is an example of the learning unit (not shown). Specifically, when executing a learning program including a learning model, learning is performed by the CPU and the GPU collaborating to perform calculations. The processing of the learning unit may be performed only by the CPU or GPU. Further, the GPU may be used for the high image quality unit as well as the learning unit. Further, the learning unit may include an error detecting unit and an updating unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error backpropagation method is a method of adjusting the coupling weighting coefficient between the nodes of each neural network so that the above error becomes small.

In addition, the operator can instruct the start of OCT measurement processing (for example, various measurement processing such as analysis result generation, diagnosis result generation, object recognition, segmentation, etc., as described later) using the input unit 700. For example, in response to an instruction from the operator, various measurement processes as described later can be executed using the tomographic image whose image quality has been improved by the trained model for image quality improvement. The processing accuracy can be improved.

In the above-described embodiment, the tomographic image has been described, but the present invention is not limited to this. The image related to processing such as display, high image quality, and image analysis according to the present embodiment may be a front image, a motion contrast image, or the like. Further, not only a tomographic image but also a different image such as an SLO image, a fundus photograph, or a fluorescent fundus photograph may be used. In that case, the user interface for executing the high image quality processing is one that instructs the execution of the high image quality processing for a plurality of different types of images, and an arbitrary image is selected from a plurality of different types of images. There may be something that instructs the execution of the high image quality processing.

With such a configuration, the display control unit 305 can display the image processed by the image quality improving unit according to the present embodiment on the display unit 600. At this time, as described above, when at least one of a plurality of conditions relating to the display of the high-quality image, the display of the analysis result, the depth range of the displayed front image, and the like is selected, the display screen is displayed. Even if it is transitioned, the selected state may be maintained.

Further, as described above, when at least one of the plurality of conditions is in the selected state, the selected state of at least one is maintained even if the other conditions are changed to the selected state. You may. For example, the display control unit 305 analyzes a low-quality image in response to an instruction from the examiner (for example, when a high-quality button (not shown) is specified) when the display of the analysis result is in the selected state. The display of the result may be changed to the display of the analysis result of the high-quality image. Further, the display control unit 305 responds to an instruction from the examiner (for example, when the designation of the high image quality button (not shown) is canceled) when the display of the analysis result is in the selected state, and the display control unit 305 produces a high image quality image. The display of the analysis result of is changed to the display of the analysis result of the low-quality image.

Further, the display control unit 305 receives an instruction from the examiner (for example, when the designation of displaying the analysis result is canceled) when the display of the high-quality image is not selected, and the display control unit 305 determines the display of the low-quality image. The display of the analysis result may be changed to the display of a low-quality image. Further, the display control unit 305 displays the low-quality image in response to an instruction from the examiner (for example, when the display of the analysis result is specified) when the display of the high-quality image is not selected. The display may be changed to display the analysis result of a low-quality image. Further, the display control unit 305 analyzes the high-quality image in response to an instruction from the examiner (for example, when the designation of displaying the analysis result is canceled) when the display of the high-quality image is in the selected state. The display of the result may be changed to the display of a high-quality image. Further, when the display of the high-quality image is selected, the display control unit 305 increases the display of the high-quality image according to the instruction from the examiner (for example, when the display of the analysis result is specified). You may change to display the analysis result of the image quality image.

Further, consider the case where the display of the high-quality image is in the non-selected state and the display of the analysis result of the first type is in the selected state. In this case, the display control unit 305 receives the instruction from the examiner (for example, when the display of the analysis result of the second type is specified), and the display control unit 305 determines the analysis result of the first type of the low-quality image. The display may be changed to display the analysis result of the second type of the low image quality image. Further, consider the case where the display of the high-quality image is in the selected state and the display of the analysis result of the first type is in the selected state. In this case, the display control unit 305 receives the instruction from the examiner (for example, when the display of the second type of analysis result is specified), and the display control unit 305 determines the analysis result of the first type of the high-quality image. The display may be changed to display the analysis result of the second type of the high-quality image.

As described above, the display screen for follow-up observation may be configured so that these display changes are collectively reflected on a plurality of images obtained at different dates and times. Here, the analysis result may be displayed by superimposing the analysis result on the image with arbitrary transparency. At this time, the change to the display of the analysis result may be changed to a state in which the analysis result is superimposed on the displayed image with arbitrary transparency, for example. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with an arbitrary transparency.

Further, in the above-described embodiment, the display control unit 305 causes the display unit 600 to display an image selected according to an instruction from the examiner among the high-quality image and the input image generated by the high-quality image unit. be able to. Further, the display control unit 305 may switch the display on the display unit 600 from a captured image (input image) to a high-quality image in response to an instruction from the examiner. That is, the display control unit 305 may change the display of the low-quality image to the display of the high-quality image in response to an instruction from the examiner. Further, the display control unit 305 may change the display of the high-quality image to the display of the low-quality image in response to an instruction from the examiner.

Furthermore, the high image quality section starts the high image quality processing (input of the image to the high image quality engine) by the high image quality engine (learned model for high image quality) according to the instruction from the examiner. Then, the display control unit 305 may display the high-quality image generated by the high-quality image-enhancing unit on the display unit 600. On the other hand, when the input image is captured by the imaging device (tomographic image capturing device 200), the high image quality engine automatically generates a high image quality image based on the input image, and the display control unit 305 causes the examiner to perform the image quality. A high-quality image may be displayed on the display unit 600 in response to an instruction from. Here, the high image quality engine includes a trained model that performs the above-mentioned image quality improvement processing (high image quality processing).

Note that these processes can be performed in the same manner for the output of the analysis result. That is, the display control unit 305 may change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image in response to the instruction from the examiner. Further, the display control unit 305 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image in response to an instruction from the examiner. Of course, the display control unit 305 may change the display of the analysis result of the low-quality image to the display of the low-quality image in response to the instruction from the examiner. Further, the display control unit 305 may change the display of the low-quality image to the display of the analysis result of the low-quality image in response to an instruction from the examiner. Further, the display control unit 305 may change the display of the analysis result of the high-quality image to the display of the high-quality image in response to an instruction from the examiner. Further, the display control unit 305 may change the display of the high-quality image to the display of the analysis result of the high-quality image in response to an instruction from the examiner.

Further, the display control unit 305 may change the display of the analysis result of the low-quality image to the display of the analysis result of another type of the low-quality image in response to the instruction from the examiner. Further, the display control unit 305 may change the display of the analysis result of the high-quality image to the display of the analysis result of another type of the high-quality image in response to the instruction from the examiner.

Here, the display of the analysis result of the high-quality image may be a superposition display of the analysis result of the high-quality image on the high-quality image with arbitrary transparency. Further, the display of the analysis result of the low-quality image may be obtained by superimposing the analysis result of the low-quality image on the low-quality image with arbitrary transparency. At this time, the change to the display of the analysis result may be changed to a state in which the analysis result is superimposed on the displayed image with arbitrary transparency, for example. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with an arbitrary transparency.

(Modification example 1)
In the various embodiments described above, the detection or prediction of the position of the black spot, which is an example of noise, may be performed using a machine learning model for object recognition by machine learning or a machine learning model for segmentation. At this time, the position of noise is detected or predicted by using the learned model obtained by learning the learning data including the medical image of the subject from the medical image obtained by photographing. Here, the machine learning model according to this modified example is a pair of input data which is a tomographic image which is an example of a medical image and data (correct answer data) in which the position of noise in the tomographic image is labeled (annotated). It refers to a function generated by performing machine learning using training data composed of groups. The above-mentioned various methods as shown in FIG. 8 and the like can be applied to the machine learning model, the processing of the learning unit, and the like. In addition, the above-mentioned noise position may be detected or predicted by a method of detecting an abnormal portion using a hostile generation network or an autoencoder described later.

(Modification 2)
The display control unit 305 in the various embodiments and modifications described above may display analysis results such as a desired layer thickness and various blood vessel densities on the report screen of the display screen. In addition, the optic nerve head, macular region, vascular region, nerve fiber bundle, vitreous region, macular region, choroidal region, scleral region, lamina cribrosa region, retinal layer boundary, retinal layer boundary edge, photoreceptor cells, blood cells, The value (distribution) of the parameter relating to the site of interest including at least one such as the vascular wall, the vascular inner wall boundary, the vascular lateral boundary, the ganglion cell, the corneal region, the corner region, and Schlemm's canal may be displayed as the analysis result. At this time, for example, by analyzing a medical image to which various artifact reduction processes are applied, it is possible to display an accurate analysis result. The artifact is, for example, a false image region generated by light absorption by a blood vessel region or the like, a projection artifact, a band-shaped artifact in a front image generated in the main scanning direction of the measured light depending on the state of the eye to be inspected (movement, blinking, etc.), or the like. There may be. Further, the artifact may be any image loss region as long as it is randomly generated for each image taken on a medical image of a predetermined portion of the subject, for example. In addition, the value (distribution) of the parameter relating to the region including at least one of the various artifacts (copy loss region) as described above may be displayed as the analysis result. Further, the value (distribution) of the parameter relating to the region including at least one such as an abnormal site such as drusen, new blood vessel, vitiligo (hard vitiligo), and pseudo-drusen may be displayed as an analysis result.

Further, the analysis result may be displayed as an analysis map, a sector showing statistical values corresponding to each divided area, or the like. The analysis result may be generated by using a trained model (analysis result generation engine, trained model for analysis result generation) obtained by learning the analysis result of the medical image as training data. .. At this time, the trained model is trained using training data including a medical image and an analysis result of the medical image, training data including a medical image and an analysis result of a medical image of a type different from the medical image, and the like. It may be obtained by. Further, the trained model is obtained by training using training data including input data in which a plurality of medical images of different types of predetermined parts are set, such as a luminance front image and a motion contrast front image. May be good. Here, the luminance front image corresponds to the En-Face image of the tomographic image, and the motion contrast front image corresponds to the En-Face image of OCTA. Further, the analysis result obtained by using the high-quality image generated by the trained model for high image quality may be displayed. The trained model for high image quality is obtained by learning training data using the first image as input data and the second image having higher image quality than the first image as correct answer data. You may. At this time, the second image is subjected to, for example, a superposition process of a plurality of first images (for example, an averaging process of a plurality of first images obtained by alignment) to increase the contrast and reduce noise. It may be a high-quality image in which the above is performed.

Further, the input data included in the training data may be a high-quality image generated by the trained model for high image quality, or may be a set of a low-quality image and a high-quality image. Further, the training data includes, for example, at least analysis values (for example, average value, median value, etc.) obtained by analyzing the analysis area, a table including the analysis values, an analysis map, the position of the analysis area such as a sector in the image, and the like. The information including one may be the data labeled (annotated) with the input data as the correct answer data (for supervised learning). In addition, according to the instruction from the examiner, the analysis result obtained by the trained model for generating the analysis result may be displayed. For example, the image processing unit 303 uses the trained model for generating the analysis result (different from the trained model for improving the image quality) to obtain the analysis result related to the noise-reduced image from the above-mentioned noise-reduced image. Can be generated. Further, for example, the display control unit 305 is related to the noise reduction image generated from the noise reduction image described above by using the learned model for generating the analysis result obtained by learning the learning data including the medical image of the subject. The analysis result to be performed can be displayed on the display unit 600.

In addition, the display control unit 305 in the various embodiments and modifications described above may display various diagnostic results such as glaucoma and age-related macular degeneration on the report screen of the display screen. At this time, for example, by analyzing a medical image to which various artifact reduction processes as described above are applied, it is possible to display an accurate diagnostic result. Further, as the diagnosis result, the position of the specified abnormal part or the like may be displayed on the image, or the state of the abnormal part or the like may be displayed by characters or the like. In addition, the classification result of the abnormal part or the like (for example, Curtin classification) may be displayed as the diagnosis result. Further, as the classification result, for example, information indicating the certainty of each abnormal part (for example, a numerical value indicating a ratio) may be displayed. In addition, information necessary for the doctor to confirm the diagnosis may be displayed as a diagnosis result. As the necessary information, for example, advice such as additional shooting can be considered. For example, when an abnormal site is detected in the blood vessel region in the OCTA image, it may be displayed that fluorescence imaging using a contrast medium capable of observing the blood vessel in more detail than OCTA is performed.

The diagnosis result may be generated by using a trained model (diagnosis result generation engine, trained model for generation of diagnosis result) obtained by learning the diagnosis result of the medical image as training data. .. In addition, the trained model is based on training using training data including a medical image and a diagnosis result of the medical image, and training data including a medical image and a diagnosis result of a medical image of a type different from the medical image. It may be obtained. Further, the diagnostic result obtained by using the high-quality image generated by the trained model for high image quality may be displayed.

Further, the input data included in the training data may be a high-quality image generated by the trained model for high image quality, or may be a set of a low-quality image and a high-quality image. In addition, the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Information including at least one such as (general medical support information, etc.) and grounds for denying the diagnosis name (negative medical support information), etc. are labeled (annotated) in the input data as correct answer data (for supervised learning). It may be data. In addition, according to the instruction from the examiner, the diagnosis result obtained by the trained model for generating the diagnosis result may be displayed. For example, the image processing unit 303 uses a trained model for generating a diagnostic result (different from the trained model for improving image quality) to obtain a diagnostic result related to the noise-reduced image from the above-mentioned noise-reduced image. Can be generated. Further, for example, the display control unit 305 is related to the noise reduction image generated from the noise reduction image described above by using the learned model for generating the diagnosis result obtained by learning the learning data including the medical image of the subject. The diagnosis result to be performed can be displayed on the display unit 600.

Further, the display control unit 305 in the various embodiments and modifications described above displays the object recognition result (object detection result) and the segmentation result of the above-mentioned attention portion, artifact, abnormal portion, etc. on the report screen of the display screen. It may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, colors and the like may be superimposed and displayed on the object in the image. The object recognition result and the segmentation result are learned models (object recognition engine, for object recognition) obtained by learning the learning data obtained by labeling (annotating) the medical image with the information indicating the object recognition and the segmentation as the correct answer data. It may be generated using a trained model, a segmentation engine, a trained model for segmentation). The above-mentioned analysis result generation and diagnosis result generation may be obtained by using the above-mentioned object recognition result and segmentation result. For example, analysis result generation and diagnosis result generation processing may be performed on a partial region such as a region of interest obtained by object recognition or segmentation processing. For example, the image processing unit 303 uses a trained model for object recognition (different from the trained model for improving image quality) to obtain an object recognition result related to the noise-reduced image from the above-mentioned noise-reduced image. Can be generated. Further, for example, the image processing unit 303 uses a trained model for segmentation (different from the trained model for high image quality) to obtain a segmentation result related to the noise-reduced image from the above-mentioned noise-reduced image. Can be generated. Further, for example, the display control unit 305 displays the object recognition result generated from the above-mentioned noise reduction image by using the learned model for object recognition obtained by learning the learning data including the medical image of the subject. Can be displayed on. Further, for example, the display control unit 305 displays the segmentation result generated from the above-mentioned noise reduction image on the display unit 600 by using the learned model for segmentation obtained by learning the learning data including the medical image of the subject. Can be made to. At this time, the object recognition result and the segmentation result are, for example, partial regions detected from the noise reduction image.

In addition, when detecting an abnormal site, a hostile generative network (GAN: Generative Adversarial Networks) or a variational autoencoder (VAE: Variational Auto-Encoder) may be used. For example, a DCGAN (Deep Convolutional GAN) consisting of a generator obtained by learning the generation of a tomographic image and a discriminator obtained by learning the discrimination between a new tomographic image generated by the generator and a real frontal image of the fundus of the eye. ) Can be used as a machine learning model.

When DCGAN is used, for example, the discriminator encodes the input tomographic image into a latent variable, and the generator generates a new tomographic image based on the latent variable. After that, the difference between the input tomographic image and the generated new tomographic image can be extracted (detected) as an abnormal part. When VAE is used, for example, the input tomographic image is encoded by an encoder to be a latent variable, and the latent variable is decoded by a decoder to generate a new tomographic image. After that, the difference between the input tomographic image and the generated new tomographic image can be extracted (detected) as an abnormal part. Although a tomographic image has been described as an example of the input data, a fundus image, a frontal image of the anterior eye, or the like may be used.

Further, the information processing apparatus may detect an abnormal portion by using a convolutional autoencoder (CAE). When CAE is used, the same image is learned as input data and output data at the time of learning. As a result, when an image with an abnormal part is input to CAE at the time of estimation, an image without an abnormal part is output according to the learning tendency. After that, the difference between the image input to the CAE and the image output from the CAE can be extracted (detected) as an abnormal portion. In this case as well, not only the tomographic image but also the fundus image, the frontal image of the anterior eye, and the like may be used as input data.

In these cases, the information processing apparatus uses information on the difference between the medical image obtained by using the hostile generation network or the autoencoder and the medical image input to the hostile generation network or the autoencoder as information on the abnormal part. Can be generated. As a result, the information processing apparatus can be expected to detect the abnormal portion at high speed and with high accuracy. Here, the autoencoder includes VAE, CAE, and the like. For example, the image processing unit 303 can generate information on the difference between the medical image obtained by using the hostile generation network or the autoencoder from the above-mentioned noise reduction image and the noise reduction image as information on the abnormal portion. .. Further, for example, the display control unit 305 uses the information on the difference between the medical image obtained from the above-mentioned noise reduction image by using the hostile generation network or the autoencoder and the noise reduction image as the information on the abnormal portion 600. Can be displayed on.

Further, in the diseased eye, the image features differ depending on the type of disease. Therefore, the trained models used in the various embodiments and modifications described above may be generated and prepared for each type of disease or for each abnormal site. In this case, for example, the image processing device 300 can select a trained model to be used for processing according to an input (instruction) of the type of disease of the eye to be inspected, an abnormal part, or the like from the operator. The trained model prepared for each type of disease or abnormal site is not limited to the trained model used for detecting the retinal layer and generating a region label image, for example, for an engine for image evaluation or for analysis. It may be a trained model used in the engine of the above. At this time, the image processing device 300 may identify the type of disease or abnormal site of the eye to be inspected from the image by using a separately prepared trained model. In this case, the image processing device 300 may automatically select the trained model to be used for the above processing based on the type of disease or abnormal site identified by using the trained model prepared separately. it can. The trained model for identifying the disease type and abnormal site of the eye to be inspected is the training data in which the tomographic image, the fundus image, etc. are input data, and the disease type and the abnormal site in these images are output data. Learning may be performed using pairs. Here, as the input data of the training data, a tomographic image, a fundus image, or the like may be used alone as input data, or a combination thereof may be used as input data.

Further, in particular, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a set of a plurality of medical images of different types of the predetermined part of the subject. Good. At this time, as the input data included in the training data, for example, input data in which a motion contrast front image of the fundus and a luminance front image (or a luminance tom image) are set can be considered. Further, as the input data included in the training data, for example, input data in which a tomographic image (B scan image) of the fundus and a color fundus image (or a fluorescent fundus image) are set can be considered. Further, the plurality of medical images of different types may be any image as long as they are acquired by different modality, different optical systems, different principles, or the like.

Further, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a plurality of medical images of different parts of the subject. At this time, as the input data included in the training data, for example, input data in which a tomographic image of the fundus of the eye (B scan image) and a tomographic image of the anterior segment of the eye (B scan image) are considered as a set can be considered. Further, as the input data included in the training data, for example, input data in which a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic nerve head of the fundus are set as a set, etc. Is also possible.

The input data included in the learning data may be different parts of the subject and a plurality of different types of medical images. At this time, the input data included in the training data may be, for example, input data in which a tomographic image of the anterior segment of the eye and a color fundus image are set. Further, the trained model described above may be a trained model obtained by learning from training data including input data including a set of a plurality of medical images having different shooting angles of view of a predetermined portion of the subject. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a predetermined portion into a plurality of regions, such as a panoramic image. At this time, by using a wide angle-of-view image such as a panoramic image as learning data, there is a possibility that the feature amount of the image can be accurately acquired because the amount of information is larger than that of the narrow angle-of-view image. The result of processing can be improved. For example, when an abnormal portion is detected at a plurality of positions in a wide angle-of-view image at the time of estimation (prediction), an enlarged image of each abnormal portion can be sequentially displayed. As a result, it is possible to efficiently confirm the abnormal portion at a plurality of positions, so that the convenience of the examiner can be improved, for example. At this time, for example, the examiner may be configured to select each position on the wide angle-of-view image in which the abnormal portion is detected, and an enlarged image of the abnormal portion at the selected position may be displayed. .. Further, the input data included in the learning data may be input data in which a plurality of medical images of different dates and times of a predetermined part of the subject are set.

Further, the display screen on which at least one of the above-mentioned analysis result, diagnosis result, object recognition result, and segmentation result is displayed is not limited to the report screen. Such a display screen is, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). It may be displayed in. For example, by displaying at least one result obtained by using the above-mentioned trained model on the shooting confirmation screen, the examiner can confirm the accurate result even immediately after shooting. Further, the above-mentioned change in the display between the low-quality image and the high-quality image may be, for example, a change in the display between the analysis result of the low-quality image and the analysis result of the high-quality image.

Here, the various trained models described above can be obtained by machine learning using the training data. Machine learning includes, for example, deep learning consisting of a multi-layer neural network. Further, for at least a part of the multi-layer neural network, for example, a convolutional neural network (CNN) can be used as a machine learning model. Further, a technique related to an autoencoder (self-encoder) may be used for at least a part of a multi-layer neural network. Further, a technique related to backpropagation (error backpropagation method) may be used for learning. However, the machine learning is not limited to deep learning, and any learning using a model capable of extracting (expressing) the features of learning data such as images by learning may be used. Here, the machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. The trained model is a model in which a machine learning model by an arbitrary machine learning algorithm is trained (learned) in advance using appropriate learning data. However, the trained model does not require further learning, and additional learning can be performed. The learning data is composed of a pair of input data and output data (correct answer data). Here, the learning data may be referred to as teacher data, or the correct answer data may be referred to as teacher data.

The GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed a plurality of times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in this modification, the GPU is used in addition to the CPU for the processing by the image processing unit 303, which is an example of the learning unit (not shown). Specifically, when executing a learning program including a learning model, learning is performed by the CPU and the GPU collaborating to perform calculations. The processing of the learning unit may be performed only by the CPU or GPU. Further, the processing unit (estimation unit) that executes the processing using the various trained models described above may also use the GPU in the same manner as the learning unit. Further, the learning unit may include an error detecting unit and an updating unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error backpropagation method is a method of adjusting the coupling weighting coefficient between the nodes of each neural network so that the above error becomes small.

In addition, as machine learning models used for high image quality and segmentation, there are an encoder function consisting of a plurality of layers including a plurality of downsampling layers and a decoder function consisting of a plurality of layers including a plurality of upsampling layers. A U-net type machine learning model having the above is applicable. In the U-net type machine learning model, position information (spatial information) that is ambiguous in a plurality of layers configured as encoders is displayed in layers of the same dimension (layers corresponding to each other) in a plurality of layers configured as a decoder. ) (For example, using a skip connection).

Further, as a machine learning model used for high image quality, segmentation, etc., for example, FCN (Full Convolutional Network), SegNet, or the like can be used. Further, a machine learning model that recognizes an object in a region unit according to a desired configuration may be used. As a machine learning model for performing object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Further, YOLO (You Only Look None) or SSD (Single Shot Detector) can be used as a machine learning model for recognizing an object in a region unit.

Further, the machine learning model may be, for example, a capsule network (Capsule Network; CapsNet). Here, in a general neural network, each unit (each neuron) is configured to output a scalar value, so that, for example, spatial information regarding the spatial positional relationship (relative position) between features in an image can be obtained. It is configured to be reduced. Thereby, for example, learning can be performed so as to reduce the influence of local distortion and translation of the image. On the other hand, in the capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is retained. Thereby, for example, learning can be performed in which the spatial positional relationship between the features in the image is taken into consideration.

Further, the high image quality engine (trained model for high image quality) may be a trained model obtained by additionally learning learning data including at least one high image quality image generated by the high image quality engine. Good. At this time, it may be configured so that whether or not to use the high-quality image as learning data for additional learning can be selected by an instruction from the examiner. It should be noted that these configurations can be applied not only to the trained model for high image quality but also to the various trained models described above. Further, in the generation of the correct answer data used for learning the various trained models described above, the trained model for generating the correct answer data for generating the correct answer data such as labeling (annotation) may be used. At this time, the trained model for generating correct answer data may be obtained by (sequentially) additionally learning the correct answer data obtained by labeling (annotation) by the examiner. That is, the trained model for generating correct answer data may be obtained by additional training of training data in which the data before labeling is used as input data and the data after labeling is used as output data. Further, in a plurality of consecutive frames such as a moving image, the result of the frame determined to have low accuracy of the result is corrected in consideration of the results of object recognition and segmentation of the preceding and following frames. May be good. At this time, according to the instruction from the examiner, the corrected result may be additionally learned as correct answer data.

(Modification example 3)
In the various embodiments and modifications described above, when a region (partial region) of an eye to be inspected is detected using a trained model for object recognition or a trained model for segmentation, a predetermined region is determined for each detected region. Image processing can also be applied. For example, consider the case of detecting at least two regions of the vitreous region, the retinal region, and the choroid region. In this case, when performing image processing such as contrast adjustment on at least two detected regions, adjustments suitable for each region can be performed by using different image processing parameters. By displaying an image adjusted suitable for each area, the operator can more appropriately diagnose a disease or the like in each area. Note that the configuration using different image processing parameters for each detected region may be similarly applied to the region of the eye to be inspected detected without using the trained model, for example.

(Modification example 4)
In the preview screens in the various embodiments and modifications described above, the trained model described above may be used for at least one frame of the live moving image. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the trained model corresponding to each live moving image may be used. As a result, for example, even if it is a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of shooting. Therefore, for example, the failure of re-imaging can be reduced, so that the accuracy and efficiency of diagnosis can be improved. The plurality of live moving images may be, for example, a moving image of the anterior segment for alignment in the XYZ direction, and a frontal moving image of the fundus for focusing adjustment or OCT focus adjustment of the fundus observation optical system. Further, the plurality of live moving images may be, for example, a tomographic moving image of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measured optical path length and the reference optical path length). At this time, the above-mentioned various adjustments may be performed so that the region detected by using the above-described trained model for object recognition or the above-mentioned trained model for segmentation satisfies a predetermined condition. For example, a value (for example, a contrast value or an intensity value) related to a predetermined retinal layer such as a vitreous region or RPE detected by using a trained model for object recognition or a trained model for segmentation exceeds a threshold value (for example, a contrast value or an intensity value). Alternatively, various adjustments such as OCT focus adjustment may be performed so as to reach a peak value). Further, for example, the OCT so that a predetermined retinal layer such as a vitreous region or RPE detected by using a trained model for object recognition or a trained model for segmentation is at a predetermined position in the depth direction. The coherence gate adjustment may be configured to be performed.

Further, the moving image to which the above-described trained model can be applied is not limited to the live moving image, and may be, for example, a moving image stored (stored) in the storage unit 302. At this time, for example, the moving image obtained by aligning each frame of the tomographic moving image of the fundus stored (stored) in the storage unit 302 may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous region, a reference frame may be selected based on conditions such as the presence of the vitreous region on the frame as much as possible. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) sequentially generated by the trained model for high image quality may be continuously displayed for each at least one frame of the moving image.

As the above-mentioned alignment method between frames, the same method may be applied to the alignment method in the X direction and the alignment method in the Z direction (depth direction), and all different methods may be applied. May be applied. Further, the alignment in the same direction may be performed a plurality of times by different methods, and for example, a precise alignment may be performed after performing a rough alignment. Further, as a method of alignment, for example, a plurality of alignments obtained by segmenting a tomographic image (B scan image) using a retinal layer boundary obtained by performing segmentation processing (coarse in the Z direction) and dividing the tomographic image. Alignment using the correlation information (similarity) between the region and the reference image (precise in the X and Z directions), and the one-dimensional projection image generated for each tomographic image (B scan image) was used (in the X direction). ) Alignment, etc. Alignment (in the X direction) using a two-dimensional front image. Further, it may be configured so that the alignment is roughly performed in pixel units and then the precise alignment is performed in subpixel units.

Here, during various adjustments, it is possible that the imaged object such as the retina of the eye to be inspected has not yet been successfully imaged. Therefore, since there is a large difference between the medical image input to the trained model and the medical image used as the training data, there is a possibility that a high-quality image cannot be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the display of the high-quality moving image (continuous display of the high-quality frame) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the image quality enhancement button may be configured to be changed to a state (active state) that can be specified by the examiner. The high image quality button is a button for designating the execution of the high image quality processing. Of course, the high image quality button may be a button for instructing the display of a high image quality image.

Further, a trained model for high image quality is prepared for each shooting mode having a different scanning pattern and the like, and the trained model for high image quality corresponding to the selected shooting mode is selected. May be good. Further, one trained model for high image quality obtained by learning learning data including various medical images obtained in different imaging modes may be used.

(Modification 5)
In the various embodiments and modifications described above, when the trained model is undergoing additional learning, it may be difficult to output (infer / predict) using the trained model itself during the additional learning. Therefore, it is preferable to prohibit the input of the medical image to the trained model during the additional learning. Further, another trained model that is the same as the trained model being additionally trained may be prepared as another preliminary trained model. At this time, during the additional learning, it is preferable to enable the input of the medical image to the preliminary trained model. Then, after the additional learning is completed, the trained model after the additional learning is evaluated, and if there is no problem, the preliminary trained model may be replaced with the trained model after the additional learning. Also, if there is a problem, a preliminary trained model may be used. As the evaluation of the trained model, for example, a trained model for classification for classifying a high-quality image obtained by the trained model for high image quality with another type of image may be used. The trained model for classification uses, for example, a plurality of images including a high-quality image and a low-quality image obtained by the trained model for high image quality as input data, and the types of these images are labeled (annotation). It may be a trained model obtained by training training data including the obtained data as correct answer data. At this time, the image type of the input data at the time of estimation (prediction) is displayed together with the information (for example, a numerical value indicating the ratio) indicating the certainty of each type of image included in the correct answer data at the time of learning. May be good. In addition to the above images, the input data of the trained model for classification includes overlay processing of a plurality of low-quality images (for example, averaging processing of a plurality of low-quality images obtained by alignment) and the like. It may include a high-quality image in which high contrast and noise reduction are performed.

In addition, the trained model obtained by learning for each imaging site may be selectively used. Specifically, learning including a first learned model obtained by using learning data including a first imaging site (lung, eye to be examined, etc.) and a second imaging site different from the first imaging site. A second trained model obtained using the data and a plurality of trained models including the second trained model can be prepared. Then, the image processing unit 303 may have a selection means (not shown) for selecting one of the plurality of trained models. At this time, the image processing unit 303 may have a control means (not shown) that executes additional learning for the selected trained model. The control means searches for data in which the captured part corresponding to the selected trained model and the captured image of the captured part are paired in response to an instruction from the examiner, and the data obtained by the search is the learning data. Can be executed as additional learning for the selected trained model. The imaging region corresponding to the selected trained model may be acquired from the information in the header of the data or manually input by the examiner. Further, the data search may be performed from a server of an external facility such as a hospital or a research institute via a network, for example. As a result, additional learning can be efficiently performed for each imaged part by using the photographed image of the imaged part corresponding to the trained model.

The selection means and the control means may be composed of a software module executed by a processor such as a CPU or an MPU of the image processing unit 303. Further, the selection means and the control means may be composed of a circuit that performs a specific function such as an ASIC, an independent device, or the like.

In addition, when acquiring learning data for additional learning from a server of an external facility such as a hospital or research institute via a network, it is desired to reduce reliability deterioration due to falsification or system trouble during additional learning. Therefore, the correctness of the learning data for additional learning may be detected by confirming the consistency by digital signature or hashing. As a result, the learning data for additional learning can be protected. At this time, if the validity of the training data for additional learning cannot be detected as a result of confirming the consistency by digital signature or hashing, a warning to that effect is given and additional learning is performed using the training data. Absent. The server may be in any form, for example, a cloud server, a fog server, an edge server, or the like, regardless of its installation location.

(Modification 6)
In the various embodiments and modifications described above, the instruction from the examiner may be an instruction by voice or the like in addition to a manual instruction (for example, an instruction using a user interface or the like). At this time, for example, a machine learning model including a voice recognition model (speech recognition engine, trained model for voice recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by character input or the like using a keyboard, a touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (character recognition engine, trained model for character recognition) obtained by machine learning may be used. Further, the instruction from the examiner may be an instruction by a gesture or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, learned model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be the line-of-sight detection result of the examiner on the display screen of the display unit 600 or the like. The line-of-sight detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from the periphery of the display screen on the display unit 600. At this time, the object recognition engine as described above may be used for the pupil detection from the moving image. Further, the instruction from the examiner may be an instruction by an electroencephalogram, a weak electric signal flowing through the body, or the like.

In such a case, for example, as the training data, various trained data such as character data or voice data (waveform data) indicating an instruction for displaying the result by processing the various trained models as described above are used as input data. The training data may be training data in which the execution instruction for actually displaying the result of the model processing on the display unit is the correct answer data. Further, as the training data, for example, character data or audio data indicating an instruction for displaying a high-quality image obtained by a trained model for high image quality is used as input data, and an execution command for displaying a high-quality image and a high image quality are used. It may be learning data in which the execution instruction for changing the image quality button to the active state is the correct answer data. Of course, the learning data may be any data as long as the instruction content and the execution instruction content indicated by the character data, the voice data, or the like correspond to each other. Further, voice data may be converted into character data by using an acoustic model, a language model, or the like. Further, the waveform data obtained by the plurality of microphones may be used to perform a process of reducing the noise data superimposed on the audio data. Further, the instruction by characters or voice and the instruction by a mouse, a touch panel or the like may be configured to be selectable according to the instruction from the examiner. Further, the on / off of the instruction by characters or voice may be selectably configured according to the instruction from the examiner.

Here, machine learning includes deep learning as described above, and for at least a part of a multi-layer neural network, for example, a recurrent neural network (RNN) can be used. Here, as an example of the machine learning model according to this modified example, RNN, which is a neural network that handles time series information, will be described with reference to FIGS. 9 (a) and 9 (b). Further, a Long short-term memory (hereinafter referred to as LSTM), which is a kind of RNN, will be described with reference to FIGS. 10 (a) and 10 (b).

FIG. 9A shows the structure of the RNN, which is a machine learning model. The RNN 3520 has a loop structure in the network, ^{inputs data x t} 3510 at time t, and outputs ^{data h t 3530.} Since the RNN3520 has a loop function in the network, the current state can be inherited to the next state, so that time-series information can be handled. FIG. 9B shows an example of input / output of the parameter vector at time t. The data x ^t 3510 contains N pieces of data (Params1 to ParamsN). Further, the data h ^t 3530 output from the RNN 3520 includes N data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information at the time of error back propagation, LSTM may be used. The LSTM can learn long-term information by including a forgetting gate, an input gate, and an output gate. Here, FIG. 10A shows the structure of the LSTM. In the RSTM3540, the information that the network takes over at the next time t is the internal state c ^{t-1 of the} network called the cell and the output data h ^t-1 . The lowercase letters (c, h, x) in the figure represent vectors.

Next, FIG. 10 (b) shows the details of RSTM3540. In FIG. 10B, FG indicates a forgetting gate network, IG indicates an input gate network, and OG indicates an output gate network, each of which is a sigmoid layer. Therefore, a vector in which each element has a value of 0 to 1 is output. The oblivion gate network FG determines how much past information is retained, and the input gate network IG determines which value to update. The CU is a cell update candidate network and is an activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gate network OG selects the cell candidate element and selects how much information to convey at the next time.

Since the above-mentioned LSTM model is a basic model, it is not limited to the network shown here. You may change the coupling between the networks. QRNN (Quasi Recurrent Neural Network) may be used instead of RSTM. Further, the machine learning model is not limited to the neural network, and boosting, a support vector machine, or the like may be used. Further, when the instruction from the examiner is input by characters, voice, or the like, a technique related to natural language processing (for example, Sequence to Sequence) may be applied. Further, a dialogue engine (dialogue model, trained model for dialogue) that responds to the examiner by outputting characters or voices may be applied.

(Modification 7)
In the various embodiments and modifications described above, the high-quality image or the like may be stored in the storage unit 302 in response to an instruction from the examiner. At this time, after the instruction from the examiner to save the high-quality image etc., when registering the file name, as a recommended file name, any part of the file name (for example, the first part, the last part) A file name containing information (for example, characters) indicating that the image is an image generated by processing using a trained model for high image quality (high image quality processing) is instructed by the examiner. It may be displayed in an editable state accordingly.

Further, when displaying a high-quality image on the display unit 600 on various display screens such as a report screen, the displayed image is a high-quality image generated by processing using a learned model for high image quality. A display indicating that the image may be displayed together with a high-quality image. In this case, the examiner can easily identify from the display that the displayed high-quality image is not the image itself acquired by shooting, so that misdiagnosis can be reduced or the diagnosis efficiency can be improved. be able to. The display indicating that the image is a high-quality image generated by the process using the trained model for high image quality is a display that can distinguish the input image and the high-quality image generated by the process. It may be of any aspect. Further, not only the processing using the trained model for high image quality but also the processing using various trained models as described above is the result generated by the processing using the trained model of the same type. A display indicating that there is may be displayed with the result.

At this time, the display screen such as the report screen may be saved in the storage unit 302 as image data according to the instruction from the examiner. For example, the report screen is stored as one image in which a high-quality image or the like and a display indicating that these images are high-quality images generated by processing using a trained model for high image quality are arranged. It may be stored in 302.

In addition, what kind of training data was used by the trained model for high image quality to learn the display indicating that the image is a high quality image generated by processing using the trained model for high image quality. A display indicating the presence or absence may be displayed on the display unit. The display may include an explanation of the types of the input data and the correct answer data of the learning data, and an arbitrary display regarding the correct answer data such as the imaging part included in the input data and the correct answer data. It should be noted that not only the processing using the trained model for high image quality but also the processing using the various trained models as described above is trained by what kind of training data the trained model of that type uses. A display indicating whether or not the data is used may be displayed on the display unit.

In addition, information (for example, characters) indicating that the image is generated by processing using a trained model for high image quality is displayed or saved in a state of being superimposed on the high image quality image or the like. You may. At this time, the portion to be superimposed on the image may be any region (for example, the edge of the image) that does not overlap the region where the region of interest to be photographed is displayed. Further, the non-overlapping areas may be determined and superimposed on the determined areas.

In addition, when the high image quality button is set to the active state (high image quality processing is on) by default as the initial display screen of the report screen, the high image quality image is instructed by the examiner. The report image corresponding to the report screen including the above may be configured to be transmitted to a server such as the external storage unit 500. In addition, when the high image quality button is set to the active state by default, it is changed from the shooting confirmation screen or the preview screen to the report screen at the end of the inspection (for example, according to the instruction from the inspector). In some cases), the report image corresponding to the report screen including the high-quality image and the like may be configured to be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, the presence / absence of superimposition of the analysis map, whether or not the image is high quality, and the display screen for follow-up observation. The report image generated based on (settings related to at least one such as whether or not) may be configured to be transmitted to the server.

(Modification 8)
In the various embodiments and modifications described above, the analysis results of images (for example, high-quality images, analysis maps, etc.) obtained by the first type of trained model among the various trained models described above are shown. An image, an image showing an object recognition result, an image showing a segmentation result) may be input to a second type of trained model different from the first type. At this time, the result (for example, analysis result, diagnosis result, object recognition result, segmentation result) of the processing of the second type of trained model may be generated.

Further, among the various trained models as described above, the result of processing the first type of trained model (for example, analysis result, diagnosis result, object recognition result, segmentation result) is used to use the first type. An image to be input to a second type of trained model different from the first type may be generated from the image input to the trained model of. At this time, the generated image is likely to be an image suitable as an image to be processed by the second type of trained model. Therefore, an image obtained by inputting the generated image into the second type of trained model (for example, a high-quality image, an image showing an analysis result such as an analysis map, an image showing an object recognition result, and a segmentation result are displayed. The accuracy of the image shown) can be improved.

Further, the various trained models as described above may be trained models obtained by learning learning data including a two-dimensional medical image of the subject, or may be a three-dimensional medical model of the subject. It may be a trained model obtained by learning training data including images.

Further, a similar case image search using an external database stored in a server or the like may be performed using the analysis result, the diagnosis result, etc. obtained by the processing of the learned model as described above as a search key. If a plurality of images stored in the database are already managed by machine learning or the like with the feature amount of each of the plurality of images attached as incidental information, the image itself is used as a search key. Similar case image search engine (similar case image search model, learned model for similar case image search) may be used. For example, the image processing unit 303 uses a trained model for searching for a similar case image (different from the trained model for improving image quality) to obtain a similar case related to the noise-reduced image from the above-mentioned noise-reduced image. You can search for images. Further, for example, the display control unit 305 is related to the noise reduction image obtained from the noise reduction image described above by using the learned model for searching the similar case image obtained by learning the learning data including the medical image of the subject. The similar case image can be displayed on the display unit 600.

(Modification 9)
In the various embodiments and modifications described above, the various processes are not limited to the configuration performed based on the brightness value of the tomographic image. The various processes are applied to fault data including interference signals acquired by the tomographic image capturing apparatus 200, signals obtained by subjecting the interference signals to Fourier transform, signals obtained by subjecting the signals to arbitrary processing, and tomographic images based on these. It may be applied to. In these cases as well, the same effect as the above configuration can be obtained. For example, although a fiber optical system using an optical coupler is used as the dividing means, a spatial optical system using a collimator and a beam splitter may be used. Further, the configuration of the tomographic imaging apparatus 200 is not limited to the above configuration, and a part of the configuration included in the tomographic imaging apparatus 200 may be a configuration separate from the tomographic imaging apparatus 200. Further, in the above configuration, the configuration of the Michelson type interferometer is used as the interference optical system of the tomographic imaging apparatus 200, but the configuration of the interference optical system is not limited to this. For example, the interferometric optical system of the tomographic imaging apparatus 200 may have the configuration of a Mach-Zehnder interferometer. Further, as the OCT device, a spectral domain OCT (SD-OCT) device using an SLD as a light source is described, but the configuration of the OCT device is not limited to this. For example, the present invention can be applied to any other type of OCT device such as a wavelength sweep type OCT (SS-OCT) device using a wavelength sweep light source capable of sweeping the wavelength of emitted light. The present invention can also be applied to a Line-OCT device (or SS-Line-OCT device) using line light. The present invention can also be applied to a Full Field-OCT device (or SS-Full Field-OCT device) using area light. Further, the image processing unit 303 acquires interference signals acquired by the tomographic image capturing apparatus 200, a three-dimensional tom image generated by the image processing unit, and the like, and the image processing unit 303 obtains these signals and images. The configuration to be acquired is not limited to this. For example, the image processing unit 303 may acquire these signals from a server or a photographing device connected via a LAN, WAN, the Internet, or the like.

The trained model can be provided in the image processing unit 303. The trained model can be composed of, for example, software modules executed by a processor such as a CPU. Further, the trained model may be provided on another server or the like connected to the image processing unit 303. In this case, the image processing unit 303 can perform high image quality processing using the trained model by connecting to a server including the trained model via an arbitrary network such as the Internet.

(Modification example 10)
Further, the image processed by the image processing apparatus or the image processing method according to the various embodiments and modifications described above includes a medical image acquired by using an arbitrary modality (imaging apparatus, imaging method). The medical image to be processed may include a medical image acquired by an arbitrary imaging device or the like, or an image created by an image processing device or an image processing method according to the above-described embodiment and modification.

Further, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. In addition, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black-and-white image or a color image. Further, the medical image may be an image showing the structure (morphology) of a predetermined part or an image showing the function thereof. The image showing the function includes, for example, an OCTA image, a Doppler OCT image, an fMRI image, and an image showing blood flow dynamics (blood flow volume, blood flow velocity, etc.) such as an ultrasonic Doppler image. The predetermined part of the subject may be determined according to the subject to be imaged, and the human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, liver and other organs, head, chest, etc. Includes any part such as legs and arms.

Further, the medical image may be a tomographic image of the subject or a frontal image. The frontal image is, for example, a frontal image of the fundus, a frontal image of the anterior segment of the eye, a fundus image photographed by fluorescence, and data acquired by OCT (three-dimensional OCT data) in at least a part range in the depth direction of the imaged object. Includes En-Face images generated using the data from. The En-Face image is an OCTA En-Face image (motion contrast front image) generated by using at least a part of the data in the depth direction of the shooting target for the three-dimensional OCTA data (three-dimensional motion contrast data). ) May be. Further, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Here, the motion contrast data is data indicating a change between a plurality of volume data obtained by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data showing changes between a plurality of tomographic images obtained at substantially the same position at different positions. The motion contrast frontal image is also referred to as an OCTA frontal image (OCTA En-Face image) relating to OCTA angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also referred to as OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals. , It may be obtained by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected.

The En-Face image is, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions. At this time, the front image is at least a part of the depth range of the volume data (three-dimensional tomographic image) obtained by using optical interference, and is the data corresponding to the depth range determined based on the two reference planes. Is projected or integrated on a two-dimensional plane. The En-Face image is a frontal image generated by projecting the data corresponding to the depth range determined based on the detected retinal layer of the volume data onto a two-dimensional plane. As a method of projecting data corresponding to a depth range determined based on two reference planes on a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on the two-dimensional plane. Techniques can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Further, the depth range related to the En-Face image is, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries relating to the detected retinal layer. May be good. Further, the depth range related to the En-Face image may be, for example, a range changed (offset) according to an operator's instruction from a range between two layer boundaries related to the detected retinal layer. Good.

The imaging device is a device for capturing an image used for diagnosis. The photographing device detects, for example, a device that obtains an image of a predetermined part by irradiating a predetermined part of the subject with radiation such as light or X-rays, electromagnetic waves, ultrasonic waves, or the like, or radiation emitted from the subject. This includes a device for obtaining an image of a predetermined part. More specifically, the imaging devices according to the various embodiments and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, and a fundus. Includes cameras, endoscopes, etc.

The OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Further, the Fourier domain OCT apparatus may include a spectral domain OCT (SD-OCT) apparatus and a wavelength sweep type OCT (SS-OCT) apparatus. Further, the SLO device and the OCT device may include a wave surface compensation SLO (AO-SLO) device using a wave surface compensation optical system, a wave surface compensation OCT (AO-OCT) device, and the like. Further, the SLO device and the OCT device may include a polarized SLO (PS-SLO) device, a polarized OCT (PS-OCT) device, and the like for visualizing information on polarization phase difference and polarization elimination.

(Other embodiments)
Each of the above embodiments has been realized as an image processing device. However, the present invention is not limited to the image processing apparatus. It is also possible to realize the present invention as software that operates on a computer. The CPU of the image processing device controls the entire computer using computer programs and data stored in RAM or ROM. In addition, the execution of software corresponding to each part of the image processing device is controlled to realize the function of each part. In addition, the user interface such as buttons and the layout of the display are not limited to those shown above.

The present invention is also realized by executing the following processing. That is, software (program) that realizes one or more functions of the various embodiments and modifications described above is supplied to the system or device via a network or various storage media, and the computer (or CPU) of the system or device is supplied. This is a process in which a program is read and executed by an MPU or the like.

Further, the present invention supplies software (program) that realizes one or more functions of the various embodiments and modifications described above to a system or device via a network or storage medium, and the computer of the system or device supplies the software (program). It can also be realized by the process of reading and executing the program. A computer may have one or more processors or circuits and may include multiple separate computers or a network of separate processors or circuits to read and execute computer executable instructions.

At this time, the processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Claims (27)

A means for detecting the position of noise in a first medical image obtained by imaging a subject, and
A means for generating a second medical image in which the pixel value of the detected noise position in the first medical image is reduced, and an estimated image in which noise in the first medical image is presumed to be reduced. When,
A means for changing the estimated image by using an objective function including a first term relating to the difference between the second medical image and the estimated image and a second term relating to the estimated image.
A means for outputting an estimated image obtained by changing the objective function so as to satisfy a predetermined condition as a noise reduction image, and
An image processing device having. The image processing apparatus according to claim 1, further comprising means for changing a threshold value for detecting the position of the noise according to a control parameter at the time of photographing of the OCT apparatus. The means for detecting the position of the noise includes at least one pixel having a signal level higher than the peripheral vicinity in the first medical image and a pixel having a signal level smaller than the peripheral vicinity in the first medical image. The image processing apparatus according to claim 1 or 2, which detects the position of noise. A means of predicting the position of noise in the first medical image obtained by photographing the subject,
A means for generating a second medical image in which the pixel value of the predicted noise position in the first medical image is reduced, and an estimated image in which noise in the first medical image is presumed to be reduced. When,
A means for changing the estimated image by using an objective function including a first term relating to the difference between the second medical image and the estimated image and a second term relating to the estimated image.
Noise reduction is an estimated image obtained by changing the objective function so as to satisfy a predetermined condition, and is obtained by synthesizing a plurality of the estimated images obtained by predicting the position of the noise as a random number. Means to output as an image and
An image processing device having. The fourth aspect of claim 4, wherein when the position of the noise is predicted as a random number, the ratio of the number of pixels related to noise to the number of pixels in the first medical image is changed according to the control parameter at the time of imaging by the OCT device. Image processing equipment. The image according to any one of claims 1 to 5, wherein the objective function has a parameter for changing the ratio of the error term according to the first term and the total variation regularization term according to the second term. Processing equipment. The image processing apparatus according to claim 6, further comprising means for setting the parameters in response to an instruction from the operator. The image processing device according to claim 6 or 7, wherein the parameters are changed according to control parameters at the time of photographing of the OCT device. The image processing device according to any one of claims 1 to 8, wherein the first medical image is a tomographic image or a frontal image obtained by imaging the subject by the OCT device. The image processing device according to any one of claims 1 to 9, wherein the subject is an eye to be inspected. The image processing apparatus according to any one of claims 1 to 10, further comprising means for determining that the objective function satisfies a predetermined condition when the objective function is minimized. The image processing apparatus according to any one of claims 1 to 10, wherein the means for changing the estimated image is a means for changing the estimated means using an autoencoder. The noise reduction images obtained by using the plurality of first medical images obtained by using the means for controlling the scanning of the measurement light are aligned so that the same portion of the subject is scanned a plurality of times. The image processing apparatus according to any one of claims 1 to 12, wherein a plurality of noise reduction images obtained by alignment are added and averaged. A plurality of the first medical images obtained by using a means for controlling scanning of the measurement light are aligned so that the same portion of the subject is scanned a plurality of times, and a plurality of the aligned first medical images are obtained. The image processing apparatus according to any one of claims 1 to 12, wherein a plurality of the noise reduction images obtained by using the medical image of 1 are added and averaged. A high-quality image obtained by performing high-quality processing on the noise-reduced image is generated from the noise-reduced image by using a trained model for high image quality obtained by learning training data including a medical image of a subject. The image processing apparatus according to any one of claims 1 to 12. The image processing apparatus according to any one of claims 1 to 15, further comprising a display control means for displaying the noise reduction image on the display means. The display control means uses a trained model for generating analysis results obtained by learning training data including a medical image of a subject to obtain analysis results related to the noise reduction image generated from the noise reduction image. The image processing apparatus according to claim 16, wherein the display means displays the image. The display control means uses a learned model for generating a diagnostic result obtained by learning learning data including a medical image of a subject to obtain a diagnostic result related to the noise-reduced image generated from the noise-reduced image. The image processing apparatus according to claim 16 or 17, which is displayed on the display means. The display control means uses a trained model for object recognition obtained by learning training data including a medical image of a subject or a trained model for segmentation obtained by learning training data including a medical image of a subject. The image processing apparatus according to any one of claims 16 to 18, wherein a partial region detected from the noise-reduced image is displayed on the display means. A claim that the display control means causes the display means to display information on a difference between a medical image obtained from the noise reduction image by using a hostile generation network or an autoencoder and the noise reduction image as information on an abnormal portion. The image processing apparatus according to any one of 16 to 19. The display control means uses a trained model for searching for a similar case image obtained by learning training data including a medical image of a subject to obtain a similar case image related to the noise reduction image obtained from the noise reduction image. The image processing apparatus according to any one of claims 16 to 20, which is displayed on the display means. The position of the noise is any one of claims 1 to 21 which is detected or predicted by using a learned model obtained by learning learning data including a medical image of a subject from the first medical image. The image processing apparatus according to. A means of acquiring a first medical image obtained by imaging a subject,
A means for detecting the position of noise in the acquired first medical image from the acquired first medical image by using a learned model obtained by learning learning data including a medical image of a subject. ,
An image processing device having. The image processing apparatus according to any one of claims 1 to 23,
A system including an OCT device for acquiring the first medical image as a tomographic image. The process of detecting the position of noise in the first medical image obtained by imaging the subject, and
A step of generating a second medical image in which the pixel value of the detected noise position in the first medical image is reduced, and an estimated image in which noise in the first medical image is presumed to be reduced. When,
A step of changing the estimated image by using an objective function including a first term relating to the difference between the second medical image and the estimated image and a second term relating to the estimated image.
A step of outputting an estimated image obtained by changing the objective function so as to satisfy a predetermined condition as a noise reduction image, and
Image processing method having. The process of predicting the position of noise in the first medical image obtained by photographing the subject, and
A step of generating a second medical image in which the pixel value of the predicted noise position in the first medical image is reduced, and an estimated image in which noise in the first medical image is presumed to be reduced. When,
A step of changing the estimated image by using an objective function including a first term relating to the difference between the second medical image and the estimated image and a second term relating to the estimated image.
Noise reduction is an estimated image obtained by changing the objective function so as to satisfy a predetermined condition, and is obtained by synthesizing a plurality of the estimated images obtained by predicting the position of the noise as a random number. The process of outputting as an image and
Image processing method having. A program that executes each step of the image processing method according to claim 25 or 26.
7

Images