JP2021184169A - Image processing device, image processing method, and program - Google Patents

Abstract

To estimate a disease with high accuracy.SOLUTION: An image processing device includes a generation unit that processes a medical image of a subject to generate a symmetry map showing the symmetry of the subject, and an output unit that outputs an estimation result regarding a disease estimated from a generated symmetry map by using a learnt model obtained by learning from the use of the symmetry map.SELECTED DRAWING: Figure 1

Description

The disclosure of this specification relates to an image processing apparatus, an image processing method, and a program.

In recent years, there are increasing expectations for a technology that analyzes medical information such as medical images with a computer and uses the obtained results as an aid to diagnosis. Patent Document 1 discloses a method of estimating information on eye symptoms such as glaucoma by machine learning a fundus photograph. Further, it is known that a tomographic image to be measured can be acquired non-invasively by an apparatus (OCT apparatus) using an optical coherence tomography (OCT).

International Publication No. 2019-073962

When a machine learning model for estimating a disease such as glaucoma is generated from a photograph of the fundus of the eye, if the estimation accuracy of the model is insufficient, a disease such as early glaucoma may be overlooked. Therefore, it is required to improve the accuracy of disease estimation by a machine learning model.

The image processing apparatus according to one embodiment of the present disclosure is obtained by image processing of a medical image of a subject to generate a symmetry map showing the symmetry of the subject, and learning using the symmetry map. Using the trained model, it is provided with an output unit that outputs an estimation result regarding the disease estimated from the generated symmetry map.

According to at least one embodiment of the present disclosure, the disease can be estimated with high accuracy.

An example of the overall configuration of the OCT apparatus according to the first embodiment is schematically shown. An example of a flowchart of the operation of image processing according to the first embodiment is shown. An example of the flowchart of the symmetry map generation process according to the first embodiment is shown. An example of the process of the geometric correction processing according to the first embodiment is shown. An example of the process of generating the symmetry map according to the first embodiment is shown. An example of the flowchart of the correlation map generation process according to the third embodiment is shown. The schematic diagram of the generation process of the correlation map which concerns on 3rd Embodiment is shown. An example of the machine learning model according to the third embodiment is shown. An example of the machine learning model according to the third embodiment is shown. An example of the layer thickness profile according to the modification 2 is shown. An example of the machine learning model according to the modification 8 is shown.

Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. In the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar. In addition, some components, members, and processes that are not important for explanation may be omitted in each drawing.

In the following, the machine learning model refers to a learning model based on a machine learning algorithm. Specific algorithms for machine learning include the nearest neighbor method, the naive Bayes method, the decision tree, and the support vector machine. In addition, deep learning (deep learning) in which features for learning and coupling weighting coefficients are generated by themselves using a neural network can also be mentioned. As appropriate, any of the above algorithms that can be used can be applied to the following embodiments and modifications. The teacher data refers to learning data and is composed of a pair of input data and output data. The correct answer data means the output data of the learning data (teacher data).

The trained model is a model in which training (learning) is performed in advance using appropriate teacher data (learning data) for a machine learning model that follows an arbitrary machine learning algorithm such as deep learning. .. However, although the trained model is obtained by using appropriate training data in advance, it is not that no further training is performed, and additional training can be performed. Additional learning can be done even after the device has been installed at the site of use.

Further, in the present specification, images of the fundus of the eye to be inspected obtained by each imaging technique (modality) such as OCT, fundus camera, and scanning laser ophthalmoscope (SLO) are collectively referred to as “fundus image”. I will call it. Further, in the present specification, the depth direction of the subject is the Z direction, the direction perpendicular to the Z direction is the X direction, and the direction perpendicular to the Z direction and the X direction is the Y direction.

(First Embodiment)
Hereinafter, the optical coherence tomography apparatus (OCT apparatus) 100, which is an example of the ophthalmologic imaging apparatus according to the first embodiment of the present invention, will be described with reference to FIGS. 1 to 5. As illustrated in FIG. 1, the OCT device 100 according to the present embodiment is provided with a photographing device 101, a control unit 102, a display unit 103, and an operation unit 104.

The photographing apparatus 101 is an apparatus provided with an imaging optical system for photographing a medical image of the eye to be inspected. The photographing optical system provided in the photographing apparatus 101 may include an optical system for photographing a tomographic image of the eye to be inspected, and may be configured by a known OCT optical system. Here, the OCT optical system included in the photographing apparatus 101 may be any kind of OCT optical system such as SD-OCT (Spectral Domain OCT) or SS-OCT (Swept Source OCT). Further, in addition to the OCT optical system, the photographing apparatus 101 may include, for example, an optical system for SLO for acquiring an image of the eye to be inspected, a fundus camera, an anterior eye portion observation, and the like.

The control unit 102 is provided with an acquisition unit 121, an image processing unit 122, an estimation unit 123, a drive control unit 124, a display control unit 125, and a storage unit 126. The control unit 102 can be configured by using a general computer including a processor, a memory, and the like, but may be configured as a dedicated computer of the OCT device.

Here, the control unit 102 may be a built-in (internal) computer of the OCT device 100, or may be a separate (external) computer to which the OCT device 100 is communicably connected. Further, the control unit 102 may be, for example, a personal computer, or a desktop PC, a notebook PC, or a tablet PC (portable information terminal) may be used. At this time, the communication connection between the control unit 102 and the photographing device 101 may be a connection by wire communication or a connection by wireless communication. The processor may be a CPU (Central Processing Unit). Further, the processor may be, for example, an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), an FPGA (Field-Programmable Gate Array), or the like.

The acquisition unit 121 acquires various data from the photographing device 101 via an interface using, for example, a USB (UniversalSerialBus) cable. Further, the acquisition unit 121 may acquire various data including images from an external device (not shown) such as a server connected to the control unit 102. Here, the control unit 102 and an external device (not shown) may be connected via an arbitrary network such as the Internet. Further, the acquisition unit 121 may acquire the image of the eye to be inspected generated by the image processing unit 122 and various data stored in the storage unit 126.

The image processing unit 122 can generate an image of the eye to be inspected or perform arbitrary image processing on the image of the eye to be inspected by using the data or the image acquired by the acquisition unit 121. For example, the image processing unit 122 can generate a fundus image such as a tomographic image of the eye to be inspected by using the data related to the eye to be inspected acquired by the acquisition unit 121. As a method for generating a tomographic image or the like, any known method may be used. Further, the image processing unit 122 can also generate an analysis map such as a thickness map relating to the layer thickness of each tissue of the eye to be inspected by using a tomographic image or the like of the eye to be inspected. In the present embodiment, the image processing unit 122 generates a symmetry map showing symmetry regarding the eye to be inspected by using the thickness map.

The estimation unit 123 uses the trained model to estimate the disease of the subject from the image of the subject and the like. In the present embodiment, the estimation unit 123 estimates the ophthalmologic disease based on the fundus image, which is an example of the medical image of the eye to be inspected acquired in a single time. More specifically, the estimation unit 123 according to the present embodiment estimates the ophthalmic disease such as glaucoma of the inspected eye from the symmetry map of the inspected eye using the trained model. Here, the estimation unit 123 functions as an example of an output unit that outputs an estimation result regarding the disease estimated from the symmetry map using the trained model.

The drive control unit 124 controls the drive of various components in the photographing apparatus 101. The display control unit 125 can control the display of the display unit 103 connected to the control unit 102, and cause the display unit 103 to display various data such as a tomographic image of the eye to be inspected and patient information. The storage unit 126 can store a program for realizing various application software including an operating system (OS), a device driver of a peripheral device, and a program for performing processing described later. Further, the storage unit 126 can also store the information acquired by the acquisition unit 121, various images processed by the image processing unit 122, the estimation result by the estimation unit 123, and the like.

Each component other than the storage unit 126 of the control unit 102 may be configured by a software module executed by a processor such as a CPU or MPU. The processor may be, for example, GPU, FPGA, or the like. Further, each component may be configured by a circuit or the like that performs a specific function such as an ASIC. The storage unit 126 may be composed of, for example, an optical disk such as a hard disk or an arbitrary storage medium such as a memory.

The display unit 103 is an arbitrary display or the like, and based on the control by the display control unit 125, for example, patient information regarding a subject, various images, estimation results regarding a disease, and the like can be displayed. The operation unit 104 includes a mouse, a keyboard, a touch panel, and the like, and can input instructions from the user to the control unit 102.

Next, the image processing according to the present embodiment will be described step by step with reference to FIG. FIG. 2 is a flowchart showing the operation of image processing according to the present embodiment. In this embodiment, in particular, a case where estimation of glaucoma is performed using a tomographic image of the fundus will be described as an example.

[Step S21: Image acquisition]
First, in step S21, the acquisition unit 121 acquires data related to the tomographic image of the fundus of the eye to be inspected from the photographing apparatus 101. Here, the data related to the tomographic image of the fundus of the eye to be inspected may be an interference signal related to the fundus of the eye to be inspected acquired by photographing the eye to be inspected by an imaging device 101 or the like, or such an interference signal. It may be a tomographic image itself obtained by performing known image processing. When the acquisition unit 121 acquires the interference signal, the acquisition unit 121 may acquire the tomographic image generated by the image processing unit 122 performing the image processing of the interference signal.

The data related to the tomographic image acquired by the acquisition unit 121 includes those taken at a plurality of B scan positions of the eye to be inspected, whereby the acquisition unit 121 can acquire three-dimensional data regarding the fundus of the eye to be inspected. Here, the A scan means to acquire the tomographic information from one point of the eye to be inspected, and the A scan is performed a plurality of times in an arbitrary crossing direction (main scanning direction) to obtain the crossing direction and the depth of the eye to be inspected E. Acquiring information on a two-dimensional fault in the direction is called B scan. The acquisition unit 121 may acquire data related to a tomographic image of the fundus of the eye to be inspected acquired at another location from an external device (not shown).

Further, the acquisition unit 121 is acquired by another imaging technique such as a fundus photograph or anterior segment photograph taken by a fundus camera, or an image of the fundus or anterior segment taken by using an SLO optical system (SLO image). It is also possible to acquire the images and data that have been created. Here, the fundus photograph or the anterior segment photograph is an image taken using visible light, and may be a color image using at least two of RGB or a monochrome image using any one of RGB. good. Further, the acquisition unit 121 may also acquire patient data including at least one of the patient's identification number, axial length, age, visual acuity, race, medical history, and severe myopia regarding the eye to be inspected. can. The acquisition unit 121 may also acquire these images and data from an external device (not shown). The control unit 102 may extract patient data from the image of the medical record acquired by the acquisition unit 121 by using a text mining technique or the like.

[Step S22: Image processing]
In step S22, the image processing unit 122 performs image processing on the acquired three-dimensional tomographic image, and generates an image used by the estimation unit 123 for the estimation processing. The image processing unit 122 may process the interference signal acquired by the acquisition unit 121 to generate an image used by the estimation unit 123 for the estimation processing. The image processing unit 122 according to the present embodiment generates a symmetry map using the acquired tomographic image. An example of a method for generating a symmetry map will be described with reference to FIG. FIG. 3 shows an example of a flowchart of the symmetry map generation process according to the present embodiment.

[Step S221: Thickness map generation]
When the image processing in step S22 is started, the processing shifts to step S221. In step S221, the image processing unit 122 uses the tomographic image to generate a thickness map showing the thickness of at least a part of the subject. Here, the thickness map is an anatomically determined observation target layer in the retina of the subject, for example, the subject, for each XY position in the XY plane perpendicular to the depth direction (Z direction) of the subject. It is a map (map image) in which the thickness (layer thickness) of the retina is represented by a brightness value or the like. For example, the thickness map can be represented by converting the layer thickness at each XY position into a luminance value of a pseudo color scale. Further, the thickness map may be represented by converting the layer thickness at each XY position into a grayscale luminance value. The thickness map is an example of an analysis map showing information (analysis information) obtained by analyzing a medical image of a subject, and the analysis map is obtained by analyzing, for example, an OCTA (OCT Angiografy) image described later. It can include a blood vessel density map or the like showing the blood vessel density (area density or length density).

Specifically, the image processing unit 122 performs segmentation processing on the acquired tomographic image, extracts the layer structure in the tomographic area of the eye to be inspected, and calculates the thickness of each layer structure. As a method for segmentation processing and a method for calculating the thickness of the layer structure, any known method may be used. The image processing unit 122 generates a thickness map of the eye to be inspected using the thickness calculated for the observation target layer.

[Step S222: Correction process]
Next, in step S222, the image processing unit 122 performs correction processing on the thickness map. An example of the correction process in step S222 will be described below. Although the image processing unit 122 according to the present embodiment implements the correction processing described below in combination, only one of the correction processing described below may be performed.

An example of the correction process is a process (error correction process) for correcting the luminance value of an inappropriate portion of the thickness map. For example, if an error occurs in the segmentation process for a layer, the thickness at that position becomes a specific value and is represented as overexposure or underexposure in the thickness map. The image processing unit 122 performs a process of replacing the luminance value of such an inappropriate portion having a specific value in the thickness map with an average value of the surroundings as an example. Alternatively, the image processing unit 122 may remove such an inappropriate portion from the thickness map. Further, the image processing unit 122 may similarly replace the luminance value of a region considered unnecessary for diagnosis, such as the optic nerve papilla (hereinafter, also simply referred to as “papillary head”), from the thickness map. It may be removed. Further, the image processing unit 122 may perform edge enhancement processing or contrast adjustment processing on the thickness map.

Another example of the correction process is a process of correcting the layer thickness value based on the axial length data of the eye to be inspected when the thickness map is generated. Here, the axial length data of the eye to be inspected may be acquired in step S21 or may be input by the user. When the axial length of the eye to be examined is longer than that of the normal eye database, the retina is stretched, which has the effect of thinning the entire layer thickness. Therefore, the image processing unit 122 can correct the thickness map so as to reduce the influence of the layer thickness according to the axial length. The layer thickness may be corrected according to the axial length by using any known method. By such correction, it is possible to distinguish between the case where the layer thickness is thinned due to the lesion and the case where the layer thickness is thinned from normal.

Further, since the photographing range of the fundus with respect to the scan angle changes when the axial length is different, the image processing unit 122 may perform a process of correcting the scale of the thickness map. The image processing unit 122 may similarly correct the scale of the tomographic image used when generating the thickness map. Also, when the patient has severe myopia, the axial length tends to be long. Therefore, if the patient has severe myopia, the image processing unit 122 may make similar corrections to the thickness map. The axial length of the eye to be inspected and the non-existence of severe myopia may be determined based on the acquired patient data.

Further, another example of the correction process is a process of converting the value of each XY position on the thickness map into a value obtained by taking the ratio of the thickness of the layer to be observed to the thickness of a certain reference layer. The image processing unit 122 may correct the thickness map so that the ratio of the thickness of the nerve fiber layer to the value of the thickness from the nerve fiber layer to the inner plexiform layer is the value at each XY position, for example. can. By doing so, it is possible to reduce the influence of the difference in the overall thickness of the retina depending on the individual, and to extract the thickness of the layer of particular interest. The image processing unit 122 may perform the same processing when the thickness map is generated.

Another example of the correction process is a process of geometrically correcting the thickness map (geometric correction process). An example of the geometric correction process will be described with reference to FIGS. 4 (a) and 4 (b). Here, FIG. 4A shows an example of a thickness map before the geometric correction process, and FIG. 4B shows an example of a thickness map after the machine correction process.

As the geometric correction process, the image processing unit 122 refers to, for example, reference information about the thickness map stored in the storage unit 126, and expands / contracts (enlarges or reduces), rotates, translates, and so on with respect to the generated thickness map. , Performs geometric correction processing. An example of the reference information stored in the storage unit 126 is the center position of the macula, and the distance from the macula A to the papilla B in the image as shown by the dotted lines in FIGS. 4 (a) and 4 (b). The angle. The angle from the macula A to the nipple B may be, for example, the angle of the line connecting the macula A and the nipple B with respect to the horizontal direction (horizontal direction) of the image.

Specifically, first, the image processing unit 122 translates the thickness map so that the center position of the macula portion A comes to the center of the thickness map. The center position information of the macula A can be obtained by analyzing a tomographic image, a thickness map, or the like. After that, the image processing unit 122 expands / contracts and rotates the thickness map so that the line connecting the macula portion A and the papilla B shown by the dotted line in FIG. 4A has a reference length and angle (horizontal). Make adjustments. By such adjustment, the thickness map as shown in FIG. 4B can be obtained. It should be noted that, for example, the average value of the peripheral pixel values can be used as the pixel value at the edge portion where the pixel value has disappeared due to the geometric correction process. Further, a process of cutting out the entire image may be performed so that the edges are sufficiently filled.

[Step S23: Symmetry map generation]
Next, in step S223, the image processing unit 122 uses the thickness map to generate a symmetry map showing symmetry with respect to the eye to be inspected. The symmetry map is an analysis result obtained by analyzing an image of an eye to be inspected, and is an example of a comparison map that can be generated by comparing the analysis results of positions related to each other. Hereinafter, an example of a method for generating a symmetry map will be described with reference to FIGS. 5 (a) and 5 (b). FIG. 5A is an example of a thickness map obtained from a tomographic image of the right eye of a subject, and FIG. 5B is a thickness map obtained from a tomographic image of the left eye of the same subject. In FIGS. 5A and 5B, the central lines H1 and H2 extending in the horizontal direction and the central lines V1 and V2 extending in the vertical direction of the thickness map are single-point chain lines as criteria for the symmetry of the eye to be inspected. It is shown. In the present embodiment, the center lines H1, H2, V1 and V2 are defined with reference to the center of the macula in the thickness map. The central lines H1, H2, V1 and V2, which are the criteria for the symmetry of the eye to be inspected, are not limited to this, and may be determined based on an object capable of ensuring the symmetry of the eye to be inspected.

In this embodiment, the symmetry map is generated by evaluating any of the vertical symmetry, the horizontal symmetry, and the vertical and horizontal symmetry of the eye to be inspected. The method of generating these symmetry maps will be described in order below. In the following generation method, the relationship between the right eye and the left eye may be interchanged.

First, an example of a method of evaluating the vertical symmetry of the eye to be inspected to generate a symmetry map will be described. In this example, attention is paid to a pixel P1 in the thickness map of FIG. 5A and a pixel P2 whose position is vertically symmetrical with respect to the center line H1 with respect to the position of the pixel P1. The luminance value of the thickness map in the pixel P1 is I1, and the luminance value of the thickness map in the pixel P2 is I2. In the method of generating the symmetry map according to this example, the pixel value in the pixel P1 of the symmetry map is I1 / I2, and the pixel value in the pixel P2 of the symmetry map is I2 / I1. By performing the same processing on the other pixels, the image processing unit 122 generates an overall symmetry map using the thickness map. Further, although the ratio to the luminance value at the vertically symmetrical position is used as the pixel value here, the difference between the luminance values may be used as the pixel value instead. Further, the pixel value calculation method may include a logarithmic conversion process of the luminance value.

Next, an example of a method of evaluating the left-right symmetry of the eye to be inspected to generate a symmetry map will be described. In this example, the positions of a pixel P1 in the thickness map shown in FIG. 5 (a) and the pixel P1 with respect to the center line V1 in FIG. 5 (a) are opposite eyes shown in FIG. 5 (b). Focus on the pixel P3 at a position symmetrical with respect to the center line V2 in the thickness map of. In this embodiment, since the center of the thickness map coincides with the center of the yellow spot portion, the pixel P3 is the position and the left and right of the pixel P1 with respect to the center in the left-right direction in the thickness map shown in FIG. 5 (b). The pixels are in symmetrical positions. Let the luminance value of the pixel P1 be I1, and let the luminance value of P3 be I3. In the method of generating the symmetry map according to this example, the pixel value in the pixel P1 of the symmetry map is I1 / I3, and the pixel value in the pixel P3 of the symmetry map is I3 / I1. By performing the same processing on the other pixels, the image processing unit 122 generates an overall symmetry map using the thickness map. Further, similarly to the above, the difference may be used as the pixel value instead of the ratio with the luminance value at the symmetrical position. Further, the pixel value calculation method may include a logarithmic conversion process of the luminance value.

Next, an example of a method of evaluating the vertical and horizontal symmetry of the eye to be inspected to generate a symmetry map will be described. In this example, a pixel P1 in the thickness map of FIG. 5A, a pixel P2 having a vertically symmetrical position with respect to the position of the pixel P1, and a position symmetrical with respect to the position of the pixel P1 as described above. Attention is paid to the pixel P3 in FIG. 5B and the pixel P4 which is a pixel whose position is vertically symmetrical with respect to the center line H2 with respect to the position of the pixel P3. Let the luminance values of the pixels P1, P2, P3, and P4 be I1, I2, I3, and I4, respectively. In the method of generating the symmetry map according to this example, the ratios I1 / I2, I1 / I3, and I1 / I4 of the luminance values of the other three points to I1 are calculated, respectively. After that, the pixel value in the pixel P1 of the symmetry map is set to a value lower (minimum value) than the other values among these ratios. By performing the same processing on the other pixels, the image processing unit 122 generates an overall symmetry map using the thickness map. Further, similarly to the above, the difference may be used as the pixel value instead of the ratio with the luminance value at the symmetrical position. Further, the pixel value calculation method may include a logarithmic conversion process of the luminance value.

The image processing unit 122 may perform a process of smoothing the thickness map before calculating the ratio of the luminance values in the method of generating the symmetry map described above. The smoothing process can be performed by a moving average filter process, a Gaussian filter process, a median filter process, or the like. By performing the smoothing process, it is possible to reduce the influence of the positional deviation and the rotation deviation of each thickness map.

In the symmetry map generated as described above, the value related to the symmetry of the eye to be inspected is used as the pixel value of each XY position. Therefore, by using the symmetry map, it is possible to improve the accuracy of the estimation processing of the symptom that shows symmetry in normal condition but impairs symmetry at the time of disease, for example, glaucoma.

The image processing unit 122 outputs the symmetry map thus generated to the estimation unit 123. Further, the image processing unit 122 may output a thickness map showing the layer thickness at each XY position to the estimation unit 123. Further, the symmetry map and the thickness map may be combined to generate and output one composite map. This composite map can be generated, for example, by applying a symmetry map or a thickness map to any of the RGB channels. Further, instead of or in addition to the thickness map, a composite map may be generated by using a fundus photograph, an anterior segment photograph, an SLO image, an OCTA image, a frontal image of luminance (En-Face image), and another analysis map. .. When the composite map is generated, if the pixel sizes of the images to be composited are different, a process of matching the pixel sizes may be performed. Further, the image processing unit 122 may output a feature vector in which the luminance values of the respective XY positions of the analysis map such as the thickness map or the symmetry map are arranged in the vector to the estimation unit 123. Further, the image processing unit 122 may link the patient data acquired by the acquisition unit 121 to the symmetry map or the composite map and output the patient data to the estimation unit 123.

[Step S23: Estimate]
When the symmetry map is generated in step S223 and output to the estimation unit 123, the process proceeds to step S23. In step S23, the estimation unit 123 estimates the ophthalmic disease of the eye to be inspected from the symmetry map generated by the image processing unit 122 by using the trained model obtained by learning using the symmetry map as learning data.

[Example of machine learning model]
In the present embodiment, the trained model used by the estimation unit 123 is stored in the storage unit 126. An example of the machine learning model used by the estimation unit 123 is a neural network, and may be, for example, deep learning consisting of a multi-layered neural network. Further, for at least a part of the multi-layered neural network, for example, a convolutional neural network (CNN) can be used. Further, a technique related to an autoencoder (self-encoder) may be used for at least a part of a multi-layer neural network.

Further, a technique related to backpropagation (error backpropagation method) may be used for learning. Further, for learning, a method (dropout) in which each unit (each neuron or each node) is randomly inactivated may be used. Further, for learning, a method (batch normalization) may be used in which the data transmitted to each layer of the multi-layer neural network is normalized before the activation function (for example, the ReLu function) is applied. However, the machine learning is not limited to deep learning, and any learning using a model capable of extracting (expressing) the features of learning data such as images by learning may be used.

Further, the trained model used by the estimation unit 123 may be one generated by using transfer learning. For transfer learning, for example, a trained model such as Injection v3 can be used. In this case, for example, transfer learning may be performed on a machine learning model learned in a disease different from the disease to be diagnosed to generate a trained model used for estimation processing. By performing such transfer learning, it is possible to efficiently generate a trained model even for a disease for which it is difficult to obtain a large amount of training data. The other disease referred to here may be, for example, another disease of the eye. In addition, transfer learning may be performed on a machine learning model that detects lesions at these sites from endoscopic images of the stomach or large intestine.

[Example of learning data]
The trained model used in this embodiment learns the relationship between the symmetry map and information on ophthalmic diseases. More specifically, the trained model uses the symmetry map generated as described above as input data for training data, and outputs information on ophthalmic diseases related to the eye to be inspected related to the symmetry map used as input data. It can be a machine learning model learned as data.

A composite map may be used as the input data of the training data instead of the symmetry map. Further, the input channel of the machine learning model used by the estimation unit 123 is not limited to one channel. For example, any plurality of channels such as three channels for RGB can be used. In this case, various data such as a symmetry map, a thickness map, a feature vector, and patient data can be input to each input channel. In relation to this, as input data of training data, in addition to a symmetry map, a two-dimensional or three-dimensional tomographic image, an analysis map such as a thickness map, an En-Face image, an OCTA image, a feature vector, and a fundus photograph, Anterior segment photograph, SLO image, measurement result by a perimeter, angle information, patient data, and the like may be used. Here, the angle information refers to information related to the angle such as an image obtained by taking the angle of the eye to be inspected and the inspection result of the angle. In this case, the input data of the training data and the input data to the trained model may be a plurality of data of the same type.

The machine learning model used by the estimation unit 123 may be generated by using the data generated by GAN (hostile generation network) for learning. For example, a symmetry map labeled "normal" can be used to increase the symmetry map with the "normal" property in GAN. The term "normal" as used herein refers to a state without a disease. Also, the symmetry map of patients diagnosed with "glaucoma" can be used to increase the symmetry map with the nature of "glaucoma" in GAN. This can make up for the lack of data. The data generated by using GAN is not limited to the symmetry map, and may be an analysis map such as a tomographic signal, a tomographic image, a thickness map, or an OCTA image used for generating the symmetry map.

An example of disease information used as output data of the learning data according to the present embodiment is a label of the degree of progression (stage) such as normal, early, middle, and late stages of glaucoma. As the output data of the learning data, information (label) of the disease diagnosed for the eye to be inspected related to the symmetry map by an expert such as a doctor may be used. In this case, an expert such as a doctor may make a diagnosis of the eye to be inspected regarding the symmetry map by using the measurement result by the perimeter. The information used for diagnosis includes measurement results by a perimeter, tomographic images, analysis maps such as thickness maps, symmetry maps, En-Face images, OCTA images, fundus photographs, anterior ocular segment photographs, SLO images, and angle information. , And at least one of the other information.

In addition, analysis maps such as tomographic images and thickness maps, symmetry maps, En-Face images, OCTA images, fundus photographs, anterior segment photographs, SLO images, corner angle information, and , And only one of the other information may be used to make the diagnosis. By using such a diagnosis result as output data of training data, the trained model has a positive glaucoma judgment result using a perimeter, but it seems that no glaucoma findings appear in OCT images such as tomographic images. Examples can be treated with the same criteria as doctors.

The trained model trained using such training data outputs the certainty (ratio) for each estimation label (label of output data). For example, by using the softmax function in the final layer of the machine learning model, the output result from the trained model can be interpreted as the probability for each estimated label. Further, the weight for each estimated label may be output without performing the conversion by the softmax function. In this case, the output result can be interpreted as a probability by proportionally multiplying so that the total weight for each label becomes 1.

The type of ophthalmic disease related to the output data is not limited to glaucoma, and may be age-related macular degeneration, diabetic retinopathy, or the like, or a plurality of these diseases. In addition, the output data may include a probability of having a disease such as glaucoma, a probability of each degree of progression, a probability of progressing in the future, and the like. Also, as another example of output data, putative labels can be attached by disease name, such as normal, glaucoma, age-related macular degeneration, diabetic retinopathy, or other diseases. As yet another example of the output data, the putative label may be findings such as thinning of the nerve fiber layer, bleeding, and dilated depressions.

By using the trained model obtained by such learning, the estimation unit 123 can acquire the ratio (probability) for each estimated label which is information on the ophthalmic disease from the symmetry map of the eye to be inspected. The estimation unit 123 may correct the probability value to be output according to the patient data such as age and medical history. For example, when evaluating the probability of age-related macular degeneration, the estimation unit 123 corrects the output of the trained model according to age so that the older the person is, the higher the probability of being ill. It can be multiplied or added.

The estimation unit 123 outputs an estimation result based on the probability of each estimation label obtained by using the trained model. For example, the estimation unit 123 can output an estimation label having a higher probability than other estimation labels among the estimation labels as an estimation result. Further, among the estimated labels, the estimated label having a probability higher than the threshold value can be output as the estimation result. At this time, if there are a plurality of estimated labels with a probability higher than the threshold value, all of them may be output, or an estimated label with a higher probability than the other estimated labels may be output as the estimation result. .. Further, the estimation unit 123 may determine the estimation result by using the machine learning model from the probabilities of each estimation label obtained by using the trained model. The machine learning algorithm used in this case may be a different type of machine learning algorithm from the machine learning algorithm used to obtain the probability of the estimated label, for example, a neural network, a support vector machine, AdaBoost, a Basian network, etc. Alternatively, it may be a random forest or the like.

Further, the estimation unit 123 may determine whether the condition of the eye to be inspected is normal or abnormal by the threshold value of the probability of being normal or the probability of being abnormal. The "abnormality" here refers to a state having some kind of disease. In this case, even if the probability of being normal is greater than the probability of being abnormal, if the probability of being normal is less than the threshold, the estimation unit 123 outputs that the estimation result may have a disease. be able to.

The estimation result output by the estimation unit 123 can be displayed on the display unit 103 by the display control unit 125. Further, the estimation result can be stored in the storage unit 126. The estimation result output by the estimation unit 123 may be transmitted to an external device (not shown) connected to the control unit 102.

As described above, the control unit 102 according to the present embodiment includes an image processing unit 122 and an estimation unit 123. The image processing unit 122 functions as an example of a generation unit that processes a medical image of the eye to be inspected to generate a symmetry map showing the symmetry of the eye to be inspected. The estimation unit 123 functions as an example of an output unit that outputs an estimation result regarding the ophthalmic disease estimated from the generated symmetry map by using the trained model obtained by learning using the symmetry map. The trained model can output the probability related to the ophthalmic disease, and the estimation unit 123 can output the estimation result about the ophthalmic disease estimated based on the probability output by the trained model.

Here, ophthalmic diseases include at least one of glaucoma, diabetic retinopathy, and age-related macular degeneration. Further, the symmetry map generated by the image processing unit 122 is an analysis map showing information obtained by analyzing the fundus photograph, the anterior segment photograph, the SLO image, the OCTA image, the En-Face image, and the medical image of the eye to be inspected. It is a symmetry map which evaluated the symmetry of at least one of the vertical direction and the horizontal direction. In addition, the estimation result can include the estimation result regarding the stage of the ophthalmic disease.

With such a configuration, the control unit 102 according to the present embodiment generates a symmetry map based on the medical image of the eye to be inspected, and estimates the ophthalmic disease from the symmetry map using the trained model with high accuracy. Can be done. By using the symmetry map, it is possible to more accurately estimate a symptom such as glaucoma, which shows symmetry in normal conditions but impairs symmetry during a disease.

Further, in the estimation unit 123, in addition to the symmetry map generated by the image processing unit 122, the measurement result by the perimeter, the two-dimensional tomographic image, the three-dimensional tomographic image, the 3D volume data, the fundus photograph, and the anterior segment photograph. , SLO image, En-Face image, OCTA image, analysis map, corner angle information, layer thickness profile or feature vector, feature map extracted feature of medical image, and patient data. The obtained estimation result can be output. As a result, the feature amount used by the trained model for processing increases, and it can be expected that the accuracy of the estimation processing can be further improved.

Further, the image processing unit 122 can perform correction processing of the medical image before generating the symmetry map. The correction process is at least one of a smoothing process, a correction process based on the axial length, a layer thickness correction process based on a reference layer, a geometric correction process, an error correction process, an edge enhancement process, and a contrast adjustment process. including. As a result, the image quality of the medical image for performing the estimation process can be improved, or the image can be made more suitable for the actual state of the eye to be inspected, and the accuracy of the estimation process can be improved. The medical image acquired by the acquisition unit 121 and used by the image processing unit 122 may be an image that has already undergone some image processing such as the above correction processing.

In the present embodiment, the estimation process for the ophthalmic disease is performed, but the disease estimated by the estimation process is not limited to the ophthalmic disease and may be a systemic disease. The diseases estimated by the estimation process may be, for example, cardiovascular diseases such as hypertension and arteriosclerosis, endocrine diseases such as diabetes and hyperthyroidism, and brain diseases such as cerebral infarction, brain tumor, and dementia. With respect to such a disease, for example, the state and stage of the disease can be estimated from information on the blood vessels of the eye to be inspected. In this case, the output data of the learning data may be an estimation label or the like corresponding to the disease estimated by the estimation process.

[Modified example of the first embodiment]
Further, the estimation unit 123 may output a caution map (activation map) in which the region of interest of the trained model is shown (mapped) by a heat map. Note that the attention map can be obtained from the trained model used to obtain the probabilities of the estimated labels. The display control unit 125 can display the attention map output by the estimation unit 123 on the display unit 103. For example, the display control unit 125 may display the attention map on the display unit 103 by superimposing the image such as the fundus photograph, the anterior eye portion photograph, the SLO image, or the thickness map generated by the image processing unit 122. In this case, when the doctor looks at the image and makes a diagnosis, the information can be presented as an auxiliary.

Further, the estimation unit 123 may output the possibility that the eye to be inspected has glaucoma as an estimation result when the thickness of the observation target layer or the like of the eye to be inspected is thin as a whole. As an index of the overall layer thickness, for example, the average value or the median value of the thickness map corrected for the axial length can be used. In this case, the estimation unit 123 can estimate whether or not it is early glaucoma with a symmetry map, and can estimate whether or not it is late-stage glaucoma based on the overall layer thickness, and can perform a plurality of stages of estimation processing. ..

In this embodiment, the estimation unit 123 acquires the estimation result from the symmetry map using the trained model stored in the storage unit 126. On the other hand, the estimation unit 123 may function as an output unit that outputs the estimation processing result. In this case, the output unit outputs the symmetry map generated by the image processing unit 122 to an external device (not shown) connected to the control unit 102, and outputs the estimation result from the external device that performs the estimation processing as described above. You may get it. Further, the estimation unit 123 of the control unit 102 may perform estimation processing using a learned model provided in the external device.

The trained model used by the estimation unit 123 may be generated by two-step machine learning. For example, in the first stage of learning, a first trained model for estimating whether the eye to be inspected is normal or glaucoma is generated. When the first trained model is generated, it is tested by the cross-validation method whether the estimated result of the generated trained model is correct.

At this time, among the training data, although the output data of the data used for verification is labeled as abnormal, the symmetry map in which the output from the trained model is erroneously determined as a normal label is false. Label it negative. The output data of the learning data may be the information diagnosed using the measurement result by the perimeter as described above.

Next, as the second stage learning, learning is performed using the learning data used in the first stage, and a second trained model is generated. At this time, for the data labeled as false negative as described above, the label related to the output data of the training data is used for training as false negative. In this case, the estimation unit 123 can use the second trained model for the estimation process and acquire the respective probabilities for the estimated labels of "normal", "abnormal", and "false negative". The estimation unit 123 can output any of these labels as the estimation result by the same processing as described above. Further, when the probability of the estimated label of "false negative" is high, the estimation unit 123 outputs "abnormal" as the estimation result and outputs the label of "false negative" in such a manner that the examiner can confirm it. May be.

By using the trained model by two-step learning in this way, the glaucoma judgment result by the perimeter is positive, but it seems that the glaucoma findings do not appear in the image obtained by using the interference signal such as the symmetry map. Cases can be treated individually. Furthermore, cases such as premimeric glaucoma (PPG) in which changes do not appear in the measurement results by the perimeter and changes appear in the symmetry map and layer thickness map can be treated individually. In addition, by using the cross-validation method in the first stage of learning, it is possible to confirm whether or not more data should be labeled as false negative.

Here, it has been described that the second stage learning is performed using the data labeled as "false negative". For example, for an image judged to be "false negative", the label is set to "abnormal" and the second step is performed. The trained model of 1 may be additionally trained. In this case, the estimation accuracy by the first trained model can be improved, and the estimation unit 123 can perform highly accurate estimation processing by using the first trained model that has undergone additional learning. Can be expected. In this case, it is not necessary to generate the second trained model.

Further, the trained model used by the estimation unit 123 may be generated by using unsupervised learning. In unsupervised learning, map and image data with similar characteristics can be classified together. An expert such as a doctor picks up only a certain number of data from the classified data group and labels them as normal or disease. This makes it possible to reduce the load of annotation by the doctor.

Further, the trained model used by the estimation unit 123 may be trained using only the data labeled as “normal”. In this case, for example, the trained model outputs how much the symmetry map deviates from the symmetry map for the normal eye to be inspected (dissociation degree). Based on this degree of deviation, the estimation unit 123 can estimate whether the state of the eye to be inspected related to the symmetry map is normal or sick. Further, the threshold value of the degree of deviation used for this estimation may be separately set by testing with disease data. In this case, the trained model may be configured by using a CNN, a GAN described later, a variational autoencoder, a convolutional autoencoder, or the like.

Further, when the trained model used by the estimation unit 123 is generated, the non-information on the intensity myopia may be included in the input data of the training data for training. This makes it possible to learn about the effect of thinning the entire retina due to the long axial length.

Further, the examiner looks at the image acquired by using the photographing apparatus 101, the thickness map, the symmetry map, etc. generated by the image processing unit 122, and determines the determination result based on his / her medical knowledge by the operation unit 104. It may be input to the control unit 102. The display control unit 125 can cause the display unit 103 to display a warning message when the determination result input by the examiner and the estimation result output by the estimation unit 123 are different. In this case, the display control unit 125 may display the above-mentioned caution map on the display unit 103 at the same time. Further, the display control unit 125 may display the estimation result of the estimation unit 123 on the display unit 103 after the examiner inputs the determination result.

In step S221, the image processing unit 122 may generate a front image (En-Face image) in which three-dimensional data regarding the tomography of the fundus of the eye to be inspected acquired by the photographing apparatus 101 or the like is volume-rendered instead of the thickness map. good. In this case, the estimation unit 123 can output the estimation result regarding the ophthalmic disease estimated from the symmetry map generated by the image processing unit 122 using the En-Face image. When the nerve fiber layer is thinned due to glaucoma or the like, the proportion of the high-luminance reflective layer is reduced, so that the brightness value of the thinned region is lowered. Therefore, thinning of the nerve fiber layer can be detected not only by using the thickness map but also by using the En-Face image.

Further, the image processing unit 122 performs image processing on images acquired by other photographing techniques such as a fundus photograph, an anterior eye portion photograph, and an SLO image in the same manner as in step S22 above to generate a symmetry map. You may. Further, the image processing unit 122 has the same image as in step S22 for the analysis map obtained by analyzing the tomographic image, the fundus photograph, the anterior segment photograph, the SLO image, the En-Face image, the OCTA image, and the like. Processing may be performed to generate a symmetry map. Here, the analysis map includes, for example, the above-mentioned thickness map, a blood vessel density map showing the blood vessel density at each XY position obtained by analyzing the OCTA image, and the like. As a method for generating an analysis map such as a blood vessel density map, any known method may be used. Also in this case, the estimation unit 123 can output the estimation result regarding the disease of the subject estimated from the symmetry map generated by the image processing unit 122. By doing so, it is possible to improve the detection accuracy of a symptom such as glaucoma in which the image shows symmetry at normal times but the symmetry is impaired at the time of disease.

The image processing unit 122 may generate a symmetry map from an OCTA image, an En-Face image having a brightness obtained by volume rendering of data related to an interference signal, or the like by the same processing as described above. In this case as well, the estimation unit 123 may output the estimation result estimated from the generated symmetry map.

Therefore, the fundus image, which is an example of the medical image used to generate the symmetry map, is an analysis map such as a thickness map, a three-dimensional tomographic image, a fundus photograph, an anterior segment photograph, an SLO image, an OCTA image, and En-. At least one of the face images may be included. In these cases, the input data of the training data of the trained model used by the estimation unit 123 may be the same as the symmetry map generated by the image processing unit 122. For example, when the image processing unit 122 generates a symmetry map from an En-Face image or an OCTA image, the input data of the training data may also be a symmetry map generated from the En-Face image or the OCTA image. In these cases, the estimation unit 123 also includes ophthalmic diseases other than glaucoma such as diabetic retinopathy and age-related yellow spot degeneration, cardiovascular diseases such as hypertension and arteriosclerosis, and endocrine diseases such as diabetes and hyperthyroidism. , Estimated results regarding brain diseases such as cerebral infarction, brain tumor, and dementia may be performed.

Further, the estimation unit 123 uses at least two of the analysis map, the fundus photograph, the anterior eye portion photograph, the SLO image, the En-Face image, and the OCTA image as the trained model. You may enter it. For example, the image processing unit 122 generates a first symmetry map from the thickness map and a second symmetry map from the OCTA image, and the estimation unit 123 generates a first symmetry map and a second symmetry map. You may enter the sex map into the trained model. In this case, the training data of the trained model may be any symmetry map of the same type as the symmetry map input by the estimation unit 123 as input data. Further, the estimation unit 123 may perform estimation processing using the trained model generated for each of these symmetry maps. In this case, the estimation unit 123 may output the output from each trained model as an estimation result, or may output the output of one of the plurality of trained models as an estimation result. good. For example, the estimation unit 123 may output the estimation result of "abnormality" when the output of one of the learned models of the plurality of trained models is "abnormal".

As the En-Face image, for example, an SLO image, a fluorescently photographed fundus image, and data acquired by OCT (three-dimensional OCT data) are used for at least a part of the data in the depth direction of the object to be imaged. It may be a front image generated by the above. Further, the OCTA image is an OCTA En-Face image (motion contrast front image) generated by using data in at least a part of the depth direction of the object to be photographed for the three-dimensional OCTA data (three-dimensional motion contrast data). ) May be.

Here, the motion contrast data is data showing a change between a plurality of volume data obtained by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data showing changes between a plurality of tomographic images obtained at substantially the same position at different positions. The motion contrast front image is also referred to as an OCTA front image (OCTA En-Face image) relating to OCTA angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also referred to as OCTA data.

The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals. , Can be determined by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. When controlling the scanning means so that the measurement light is scanned multiple times at substantially the same position, the time interval (time interval) between one scan (one B scan) and the next scan (next B scan). ) May be modified (determined). Thereby, for example, even if the blood flow velocity may differ depending on the state of the blood vessel, the blood vessel region can be visualized with high accuracy.

At this time, for example, the time interval may be changed so as to be instructed by the examiner. Further, for example, any motion contrast image may be configured to be selectable from a plurality of motion contrast images corresponding to a plurality of preset time intervals according to an instruction from the examiner. Further, for example, the time interval at which the motion contrast data is acquired may be associated with the motion contrast data so that the motion contrast data can be stored in the storage unit 126. Further, for example, the display control unit 125 may display the time interval when the motion contrast data is acquired and the motion contrast image corresponding to the motion contrast data on the display unit 103. Further, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. At this time, for example, using a machine learning model, the time interval may be determined (output) from the motion contrast image. In such a machine learning model, for example, a plurality of motion contrast images corresponding to a plurality of time intervals are used as input data, and the difference from the plurality of time intervals to the time interval when a desired motion contrast image is acquired is correctly answered. Learning as data It can be obtained by learning the data.

Further, the En-Face image can be, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions. At this time, the front image is at least a part of the depth range of the volume data (three-dimensional tomographic image) obtained by using optical interference, and is the data corresponding to the depth range determined based on the two reference planes. Is projected or integrated on a two-dimensional plane. The En-Face image is a frontal image generated by projecting the data corresponding to the depth range determined based on the detected retinal layer among the volume data onto a two-dimensional plane. As a method of projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on the two-dimensional plane. Techniques can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Further, the depth range related to the En-Face image is, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries relating to the detected retinal layer. May be good. Further, the depth range related to the En-Face image may be, for example, a range changed (offset) according to the instruction of the operator from the range between the two layer boundaries regarding the detected retinal layer. good.

Further, in another example of the present embodiment, the image processing unit 122 cuts out a part of the region from the analysis map, OCTA image, En-Face image, fundus photograph, anterior eye portion photograph, SLO image, etc. in step S22. Images can be generated. For example, from a wide angle-of-view thickness map in which both the nipple and the macula are shown, a thickness map in which only the vicinity of the nipple is extracted or a thickness map in which only the vicinity of the macula is extracted is generated. The site to be extracted can be selected according to the disease to be diagnosed. Further, the image processing unit 122 generates a symmetry map for each of the extracted thickness maps. In other words, the image processing unit 122 can generate a symmetry map using the image of the region extracted from the medical image. The estimation unit 123 can perform estimation processing on the lesion from the symmetry map of each region generated in this way by using the trained model.

The estimation unit 123 relates to the disease, for example, from the estimation result estimated from the symmetry map for the vicinity of the macula using the trained model and the estimation result estimated from the symmetry map for the vicinity of the papilla using the trained model. The final estimation result can be output. The trained model used by the estimation unit 123 may be a model generated individually for each region or a common model. For example, the estimation unit 123 makes a final estimation if the result estimated from the symmetry map near the macula is "normal" but the result estimated from the symmetry map near the papilla is "glaucoma". The result can be output as "glaucoma". In addition, even if the result estimated from the symmetry map near the macula and the result estimated from the symmetry map near the papilla are both "normal", the estimation unit 123 has a certain probability of glaucoma or more. For example, "suspicion of glaucoma" may be output. Further, the estimation result in each region and the final result by the combination thereof may be further learned to generate the final machine learning model.

Here, an example in which only the symmetry map is input to the trained model has been described. However, as in the above-mentioned trained model, in addition to the symmetry map, a two-dimensional or three-dimensional tomographic image, an analysis map, and an OCTA are described. Images, fundus photographs, anterior segment photographs, SLO images, patient information, etc. may be input to the trained model. In this case, the learning data may also correspond. The image input to the trained model may be an image about the region related to the symmetry map.

Further, the analysis map such as the thickness map is not limited to the one including the macula and the papilla. For example, it may be a thickness map including a sieve plate. Further, the thickness map may include the macula, the papilla, and the lamina cribrosa.

Further, in another example of the present embodiment, the image processing unit 122 extracts from the image acquired by the acquisition unit 121 so that the correct answer rate of the estimation result is maximized according to the disease to be diagnosed. ROI: Region Of Interest) may be optimized. This optimization may be performed, for example, using a trained model different from the trained model for acquiring the estimation result. Further, this ROI may be uniformly specified based on medical knowledge after the image is aligned. For example, near the papilla for glaucoma, retinal vascular disease, age-related macular degeneration, or near diabetic retinopathy may be designated as ROI.

Further, the estimation unit 123 may evaluate the quality of the image in the process of estimation processing. The image quality evaluation method may be performed using any known method. For example, if the ROI is too small, there is not enough information to estimate the disease and the image quality may be underestimated. If it is determined that the image quality of the ROI is insufficient, the image processing unit 122 may perform a process of expanding the ROI.

Further, when the observation target portion of the eye is other than the fundus such as the anterior segment of the eye, the above processing can be performed in the same manner to perform a desired analysis. In this case, the training data may be of the same type as the data input to the trained model by the estimation unit 123, and may be, for example, a symmetry map generated using a medical image of the anterior eye portion. .. Further, the thickness map may be a thickness map showing the thickness of at least one layer of the anterior eye portion, not the thickness of at least one layer of the retina.

[Second Embodiment]
Next, the second embodiment will be described with reference to FIG. This embodiment estimates a plurality of ophthalmic diseases based on a plurality of types (modality) of images acquired at a single time. Since the configuration of the control unit according to the present embodiment is the same as that of the control unit 102 according to the first embodiment, the same reference numerals will be used and the description thereof will be omitted. Hereinafter, the control unit 102 according to the present embodiment will be described with a focus on the differences from the first embodiment. The estimation target according to the present embodiment is not limited to ophthalmic diseases, and may be systemic diseases.

By photographing the fundus of the eye to be inspected with different types of imaging techniques such as OCT and fundus camera, it is possible to acquire different information about the fundus. Therefore, more information can be obtained by combining these information. Therefore, the control unit 102 according to the present embodiment estimates a plurality of ophthalmic diseases using a plurality of types of images. Since the operation of the image processing according to the present embodiment is the same as the operation shown in the flowchart of FIG. 2, it will be described using the same reference numerals.

[Step S21: Data acquisition]
In step S21, the acquisition unit 121 according to the present embodiment acquires images captured by a plurality of imaging techniques using the imaging device 101. This imaging technique may include OCT, fundus camera, SLO, OCTA and the like. The examiner may specify a single or a plurality of symptoms to be diagnosed via the operation unit 104, and the acquisition unit 121 may select a combination of images to be acquired accordingly. For example, the acquisition unit 121 acquires either or both of a tomographic image and a fundus photograph when diagnosing glaucoma, and acquires a fundus photograph or an OCTA image when diagnosing diabetic retinopathy. In addition, the acquisition unit 121 may acquire both a tomographic image and a fundus photograph to diagnose both glaucoma and diabetic retinopathy. As in the first embodiment, the acquisition unit 121 may acquire a tomographic image, a fundus photograph, or the like from an external device (not shown). Further, the acquisition unit 121 may acquire an interference signal.

In one example of this embodiment, the acquisition unit 121 acquires a tomographic image and a fundus photograph. When taking both a tomographic image and a fundus photograph, the tomographic image can be acquired first, and then the fundus photograph can be acquired. This makes it possible to avoid the influence of miosis on the tomographic image during fundus photography. However, the shooting order is not limited to this, and may be changed according to a desired configuration.

[Step S22: Image processing]
In step S22, the image processing unit 122 generates a symmetry map from the tomographic image as in the first embodiment. The image processing unit 122 may generate a symmetry map using an analysis map such as a blood vessel density map, a fundus photograph, an anterior eye portion photograph, an SLO image, an OCTA image, an En-Face image, or the like. Further, the image processing unit 122 can select an image to be image-processed in order to generate a symmetry map according to the ophthalmologic disease to be diagnosed. For example, when glaucoma is one of the diagnosis targets, the image processing unit 122 can select a tomographic image or a thickness map as an image for generating a symmetry map. Further, for example, when diabetic retinopathy is one of the diagnostic targets, an OCTA image can be selected as an image for generating a symmetry map.

[Step S23: Estimate]
In step S23, the estimation unit 123 performs stepwise estimation processing for a plurality of diseases using the symmetry map generated in step S22, the image acquired in step S21, and the like. The estimation unit 123 performs estimation processing using a learned model that differs for each disease. Various trained models are stored in the storage unit 126. As described in the modified example of the first embodiment, the trained model may be provided in an external device connected to the control unit 102.

For example, the estimation unit 123 first estimates whether or not the eye to be inspected is opaque from at least one of the tomographic image and the fundus photograph. This estimation process is performed by image contrast evaluation or evaluation using a trained model. The trained model may be a machine learning model in which training data is trained using a tomographic image or a fundus photograph as input data and a label as to whether or not there is opacity as output data. The output data in this case may be generated by using the diagnosis result in which an expert such as a doctor diagnoses the eye to be inspected using a tomographic image or the like.

The estimation unit 123 outputs an estimation result indicating that the eye to be inspected may have cataract when there is a high probability that the output from the trained model is “opaque”. The estimation unit 123 may output an estimation result indicating that the eye to be inspected may have cataract when the probability of "having opacity" among the outputs from the trained model is higher than the threshold value. Further, when the probability of "no turbidity" or "normal" among the outputs from the estimated unit trained model is high, the estimation result indicating that the condition of the eye to be inspected is normal is output. The display control unit 125 causes the display unit 103 to display the estimation result output from the estimation unit 123. At this time, the image processing unit 122 may reduce blurring of images such as tomographic images and fundus photographs due to turbidity by contrast enhancement processing or the like.

Next, the estimation unit 123 performs estimation processing of whether or not the patient has diabetic retinopathy from the thickness map generated from the tomographic image or the symmetry map generated based on the fundus photograph or the like using the trained model. The trained model used for the estimation processing of diabetic retinopathy may be the same trained model as the trained model described in the first embodiment. In addition, it can be expected that the accuracy of the estimation process can be improved by using the symmetry map generated using the OCTA image for the estimation process of diabetic retinopathy. However, the estimation process of diabetic retinopathy may be performed using a symmetry map generated by using a thickness map, a fundus photograph, an En-Face image, or the like.

Next, the estimation unit 123 determines whether or not it is age-related macular degeneration by a machine learning model from a thickness map or a fundus photograph generated from a tomographic image. The trained model used for the estimation process of age-related macular degeneration may be the same trained model as the trained model described in the first embodiment. Further, the symmetry map generated by using the En-Face image or the OCTA image may be used for the estimation process of age-related macular degeneration.

Finally, the estimation unit 123 determines whether or not glaucoma is caused by a machine learning model from a thickness map or a fundus photograph generated from a tomographic image. The trained model used for the glaucoma estimation process may be the same trained model as the trained model described in the first embodiment. In addition, a symmetry map generated using an En-Face image or an OCTA image may be used for glaucoma estimation processing.

The estimation unit 123 outputs each of these estimation results. The display control unit 125 can display each estimation result output from the estimation unit 123 on the display unit 103. The estimation unit 123 may output the estimation results regarding cataract, diabetic retinopathy, age-related macular degeneration, and glaucoma to an external device (not shown).

As described above, in the control unit 102 according to the present embodiment, the trained model includes a plurality of trained models, and the estimation unit 123 is selected from the plurality of trained models according to the disease to be diagnosed. Output the estimation result estimated using the selected trained model. Further, the estimation unit 123 can output a plurality of stages of estimation results according to the disease estimation processing in order to perform the estimation processing of the plurality of diseases. The image processing unit 122 can also select an image to be image-processed according to the disease to be diagnosed.

By performing the estimation step by step in this way, each trained model may be a model that processes only individual symptoms. Therefore, it is possible to improve the estimation accuracy as compared with the processing for each disease with one trained model. In addition, the load of preparing training data can be reduced.

When the estimation unit 123 estimates the disease step by step, as an example, the estimation can be performed in order from the disease that is easy to estimate (high estimation accuracy). Further, when the disease is estimated step by step, the estimation unit 123 may omit the estimation process of the subsequent steps when the disease is estimated at any stage. In this case, for diseases that are difficult to estimate (low estimation accuracy), the possibility that the subject has a disease with higher estimation accuracy is excluded, so that the detection accuracy (of the disease) of the difference between normal and the disease is excluded. Estimation accuracy) can be improved. However, the order of estimation is not limited to the order of diseases that are easy to estimate, and may be arbitrarily set according to a desired configuration. Further, the estimation unit 123 may perform estimation processing for a plurality of diseases using different trained models in parallel.

Further, for example, when the disease of the eye to be presumed is mainly glaucoma, and when it is presumed to be diabetic retinopathy or age-related macular degeneration in advance, the display control unit 125 warns the display unit 103. It may be displayed. In this case, the display control unit 125 issues a warning such as "This patient cannot be determined" or "Be careful in determining glaucoma because there is a suspicion of diabetic retinopathy or age-related macular degeneration." It may be displayed on the display unit 103.

Further, for example, when the estimation unit 123 performs glaucoma estimation processing using both a fundus photograph and a tomographic image, even if one of the estimation results is "normal", the other estimation result is "glaucoma". If, the final estimation result can be output as "glaucoma". In this case, it is possible to prevent overlooking a case in which an ophthalmic disease such as glaucoma is suspected in one of the estimation results.

[Modified example of the second embodiment]
In this embodiment, an example of performing estimation processing of diabetic retinopathy, age-related macular degeneration, and glaucoma using different learned models has been described. However, at least one of these estimation processes may be performed by a rule-based estimation process based on the structure of the eye to be inspected or the like. The rule-based estimation process may be performed by any known method.

Further, in the present embodiment, an example of performing estimation processing of diabetic retinopathy, age-related macular degeneration, and glaucoma from a symmetry map has been described. However, at least one of these estimation processes may be performed using a tomographic image, an analysis map, an En-Face image, an OCTA image, a fundus photograph, an anterior ocular segment photograph, an SLO image, or the like. In this case, based on at least one of these images, estimation processing using a trained model that performs estimation processing of the disease may be performed, or rule-based estimation processing may be performed. The estimation process using these trained models and the rule-based estimation process may be any known one.

In another example of this embodiment, the trained model used by the estimation unit 123 has a plurality of channels into which each of the images captured by the plurality of imaging techniques is input, and classifies the plurality of labeled diseases. , May be one trained model. In this case, the estimation unit 123 can output, for example, a disease having a higher probability than the other estimation labels among the estimation labels output from the trained model as the final estimation result. Further, the estimation unit 123 may determine the estimation result by using the machine learning model from the probabilities of each estimation label obtained by using the trained model. The machine learning algorithm used in this case may be a different type of machine learning algorithm from the machine learning algorithm used to obtain the probability of the estimated label, for example, a neural network, a support vector machine, AdaBoost, a Basian network, etc. Alternatively, it may be a random forest or the like.

Further, in another example of the present embodiment, the data used by the estimation unit 123 may be data acquired for each of the plurality of layers (thickness map, symmetry map, etc. obtained in each layer). .. Examples of each of the plurality of layers include only the nerve fiber layer, from the nerve fiber layer to the inner plexiform layer, from the ganglion cell layer to the inner plexiform layer, or the entire retinal layer, and various other combinations can also be used.

In another example of the present embodiment, the estimation unit 123 individually estimates the images taken by each shooting technique and the data (thickness map, symmetry map, etc.) obtained from each layer, and then outputs the data. A disease having a higher probability than other diseases may be output as a final determination result. Further, the estimation unit 123 may determine the estimation result by using the machine learning model from the probabilities of each estimation label obtained by using the trained model. The machine learning algorithm used in this case may be a different type of machine learning algorithm from the machine learning algorithm used to obtain the probability of the estimated label, for example, a neural network, a support vector machine, AdaBoost, a Basian network, etc. Alternatively, it may be a random forest or the like.

[Third Embodiment]
Next, the third embodiment will be described with reference to FIGS. 6 to 8 (b). In this embodiment, estimation of ophthalmic diseases is made based on medical images acquired at different times. Since the configuration of the control unit according to the present embodiment is the same as that of the control unit 102 according to the first embodiment, the same reference numerals will be used and the description thereof will be omitted. Hereinafter, the control unit 102 according to the present embodiment will be described with a focus on the differences from the first embodiment. The estimation target according to the present embodiment is not limited to ophthalmic diseases, and may be systemic diseases.

In order to predict the prognosis of ophthalmic diseases, it is required to estimate the progress of the disease with high accuracy from a plurality of medical images (time-series data) obtained by photographing the eye to be inspected at different times. Therefore, the control unit 102 according to the present embodiment generates a correlation map in which the correlation value indicating the correlation between the data acquired at different times from the time series data is used as the pixel value, so that the progress of the disease is highly accurate. Estimate to. Since the operation of the image processing according to the present embodiment is the same as the operation shown in the flowchart of FIG. 2, it will be described using the same reference numerals.

[Step S21: Image input]
In step S21, the acquisition unit 121 according to the present embodiment acquires a plurality of fundus images obtained by photographing the subject at different times. For example, the acquisition unit 121 can acquire at least one three-dimensional tomographic image taken in the past from an existing database through a communication network or the like. In addition, the acquisition unit 121 can acquire a newly captured tomographic image from the imaging device 101. The past tomographic image may be stored in the storage unit 126 in advance and acquired from the storage unit 126.

As in the first embodiment, the acquisition unit 121 may acquire a newly captured fundus image from an external device (not shown). Further, the acquisition unit 121 may acquire interference signals related to a plurality of fundus images. Further, the acquisition unit 121 can also acquire a fundus photograph, an anterior eye portion photograph, an SLO image, an En-Face image, an OCTA image, patient data, and the like, as in the first embodiment.

[Step S22: Image processing]
In step S22, the image processing unit 122 compares the fundus images of the same eye to be inspected obtained by taking pictures at different times to generate a correlation map. An example of correlation map generation will be described with reference to the flowchart shown in FIG. 6 and the schematic diagram shown in FIG. 7.

First, in step S621, the image processing unit 122 generates a first thickness map 711 from the newly captured tomographic image 701. Similarly, a second thickness map 712 is generated from the tomographic image 702 taken in the past. The thickness map is generated by the same method as described in the first embodiment.

Next, in step S622, the image processing unit 122 performs correction processing on the first thickness map 711 and the second thickness map 712. An example of the correction process of the present embodiment can include a geometric correction process. Similar to the method described in the first embodiment, the image processing unit 122 scales the first thickness map 711 and the second thickness map 712 while referring to the reference information stored in the storage unit 126. Performs geometric correction processing such as enlargement or reduction), rotation, and translation. The reference information may be the distance and angle from the macula A to the papilla B of the thickness map generated from the tomographic image taken at the oldest time, and the positions of the macula A and the papilla B. In this case, the thickness map generated from the tomographic images taken at other times is corrected with reference to the reference information. Further, the image used for acquiring the reference information is not limited to the oldest image, and may be an image taken at an arbitrary time. As the reference information, the information described in the first embodiment may be used.

Further, an example of the correction process of the present embodiment can include a process of smoothing the thickness map. By the smoothing process, it is possible to reduce the influence of the positional deviation and the rotational deviation between the tomographic images taken at different times. The correction process may include other image processes such as correction of the luminance value of the inappropriate portion and correction according to the axial length.

Next, in step S623, the image processing unit 122 generates a correlation map by comparing the corrected first thickness map 721 and the corrected second thickness map 722. In an example of the present embodiment, the image processing unit 122 calculates and calculates the ratio of the brightness value of the corrected first thickness map 721 to the brightness value of the corrected second thickness map 722 at each XY position. A correlation map is generated by using the ratio as the pixel value at the position.

Further, the image processing unit 122 generates a thickness map from each of the plurality of tomographic images taken in the past, and sets the brightness value of the thickness map taken at the oldest time to the brightness value of the other thickness map. A ratio may be calculated, and a plurality of correlation maps may be generated using the calculated ratio as a pixel value. As a result, the image processing unit 122 can generate time-series data of the correlation map. In addition, the target for calculating the ratio is not limited to the thickness map obtained from the image taken at the oldest time, but also the thickness map obtained from the image taken at another time and the thickness obtained from the normal image. It may be a map. In this embodiment, the thickness map is used, but other analysis maps such as the blood vessel density map may be used.

Further, in another example of the present embodiment, the image processing unit 122 may generate a first symmetry map from the first thickness map and generate a second symmetry map from the second thickness map. can. The method of generating the symmetry map is the same as that of the first embodiment. A correlation map is generated by taking the ratio of the luminance value of the first symmetry map to the luminance value of the second symmetry map at each XY position of the symmetry map.

Further, here, the correlation map is generated by using the ratio of the luminance values of the thickness maps at different times as the pixel values. However, as the pixel value of the correlation map, the difference in the luminance value of the thickness map at different times may be used instead of the ratio. Further, the pixel value calculation method may include a logarithmic transformation process of the luminance value.

[Step S23: Estimate]
When a plurality of correlation maps are generated in step S623 and output to the estimation unit 123, the process proceeds to step S23. In step S23, the estimation unit 123 estimates the progression of the disease of the eye to be inspected from the correlation map input from the image processing unit 122 by using the trained model obtained by learning using the correlation map as learning data.

In one example of the present embodiment, the estimation unit 123 outputs an estimation result that estimates the presence or absence of disease progression and its probability. Further, the estimation unit 123 may output an estimation result of estimating how much the symptom has progressed after a lapse of a predetermined time (in the future). The training data of the trained model in this case is a correlation map generated from the tomographic image of the training data as input data, and a label indicating the presence or absence of subsequent progress and how severe the disease has become after a predetermined time has elapsed. May be generated as output data.

When the time-series data of the correlation map is input to the trained model, the input data of the training data can also be the time-series data of the correlation map. In this case, the time interval between the images related to the time series data may be various or constant. However, if the time interval between the images is constant for the time-series data of the correlation data that is the input data of the training data, the time between the images related to the time-series data of the correlation data that the estimation unit 123 inputs to the trained model. The interval shall be the same time interval.

Further, in an example of the present embodiment, the storage unit 126 individually provides a trained model for estimating a disease at a certain time point or at the first visit and a trained model for estimating the progression from time series data. good. As a result, it is possible to avoid the estimation process in which data is insufficient at a certain point in time or at the time of the first medical examination, and perform an appropriate disease estimation process.

As described above, in the control unit 102 according to the present embodiment, the fundus image, which is an example of the medical image, includes a plurality of fundus images obtained by photographing the subject eye at different times. Further, the image processing unit 122 performs image processing on a plurality of fundus images to generate a correlation map showing the correlation between the images. The estimation unit 123 outputs the estimation result regarding the ophthalmic disease estimated from the generated correlation map by using the trained model obtained by the learning using the correlation map. With such a configuration, the control unit 102 according to the present embodiment can accurately estimate the presence or absence of disease progression and its probability by using the time-series data of the fundus image.

When the time-series data of the correlation map is used as the input data, a recurrent neural network (RNN), which is a neural network that handles the time-series information, can be used as the trained model. FIG. 8A shows the structure of the RNN, which is a machine learning model. The RNN 82 shown in FIG. 8A has a loop structure in the network, ^{inputs data x t} 81 at time t, and outputs ^{data h t 83.} Since the RNN 82 has a loop function in the network, the current state can be inherited to the next state, so that time-series information can be handled. FIG. 8B shows an example of input / output of the parameter vector at time t. The data x ^t 81 contains N pieces of data (Params1 to ParamsN). Further, the data h ^t 83 output from the RNN 82 includes N pieces (Params1 to ParamsN) of data corresponding to the input data.

However, since RNN cannot handle long-term time information during error back propagation, long-short-term memory (LSTM: Long Short-Term Memory) may be used. The LSTM is a kind of RNN, and can learn long-term information by including a forgetting gate, an input gate, and an output gate. LSTMs can also be used for longer-term information-based forecasts. Here, FIG. 9A shows the structure of the LSTM. In LSTM94, the information that the network takes over at the next time t is the internal state ct ^{-1 of the} network called the cell and the output data h ^t-1 . The lowercase letters (c, h, x) in the figure represent vectors.

Next, FIG. 9B shows the details of LSTM94. In FIG. 9B, the oblivion gate network FG, the input gate network IG, and the output gate network OG are shown, each of which is a sigmoid layer. Therefore, a vector in which each element has a value of 0 to 1 is output. The oblivion gate network FG determines how much past information is retained, and the input gate network IG determines which value to update. Further, in FIG. 9B, the cell update candidate network CU is shown, and the cell update candidate network CU is the activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gate network OG selects the element of the cell candidate and selects how much information is to be transmitted at the next time.

Since the above-mentioned LSTM model is a basic model, it is not limited to the network shown here. You may change the coupling between the networks. QRNN (Quasi Recurrent Neural Network) may be used instead of LSTM. Further, the machine learning model is not limited to the neural network, and boosting, a support vector machine, or the like may be used.

[Modified example of the third embodiment]
In this embodiment, the image used to generate the correlation map is a tomographic image. However, the image used to generate the correlation map is not limited to this, and may be a fundus photograph, an anterior eye portion photograph, an analysis map, an SLO image, an En-Face image, an OCTA image, or the like.

Further, the image processing unit 122 generates a symmetry map from each of the plurality of tomographic images taken in the past, and the ratio of the brightness value of the symmetry map of the oldest time to the brightness value of the other symmetry map. May be calculated at each XY position, and a correlation map may be generated using the ratio as a pixel value. Further, by the same processing, the image processing unit 122 can also generate time-series data (a plurality of correlation maps) of the correlation map. As the method for generating the symmetry map, the method described in the first embodiment may be used. The images used to generate the symmetry map are not limited to tomographic images, but are analysis maps such as thickness maps and blood vessel density maps, fundus photographs, anterior segment photographs, SLO images, En-Face images, OCTA images, and the like. There may be.

The target for calculating the ratio is not limited to the symmetry map obtained from the image taken at the oldest time, but also the symmetry map obtained from the image taken at another time and the normal image. It may be a symmetry map to be obtained. As the pixel value of the correlation map, the difference in the luminance value of the symmetry map at different times may be used instead of the ratio. Further, the pixel value calculation method may include a logarithmic transformation process of the luminance value.

As described above, in another example of the present embodiment, the image processing unit 122 generates a correlation map showing the correlation between a plurality of time symmetry maps obtained by image processing a plurality of fundus images. The estimation unit 123 outputs an estimation result of estimating the progression of an ophthalmic disease from a correlation map or time-series data of the correlation map. The estimation unit 123 can also output an estimation result for estimating the presence or absence of disease progression and its probability, as in the third embodiment, by performing the estimation process using the correlation map generated in this way. In this case, as the input data of the trained model used by the estimation unit 123, a correlation map generated by using the symmetry map can be used. In addition, the trained model in this case is a trained model for estimating eye disease from a single-time symmetry map, and the progression of eye disease based on a correlation map generated from time-series data of the symmetry map. May include a trained model for estimating.

Further, in another example of the present embodiment, the estimation unit 123 is derived from at least one of the first analysis map or the first symmetry map and at least one of the second analysis map or the second symmetry map. , The trained model may be used to estimate glaucoma at each time. The estimation process may be the same as the estimation process described in the first embodiment. In this case, the estimation unit 123 can acquire an attention map showing the region of interest in these trained models. The estimation unit 123 acquires the first attention map by the estimation process for the first analysis map or the first symmetry map, and the second analysis map or the second symmetry map is estimated by the second analysis map or the second symmetry map. Get the attention map of. In such a case, the image processing unit 122 calculates the ratio of the luminance value of the first attention map to the luminance value of the second attention map at each XY position of the attention map, and the calculated ratio is the pixel value. By doing so, a correlation map can be generated.

Further, the image processing unit 122 generates a caution map from each of a plurality of analysis maps or symmetry maps generated using the tomographic images taken in the past, and with respect to the brightness value of the caution map at the oldest time. A ratio to the brightness value of another caution map may be calculated at each XY position, and a plurality of correlation maps may be generated using the ratio as a pixel value. As a result, the image processing unit 122 can generate time-series data of the correlation map.

The target for calculating the ratio is not limited to the caution map obtained from the image taken at the oldest time, but also the caution map obtained from the image taken at another time and the caution obtained from the normal image. It may be a map. As the pixel value of the correlation map, the difference in the luminance value of the symmetry map at different times may be used instead of the ratio. Further, the pixel value calculation method may include a logarithmic transformation process of the luminance value. Further, the image processing unit 122 may output the time series data of the attention map itself to the estimation unit 123.

The estimation unit 123 also outputs an estimation result that estimates the presence or absence of disease progression and its probability, as in the third embodiment, even by performing estimation processing using the attention map or correlation map generated in this way. Can be done. In this case, as the input data of the trained model used by the estimation unit 123 for the estimation process of the presence / absence of disease progression and its probability, a caution map or a correlation map generated by using the caution map can be used.

As described above, according to the first to third embodiments described above, the symmetry map showing the symmetry of the eye to be inspected and the correlation map showing the correlation between the images taken at different times are used to show the eye to be inspected. Can make estimates about the disease. This makes it possible to improve the estimation accuracy of diseases such as glaucoma in which findings appear in features such as symmetry and correlation.

[Modification 1]
In the first modification according to the present invention, the image processing unit 122 can use the interference signal due to the reference light and the return light from the eye acquired by the photographing apparatus 101 for the image processing. The tomographic image is generally obtained by Fourier transforming the interference signal, but the image processing unit 122 may use the interference signal before the Fourier transform for image processing. The image processing unit 122 may perform processing for removing an abnormal portion (signal of a different frequency, spike noise, etc.) from the interference signal before the Fourier transform acquired by the acquisition unit 121. After removing the abnormal part, Fourier transform can be performed to generate a tomographic image, and the same processing as described above can be performed.

Further, the image processing unit 122 may output the interference signal before the Fourier transform to the estimation unit 123. In this case, the estimation unit 123 outputs an estimation result in which information on the disease of the eye to be inspected is estimated from the interference signal output from the image processing unit 122 by using the learned model obtained by learning using the interference signal as learning data. can do.

[Modification 2]
In the second modification of the present invention, the image processing unit 122 plots the thickness of the target layer from the tomographic image data 1001 as shown in FIG. 10A generated in a certain direction from the three-dimensional tomographic image. The profile 1002 as shown in FIG. 10 (b) can be generated. Further, each thickness may be replaced with a relative value that is easy to evaluate. The image processing unit 122 may output this profile 1002 to the estimation unit 123. In this case, the estimation unit 123 outputs the estimation result regarding the disease of the eye to be inspected from the profile 1002 generated by the image processing unit 122 by using the trained model obtained by learning using such a profile as training data. Can be done.

Further, the image processing unit 122 may output to the estimation unit 123 a feature vector in which the thickness values at each position are arranged in a vector instead of the profile format. In this case, the estimation unit 123 outputs an estimation result in which information on the disease of the eye to be inspected is estimated from the feature vector generated by the image processing unit 122 using the trained model obtained by learning using the feature vector as training data. You may.

[Modification 3]
In the modification 3 according to the present invention, the image processing unit 122 can generate a feature map in which the shape of a part of the feature section is extracted from the acquired fundus photograph or the like and the contour or the like is plotted. Examples of features can be optic disc depressions, optic discs, RNFLs (retinal nerve fiber bundles) and the like. As a method for extracting the shape of the feature portion, a known segmentation process may be used. Further, the segmentation process may be a rule-based process based on the structure of the eye to be inspected or the like, or may be performed by using a U-net type trained model or a trained model for region detection.

The image processing unit 122 may output the generated feature map to the estimation unit 123. In this case, the estimation unit 123 outputs an estimation result in which information on the disease of the subject is estimated from the feature map generated by the image processing unit 122 using the trained model obtained by learning using the feature map as training data. can do. By such treatment, it can be expected that symptoms such as glaucoma in which optic disc depression and findings appear in the shape of RNFL can be estimated with higher accuracy.

[Modification 4]
In the modification 4 according to the present invention, the image processing unit 122 can output the two-dimensional tomographic image obtained by the photographing apparatus 101 to the estimation unit 123. The estimation unit 123 outputs an estimation result in which information on the disease of the subject is estimated from the tomographic image output from the image processing unit 122 by using a machine learning model obtained by learning using a two-dimensional tomographic image as learning data. can do. In this case, it can be expected that symptoms such as age-related macular degeneration and glaucoma in which findings appear on tomographic images can be estimated with high accuracy.

The image processing unit 122 may generate a three-dimensional tomographic image (three-dimensional volume data) from the acquired tomographic image of each scan position and output it to the estimation unit 123. The estimation unit 123 estimates information about the disease of the subject from the three-dimensional volume data generated by the image processing unit 122 by using a machine learning model obtained by learning using the three-dimensional volume data as learning data. The result can be output. In this case, it can be expected that symptoms such as glaucoma in which findings appear in a three-dimensional structure can be estimated with high accuracy.

[Modification 5]
In the modified example 5 according to the present invention, the image processing unit 122 generates a symmetry map as in the first embodiment, and at the same time, the profile, feature vector, feature map, and three-dimensional volume data related to the layer thickness described above are described. Generate at least one. In addition to the symmetry map, the estimation unit 123 uses the symmetry map and a trained model obtained by learning using any of the above-mentioned data or tomographic images as training data to relate to the disease of the subject from the symmetry map and these data. It is possible to output the estimation result of estimating the information. In this case, various effects described in Modifications 2 to 4 can be obtained depending on the type of data used as input data. In this case, the symmetry map and the above-mentioned data may be used as the input data of the learning data, and the output data may be the same as the output data of the learning data described in the first embodiment or the like.

Further, the estimation unit 123 may perform estimation processing using the measurement results of the perimeter in addition to the symmetry map and the various data described above. In this case, as the training data of the trained model, the measurement result by the perimeter may be included in the input data in addition to the symmetry map and the various data described above. In this case, it can be expected that the features included in the measurement result by the perimeter can be used for the estimation process, and the estimation accuracy regarding the disease of the eye to be inspected can be further improved.

In addition, the control unit 102 can output the estimation result regarding the disease of the eye to be inspected by the same processing in the second embodiment and the third embodiment.

Further, in the first to third embodiments, the estimation unit 123 performs estimation processing regarding the disease of the subject from the symmetry map or the correlation map. On the other hand, the estimation unit 123 may perform estimation processing regarding the disease of the subject from the measurement result of the perimeter. In this case, as the training data of the trained model, the measurement result by the perimeter may be used as the input data. The output data may be the same as that described in the first embodiment or the like. In addition to the measurement results of the perimeter, the estimation unit 123 includes a two-dimensional tomographic image, a three-dimensional tomographic image, an analysis map, a fundus photograph, an anterior segment photograph, an SLO image, an En-Face image, an OCTA image, and the like. At least one of the fundus images, which is an example of the medical image, may be used for estimation processing related to the disease of the eye to be examined. Further, the estimation unit 123 may perform estimation processing on the disease of the subject using at least one of the angle information, the profile related to the layer thickness, the feature vector, and the feature map in addition to the measurement result of the perimeter. good. In this case, the training data of the trained model may be input data of the same type as the data used by the estimation unit 123 for the estimation process. In such a process, the disease can be estimated with high accuracy based on the measurement result of the perimeter.

[Modification 6]
Learning may be performed to adjust (tune) the learned model (learned model for estimation) used for the estimation process regarding the disease of the subject for each subject, and the trained model dedicated to the subject may be generated. .. For example, using the tomographic images acquired in the past examination of the subject, transfer learning of a general-purpose trained model for estimating the disease of the subject is performed, and the trained model dedicated to the subject is obtained. Can be generated. By associating the trained model dedicated to the subject with the ID of the subject and storing it in an external device such as a storage unit 126 or a server, the control unit 102 performs the current inspection of the subject. In addition, a trained model dedicated to the subject can be specified and used based on the ID of the subject. By using a trained model dedicated to the subject, it is possible to improve the estimation accuracy of the disease for each subject.

[Modification 7]
The control unit 102 may perform various image processing using an image or the like acquired by shooting. For example, the control unit 102 may generate a high-quality image with improved image quality by using a trained model for high image quality (high image quality model) for the image acquired by shooting. Here, the improvement of image quality includes reduction of noise, conversion to colors and gradations that make it easy to observe the object to be photographed, improvement of resolution and spatial resolution, and enlargement of image size while suppressing deterioration of resolution.

As a machine learning model for improving image quality, for example, CNN or the like can be used. Further, as the training data of the high image quality model, various images such as an anterior eye image and an SLO image are used as input data, and high quality images corresponding to the input images, for example, which have undergone high image quality processing, are output data. And. Here, the high image quality processing includes alignment of images taken at the same spatial position a plurality of times, and addition and averaging of the aligned images. The high image quality processing is not limited to the addition averaging processing, and may be, for example, a processing using a smoothing filter, a maximum a posteriori estimation processing (MAP estimation processing), a gradation conversion processing, or the like. The high-quality image may be, for example, an image that has undergone filter processing such as noise removal and edge enhancement, or an image whose contrast has been adjusted so as to change from a low-luminance image to a high-brightness image. May be used. Further, since the output data of the training data related to the high image quality model may be a high quality image, the image is taken using an OCT device having higher performance than the OCT device when the tomographic image which is the input data is taken. It may be an image taken or an image taken with a high load setting.

However, if machine learning is performed using an image that has not been properly image-enhanced as output data of training data, the image obtained by using the trained model trained using the training data is also appropriately high. There is a possibility that the image will not be image-enhanced. Therefore, by removing the pair containing such an image from the teacher data, it is possible to reduce the possibility that an inappropriate image is generated by using the trained model.

The control unit 102 can acquire an image with high image quality with high accuracy at a higher speed by performing high image quality processing using such a high image quality model.

The high image quality model may be prepared for each type of various images as input data. For example, a high image quality model for an anterior eye image, a high image quality model for an SLO image, a high image quality model for a tomographic image, a high image quality model for an OCTA front image, and the like may be prepared. Further, for the OCTA front image and the En-Face image, a high image quality model may be prepared for each depth range for generating an image. For example, a high image quality model for the surface layer, a high image quality model for the deep layer, and the like may be prepared. Further, the high image quality model may be one in which learning is performed on an image for each imaging site (for example, the center of the macula and the center of the optic nerve head), or may be one in which learning is performed regardless of the imaging site. ..

At this time, for example, using a high image quality model obtained by learning the fundus OCTA front image as training data, the fundus OCTA front image is improved in image quality, and further, the anterior eye OCTA front image is learned as training data. The high image quality model may be used to improve the image quality of the frontal image of the front eye OCTA. Further, the high image quality model may be one that has been learned regardless of the imaging region. Here, for example, the fundus OCTA frontal image and the anterior eye OCTA frontal image may have relatively similar distributions of blood vessels to be imaged. As described above, in a plurality of types of medical images in which the appearances of the objects to be imaged are relatively similar to each other, the feature quantities may be relatively similar to each other. Therefore, for example, by using a high image quality model obtained by learning the fundus OCTA front image as learning data, it is possible not only to improve the image quality of the fundus OCTA front image but also to improve the image quality of the anterior eye OCTA front image. May be done. Further, for example, by using a high image quality model obtained by learning the anterior eye OCTA front image as learning data, not only the anterior eye OCTA front image can be improved, but also the fundus OCTA front image can be improved. It may be configured. That is, using a high-quality model obtained by learning at least one type of frontal image of the fundus OCTA frontal image and the anterior eye OCTA frontal image as training data, the fundus OCTA frontal image and the anterior eye OCTA frontal image are At least one type of front image may be configured to have high image quality.

Here, consider a case where the anterior eye can also be photographed in the OCT device capable of photographing the fundus. At this time, for example, the fundus OCTA frontal image may be applied to the OCTA En-Face image in the fundus photography mode, and the anterior eye OCTA frontal image may be applied in the anterior ocular segment imaging mode. At this time, when the high image quality button is pressed, for example, in the fundus photography mode, in the display area of the En-Face image of OCTA, among the low image quality fundus OCTA front image and the high image quality fundus OCTA front image. One display may be configured to change to the other display. When the high image quality button is pressed, for example, in the anterior segment imaging mode, a low-quality front eye OCTA front image and a high-quality front eye OCTA front image are displayed in the OCTA En-Face image display area. The display of one of the above may be changed to the display of the other.

In the OCT device capable of photographing the fundus, the anterior eye adapter may be configured to be attachable when the anterior eye can also be imaged. Further, the optical system of the OCT device may be configured to be movable at a distance of about the axial length of the eye to be inspected without using the anterior eye adapter. At this time, the focus position of the OCT device may be configured to be largely changeable on the emmetropic side to the extent that an image is formed on the front eye.

Further, for example, a fundus OCT tomographic image may be applied to the tomographic image in the fundus imaging mode, and an anterior ocular OCT tomographic image may be applied in the anterior segment imaging mode. Further, the above-mentioned high image quality processing of the fundus OCTA frontal image and the anterior ocular OCTA frontal image can be applied as, for example, high image quality processing of the fundus OCT tomographic image and the anterior ocular OCT tomographic image. At this time, when the high image quality button is pressed, for example, in the fundus photography mode, one of the low image quality fundus OCT tomographic image and the high image quality fundus OCT tomographic image is displayed in the tomographic image display area. It may be configured to change to the other display. When the high image quality button is pressed, for example, in the anterior segment imaging mode, one of a low-quality anterior OCT tomographic image and a high-quality anterior OCT tomographic image is displayed in the tomographic image display area. The display of is changed to the other display.

Further, for example, the fundus OCTA tomographic image may be applied to the tomographic image in the fundus photography mode, and the anterior eye OCTA tomographic image may be applied to the tomographic image in the anterior ocular segment imaging mode. Further, the above-mentioned high image quality processing of the fundus OCTA front image and the anterior ocular OCTA front image can be applied as, for example, high image quality processing of the fundus OCTA tomographic image and the anterior ocular OCTA tomographic image. At this time, for example, in the fundus photography mode, in the tomographic image display area, the information indicating the blood vessel region (for example, motion contrast data above the threshold value) in the fundus OCTA tomographic image is superimposed on the fundus OCT tomographic image at the corresponding position. It may be configured to be displayed. Further, for example, in the anterior segment imaging mode, information indicating a vascular region in the anterior ocular OCTA tomographic image may be superimposed and displayed on the anterior ocular OCT tomographic image at a corresponding position in the tomographic image display region. ..

In this way, for example, when it is considered that the feature quantities (the appearance of the imaged object) of a plurality of types of medical images are relatively similar to each other, at least one type of the plurality of types of medical images may be used. Using a high image quality model obtained by learning a medical image as training data, at least one type of medical image of a plurality of types of medical images may be configured to have high image quality. Thereby, for example, a common trained model (common high image quality improvement model) can be used to enable execution of high image quality improvement of a plurality of types of medical images.

The display screen of the fundus photography mode and the display screen of the anterior eye part photography mode may have the same display layout or may have a display layout corresponding to each photography mode. Various conditions such as imaging conditions and analysis conditions may be the same or different between the fundus imaging mode and the anterior segment imaging mode.

Here, the target image for high image quality processing may be, for example, a plurality of OCTA front images (OCTA En-Face images, motion contrast En-Face images) (corresponding to a plurality of depth ranges). Further, the target image for high image quality processing may be, for example, one OCTA front image corresponding to one depth range. Further, the target image of the high image quality processing is, for example, a front image of luminance (En-Face image of luminance), an OCT tomographic image which is a B scan image, or a tomographic image of motion contrast data (instead of the front image of OCTA). OCTA tomographic image) may be used. Further, the target image of the high image quality processing is not only the OCTA front image, but also various medical applications such as the front image of brightness, the OCT tomographic image which is a B scan image, and the tomographic image of motion contrast data (OCTA torpedo image). It may be an image. That is, the target image for the high image quality processing may be, for example, at least one of various medical images displayed on the display screen of the display unit 103. At this time, for example, since the feature amount of the image may be different for each type of image, a trained model for high image quality corresponding to each type of the target image for high image quality processing may be used. For example, when the high image quality button is pressed in response to an instruction from the examiner, the OCTA front image is not only processed for high image quality by using the trained model for high image quality corresponding to the OCTA front image. The OCT tomographic image may also be configured to be processed for high image quality by using a trained model for high image quality corresponding to the OCT tomographic image. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, the high image quality OCTA front image generated by using the trained model for high image quality corresponding to the OCTA front image is displayed. It may be configured to be changed to display a high-quality OCT tomographic image generated by using a trained model for high image quality corresponding to the OCT tomographic image. At this time, the line indicating the position of the OCT tomographic image may be configured to be superimposed and displayed on the OCTA front image. Further, the line may be configured to be movable on the OCTA front image according to an instruction from the examiner. Further, when the display of the high image quality button is in the active state, after the above line is moved, the high image quality OCT tomographic image obtained by performing high image quality processing on the OCT tomographic image corresponding to the position of the current line is performed. It may be configured to be modified to display an image. Further, by displaying the high image quality button for each target image of the high image quality processing, the high image quality processing may be independently enabled for each image.

Further, the information indicating the blood vessel region (for example, motion contrast data equal to or higher than the threshold value) in the OCTA tomographic image may be superimposed and displayed on the OCT tomographic image which is the B scan image at the corresponding position. At this time, for example, when the image quality of the OCTA tomographic image is improved, the image quality of the OCTA tomographic image at the corresponding position may be improved. Then, the information indicating the blood vessel region in the OCTA tomographic image obtained by improving the image quality may be superimposed and displayed on the OCTA tomographic image obtained by improving the image quality. The information indicating the blood vessel region may be any information as long as it is identifiable information such as color. Further, the superimposed display and non-display of the information indicating the blood vessel region may be configured to be changeable according to an instruction from the examiner. Further, when the line indicating the position of the OCT tomographic image is moved on the OCTA front image, the display of the OCT tomographic image may be updated according to the position of the line. At this time, since the OCTA tomographic image at the corresponding position is also updated, the superimposed display of the information indicating the blood vessel region obtained from the OCTA tomographic image may be updated. Thereby, for example, the three-dimensional distribution and state of the blood vessel region can be effectively confirmed while easily confirming the positional relationship between the blood vessel region and the region of interest at an arbitrary position. Further, the image quality of the OCTA tomographic image may be improved by an addition averaging process or the like of a plurality of OCTA tomographic images acquired at the corresponding positions instead of using the trained model for the image quality improvement. .. Further, the OCT tomographic image may be a pseudo OCT tomographic image reconstructed as a cross section at an arbitrary position in the OCT volume data. Further, the OCTA tomographic image may be a pseudo OCTA tomographic image reconstructed as a cross section at an arbitrary position in the OCTA volume data. It should be noted that the arbitrary position may be at least one arbitrary position, and may be configured to be changeable according to an instruction from the examiner. At this time, a plurality of pseudo tomographic images corresponding to a plurality of positions may be configured to be reconstructed.

Only one tomographic image (for example, OCT tomographic image or OCTA tomographic image) may be displayed, or a plurality of tomographic images may be displayed. When a plurality of tomographic images are displayed, the tomographic images acquired at positions in different sub-scanning directions may be displayed, or the plurality of tomographic images obtained by, for example, cross-scanning may be displayed in high quality. When displaying, images in different scanning directions may be displayed respectively. Further, when displaying a plurality of tomographic images obtained by, for example, a radial scan with high image quality, a plurality of partially selected tomographic images (for example, two tomographic images at positions symmetrical with respect to a reference line) are displayed. ) May be displayed respectively. Furthermore, a plurality of tomographic images are displayed on the follow-up display screen (follow-up display screen), and instructions for improving the image quality and analysis results (for example, the thickness of a specific layer, etc.) are displayed by the same method as the above method. ) May be displayed. At this time, the plurality of tomographic images displayed may be a plurality of tomographic images obtained at different dates and times at a predetermined site of the eye to be inspected, or a plurality of tomographic images obtained at different times on the same inspection day. May be good. Further, the tomographic image may be subjected to high image quality processing based on the information stored in the database by the same method as the above method.

Similarly, when displaying an SLO image with high image quality, for example, the SLO image displayed on the same display screen may be displayed with high image quality. Further, when the front image of the brightness is displayed with high image quality, for example, the front image of the brightness displayed on the same display screen may be displayed with high image quality. Further, a plurality of SLO images and frontal images of luminance are displayed on the display screen for follow-up observation, and instructions for improving the image quality and analysis results (for example, the thickness of a specific layer) are displayed by the same method as the above method. May be done. Further, the high image quality processing may be executed on the SLO image or the front image of the brightness based on the information stored in the database by the same method as the above method. The display of the tomographic image, the SLO image, and the frontal image of the luminance is an example, and these images may be displayed in any manner depending on the desired configuration. Further, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the brightness front image may be displayed with high image quality by one instruction.

With such a configuration, the display control unit 125 can display the high-quality image obtained by the high-quality processing on the display unit 103. If at least one of a plurality of conditions relating to the display of a high-quality image, the display of the analysis result, the depth range of the displayed front image, etc. is selected, the selection is made even if the display screen is changed. It may be configured to maintain the specified conditions. The display control unit 125 may control the display of various high-quality images, the above lines, information indicating the blood vessel region, and the like.

Further, the high image quality model may be used for at least one frame of the live moving image in the preview screen displayed on the display unit 103 by the display control unit 125. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the trained model corresponding to each live moving image may be used. For example, as for the anterior eye image used for the alignment process, an image having a high image quality using a high image quality model for the anterior eye image may be used. Similarly, for various images used for the detection process of a predetermined region in various images, an image whose image quality has been improved by using a high image quality model for each image may be used.

At this time, for example, when the high image quality button is pressed in response to an instruction from the examiner, a plurality of different types of live moving images (for example, anterior eye image, SLO image, tomographic image) are displayed (for example). At the same time), it may be configured to be changed to display a high-quality moving image obtained by performing high-quality processing. At this time, the display of the high-quality moving image may be a continuous display of the high-quality image obtained by performing high-quality processing on each frame. Further, for example, since the feature amount of the image may be different for each type of image, a trained model for high image quality corresponding to each type of the target image for high image quality processing may be used. For example, when the high image quality button is pressed in response to an instruction from the examiner, not only the high image quality processing of the anterior eye image is performed using the high image quality model corresponding to the anterior eye image, but also the SLO image is supported. The SLO image may also be configured to perform high image quality processing using the high image quality improvement model. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, the display is changed to the high image quality front eye image generated by using the high image quality model corresponding to the front eye image. Not only that, it may be configured to be changed to display a high-quality SLO image generated by using a high-quality model corresponding to the SLO image. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, not only the high image quality processing of the SLO image is performed using the high image quality model corresponding to the SLO image, but also the tomographic image is supported. The tomographic image may also be configured to be processed for high image quality by using the high image quality model. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, the display is simply changed to the high image quality SLO image generated by using the high image quality model corresponding to the SLO image. Instead, it may be configured to be modified to display a high quality tomographic image generated using a high image quality model corresponding to the tomographic image. At this time, the line indicating the position of the tomographic image may be configured to be superimposed and displayed on the SLO image. Further, the line may be configured to be movable on the SLO image according to an instruction from the examiner. Further, when the display of the high image quality button is in the active state, after the above line is moved, the tomographic image corresponding to the position of the current line is processed to have high image quality to obtain a high image quality tomographic image. It may be configured to change to display. Further, by displaying the high image quality button for each target image of the high image quality processing, the high image quality processing may be independently enabled for each image.

As a result, for example, even if it is a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of shooting. Therefore, for example, when the operator corrects the alignment position while checking the preview screen, it is possible to reduce the failure of re-shooting and the like, so that the accuracy and efficiency of the diagnosis can be improved. Further, the control unit 102 scans the above-mentioned scan so that the partial region such as the artifact region obtained by the segmentation process or the like is photographed (rescanned) again during or at the end of the imaging in response to the instruction regarding the start of imaging. The means may be driven and controlled. Depending on the movement of the eye to be inspected, it may not be possible to take a good picture by one rescan. Therefore, the drive may be controlled so that the rescan is repeated a predetermined number of times. At this time, even during a predetermined number of rescans, the rescan may be terminated in response to an instruction from the operator (for example, after the shooting cancel button is pressed). At this time, it may be configured to save the shooting data until the rescan is completed according to the instruction from the operator. For example, a confirmation dialog may be displayed after the shooting cancel button is pressed, and the shooting data may be saved or the shooting data may be discarded according to an instruction from the operator. Also, for example, after pressing the shooting cancel button, the next rescan is not executed (although the current rescan is executed until it is completed), and it waits until there is an instruction (input) from the operator in the confirmation dialog. It may be configured as follows. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, shooting start, etc. may be automatically performed. good. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, the start of imaging, etc. can be executed according to the instruction from the examiner. It may be configured to change to a new state (release the execution prohibited state).

Here, during auto-alignment, it is possible that the imaged object such as the retina of the eye E to be inspected has not yet been successfully imaged. Therefore, since there is a large difference between the medical image input to the trained model and the medical image used as the training data, there is a possibility that a high-quality image cannot be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B scan image) exceeds the threshold value, the display of the high-quality moving image (continuous display of high-quality frames) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image exceeds the threshold value, the image quality improvement button may be configured to be changed to a state (active state) that can be specified by the examiner. The high image quality button is a button for designating the execution of the high image quality processing. Of course, the high image quality button may be a button for instructing the display of a high image quality image.

Further, a different high image quality model may be prepared for each shooting mode having a different scan pattern or the like, and a trained model for high image quality corresponding to the selected shooting mode may be selected. Further, one high image quality model obtained by learning learning data including various medical images obtained in different imaging modes may be used.

Here, in an ophthalmic apparatus, for example, an OCT apparatus, the scan pattern of the luminous flux used for the measurement and the imaging portion differ depending on the imaging mode. Therefore, for the trained model that uses the tomographic image as input data, a trained model is prepared for each shooting mode, and the trained model corresponding to the shooting mode selected according to the instruction of the operator is selected. It may be configured. In this case, the imaging mode may include, for example, a retinal imaging mode, an anterior ocular segment imaging mode, a vitreous imaging mode, a macula imaging mode, an optic disc imaging mode, an OCTA imaging mode, and the like. Further, the scan pattern may include 3D scan, radial scan, cross scan, circle scan, raster scan, Lissajous scan (scan along the Lissajous curve) and the like. In the OCTA photographing mode, the drive control unit 124 controls the scanning means described above so that the measurement light is scanned a plurality of times in the same region (same position) of the eye to be inspected. Even in the OCTA shooting mode, for example, raster scan, radial scan, cross scan, circle scan, Lissajous scan and the like can be set as the scan pattern. Further, with respect to the trained model using the tomographic image as input data, it is possible to perform training by using the tomographic image corresponding to the cross section in different directions as the training data. For example, learning may be performed using a tomographic image of a cross section in the xz direction, a tomographic image of a cross section in the yz direction, or the like as training data.

It should be noted that the operator determines whether or not it is necessary to execute the high image quality processing by the high image quality model (or to display the high image quality image obtained by the high image quality processing) with respect to the high image quality button provided on the display screen. It may be performed according to the instruction, or may be performed according to the setting stored in the storage unit 126 in advance. It should be noted that the fact that the high image quality processing using the trained model (high image quality improvement model) may be displayed in the active state of the high image quality improvement button, or that fact may be displayed on the display screen as a message. good. Further, the execution of the high image quality processing may maintain the execution state at the time of the previous examination of the ophthalmic apparatus, or may maintain the execution state at the time of the previous examination for each subject.

Further, the moving image to which various trained models such as a high image quality improvement model can be applied is not limited to the live moving image, and may be, for example, a moving image stored (stored) in the storage unit 126. At this time, for example, the moving image obtained by aligning each frame of the tomographic moving image of the fundus stored (stored) in the storage unit 126 may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous body, first, a reference frame based on a condition such as the presence of the vitreous body on the frame as much as possible may be selected. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) sequentially generated by the high-quality engine may be continuously displayed for each at least one frame of the moving image.

As the above-mentioned alignment method between frames, the same method may be applied to the alignment method in the X direction and the alignment method in the Z direction (depth direction), and all different methods may be applied. May be applied. Further, the alignment in the same direction may be performed a plurality of times by different methods, and for example, a precise alignment may be performed after performing a rough alignment. Further, as a method of alignment, for example, a plurality of alignments obtained by segmenting a tomographic image (coarse in the Z direction) using a retinal layer boundary obtained by performing segmentation processing of a tomographic image (B scan image) are performed. Alignment using the correlation information (similarity) between the region and the reference image (precise in the X and Z directions), and the one-dimensional projection image generated for each tomographic image (B scan image) was used (in the X direction). ) Alignment, alignment (in the X direction) using a two-dimensional front image, and the like. Further, it may be configured so that the alignment is roughly performed in pixel units and then the precise alignment is performed in sub-pixel units.

Further, the high image quality model may be updated by additional learning using the value of the ratio set (changed) according to the instruction from the examiner as the learning data. For example, if the examiner tends to set a high ratio of the input image to the high-quality image when the input image is relatively dark, the trained model will be additionally trained so as to have such a tendency. This allows, for example, to be customized as a trained model that can obtain a percentage of synthesis that suits the taste of the examiner. At this time, a button for deciding whether or not to use the set (changed) ratio value as the learning data for additional learning according to the instruction from the examiner may be displayed on the display screen. Further, the ratio determined by using the trained model may be set as the default value, and then the ratio value may be changed from the default value according to the instruction from the examiner. Further, the high image quality model may be a trained model obtained by additional learning of learning data including at least one high image quality image generated by using the high image quality model. At this time, whether or not to use the high-quality image as learning data for additional learning may be configured to be selectable according to an instruction from the examiner.

[Modification 8]
Further, the control unit 102 may generate a label image of the image acquired by shooting using a trained model for image segmentation, and perform image segmentation processing. Here, the label image means a label image in which a region is labeled for each pixel of the tomographic image. Specifically, it is an image in which any area is divided by a group of identifiable pixel values (hereinafter, label value) among the area groups drawn in the acquired image. Here, the specified arbitrary region includes a region of interest, a volume of interest (VOI: Volume Of Interest), and the like.

By specifying the coordinate group of the pixel having an arbitrary label value from the image, the coordinate group of the pixel that depicts the corresponding region such as the retinal layer in the image can be specified. Specifically, for example, when the label value indicating the ganglion cell layer constituting the retina is 1, the coordinate group having the pixel value of 1 among the pixel groups of the image is specified, and the coordinate group corresponds to the coordinate group from the image. Extract the pixel group to be used. This makes it possible to identify the region of the ganglion cell layer in the image.

The image segmentation process may include a process of reducing or enlarging the label image. At this time, the image complement processing method used for reducing or enlarging the label image shall use the nearest neighbor method or the like so as not to erroneously generate an undefined label value or a label value that should not exist at the corresponding coordinates. ..

The image segmentation process is a process for specifying a region called ROI (Region Of Interest) or VOI, such as an organ or a lesion depicted in an image, for use in image diagnosis or image analysis. For example, according to the image segmentation process, it is possible to identify a region group of a group of layers constituting the retina from an image acquired by OCT imaging with the back eye portion as an imaging target. If the area to be specified is not drawn in the image, the number of specified areas is 0. Further, as long as a plurality of region groups to be specified are depicted in the image, the number of the specified regions may be a plurality, or even one region surrounding the region group so as to include the region group. good.

The specified area group is output as information that can be used in other processing. Specifically, for example, the coordinate group of the pixel group constituting each of the specified area groups can be output as a numerical data group. Further, for example, a coordinate group indicating a rectangular area, an ellipsoidal area, a rectangular parallelepiped area, an ellipsoidal area, etc. including each of the specified area groups can be output as a numerical data group. Further, for example, a coordinate group indicating a straight line, a curve, a plane, a curved surface, or the like corresponding to the boundary of the specified region group can be output as a numerical data group. Further, for example, a label image showing the specified region group can be output.

Here, as a machine learning model for image segmentation, for example, a convolutional neural network (CNN) can be used. Here, with reference to FIG. 11, an example in which the machine learning model according to this modification is configured by CNN will be described. FIG. 11 shows an example of configuration 1100 of a trained model for image segmentation. In the example of the trained model, for example, when the tomographic image Im1110 is input, the label image Im1120 indicating the specified region group can be output.

The machine learning model shown in FIG. 11 is composed of a plurality of layers responsible for processing and outputting an input value group. The types of layers included in the configuration 1100 of the machine learning model include a convolution layer, a Downsampling layer, an Upsampling layer, and a Merger layer.

The convolution layer is a layer that performs convolution processing on the input value group according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image.

The downsampling layer is a layer that performs processing to reduce the number of output value groups to be smaller than the number of input value groups by thinning out or synthesizing input value groups. Specifically, as such a process, for example, there is a Max Polling process.

The upsampling layer is a layer that performs processing to increase the number of output value groups to be larger than the number of input value groups by duplicating the input value group or adding interpolated values from the input value group. Specifically, as such a process, for example, there is a linear interpolation process.

The composite layer is a layer in which a value group such as an output value group of a certain layer or a pixel value group constituting an image is input from a plurality of sources, and the processing is performed by concatenating or adding them.

It should be noted that if the parameter settings for the layers and nodes that make up the neural network are different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may differ. That is, in many cases, the appropriate parameters differ depending on the embodiment, and therefore, the values can be changed to preferable values as needed.

In addition to the method of changing the parameters as described above, there are cases where the CNN can obtain better characteristics by changing the configuration of the CNN. Better characteristics include, for example, more accurate alignment position information output, shorter processing time, shorter training time for machine learning models, and the like.

The CNN configuration 1100 used in this modification has a function of an encoder composed of a plurality of layers including a plurality of downsampling layers and a function of a decoder composed of a plurality of layers including a plurality of upsampling layers. It is a net type machine learning model. In the U-net type machine learning model, the position information (spatial information) obscured in a plurality of layers configured as an encoder is transferred to the same-dimensional layer (layers corresponding to each other) in a plurality of layers configured as a decoder. ) (For example, using a skip connection).

Although not shown, as an example of changing the configuration of the CNN, for example, a batch normalization layer or an activation layer using a normalized linear function (Rectifier Liner Unit) may be incorporated after the convolution layer. good. Through these steps of CNN, the features of the captured image can be extracted.

As the machine learning model according to this modification, for example, CNN (U-net type machine learning model) as shown in FIG. 11, a model combining CNN and LSTM, FCN (Full Convolutional Network), or SegNet and the like can be used. Further, a machine learning model or the like that performs object recognition can also be used according to a desired configuration. As a machine learning model for performing object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, a machine learning model that recognizes an object in units of regions can also be used. As a machine learning model that recognizes an object in a region unit, YOLO (You Only Look None) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used.

Further, as the training data of the machine learning model for image segmentation, the tomographic image acquired by OCT is used as input data, and the label image in which the area is labeled for each pixel of the tomographic image is used as output data. Label images include, for example, internal limiting membrane (ILM), nerve fiber layer (NFL), ganglion cell layer (GCL), photoreceptor inner segment outer segment junction (ISOS), retinal pigment epithelial layer (RPE), Bruch. Labeled images with labels such as membrane (BM) and choroid can be used. Other regions include, for example, the vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner plexiform layer (IPL), inner nuclear layer (INL), corneal membrane, anterior chamber, iris, and the like. And an image with a label such as a crystalline lens may be used.

Further, the input data of the machine learning model for image segmentation is not limited to the tomographic image. It may be an anterior eye image, an SLO image, an OCTA image, or the like. In this case, the training data can be input data of various images and output data of a label image in which a region name or the like is labeled for each pixel of the various images. For example, when the input data of the training data is an SLO image, the output data may be an image labeled with a peripheral portion of the optic disc, Disc, Cup, or the like.

The label image used as the output data may be an image in which each region is labeled in the tomographic image by a doctor or the like, or an image in which each region is labeled by the rule-based region detection process. There may be. However, if machine learning is performed using a label image that is not properly labeled as the output data of the training data, the image obtained by using the trained model trained using the training data is also properly labeled. There is a possibility that the label image will not be used. Therefore, by removing the pair containing such a label image from the training data, it is possible to reduce the possibility that an inappropriate label image is generated by using the trained model. Here, the rule-based region detection process refers to a detection process that utilizes a known regularity such as the regularity of the shape of the retina.

By performing image segmentation processing using such a trained model for image segmentation, the control unit 102 can be expected to detect a specific region of various images at high speed and with high accuracy. A trained model for image segmentation may also be prepared for each type of various images as input data. Further, for the OCTA front image and the En-Face image, a trained model may be prepared for each depth range for generating an image. Further, the trained model for image segmentation may also be one in which learning is performed on an image for each imaging site (for example, the center of the macula and the center of the optic nerve head), or the learning is performed regardless of the imaging site. You may.

Further, for the trained model for image segmentation, additional learning may be performed using the data manually modified according to the instruction of the operator as training data. Further, the determination of the necessity of additional learning and the determination of whether or not to transmit the data to the server may be performed by the same method. In these cases as well, it can be expected that the accuracy of each process can be improved and the process can be performed according to the tendency of the examiner's preference.

Further, when the control unit 102 detects a partial region of the eye E to be inspected (for example, a region of interest, an artifact region, an abnormal region, etc.) using the trained model, the control unit 102 performs predetermined image processing for each detected partial region. Can also be applied. As an example, the case of detecting at least two partial regions of the vitreous region, the retinal region, and the choroid region will be described. In this case, when performing image processing such as contrast adjustment on at least two detected partial regions, adjustments suitable for each region can be performed by using different image processing parameters. By displaying the image adjusted suitable for each area, the operator can more appropriately diagnose the disease or the like in each partial area. The configuration using different image processing parameters for each detected partial region is similarly applied to the partial region of the eye E to be inspected obtained by detecting the partial region of the eye E to be inspected without using the trained model. You may.

[Modification 9]
The display control unit 125 in the various embodiments and modifications described above may display analysis results such as a layer thickness of a desired layer and various blood vessel densities on a report screen of a display screen after taking a tomographic image. In addition, the optic disc, macular region, vascular region, capillary region, arterial region, venous region, nerve fiber bundle, vitreous region, macula region, choroid region, scleral region, lamina cribrosa region, retinal layer boundary, retina. Parameter values (distribution) for the site of interest including at least one of the layer boundary edge, photoreceptor, blood cells, blood vessel wall, blood vessel inner wall boundary, blood vessel lateral border, ganglion cell, corneal region, macula region, Schlemm's canal, etc. May be displayed as the analysis result. Here, the site of interest may be, for example, a vortex vein or the like, which is an outlet of a blood vessel in the Haller layer (an example of a blood vessel in a depth range of a part of the choroidal region) to the outside of the eye. At this time, the parameters related to the site of interest include, for example, the number of vortex veins (for example, the number for each region), the distance from the optic disc to each vortex vein, the angle at which each vortex vein centered on the optic nerve head is located, and the like. May be. This makes it possible to accurately diagnose, for example, various diseases related to Pachychoroid (thickened choroid) (for example, choroidal neovascularization). Further, for example, by analyzing a medical image to which various artifact reduction processes are applied, the above-mentioned various analysis results can be displayed as accurate analysis results. The artifact is, for example, a false image region generated by light absorption by a blood vessel region or the like, a projection artifact, a band-shaped artifact in a front image generated in the main scanning direction of the measured light depending on the state of the eye to be inspected (movement, blinking, etc.), or the like. There may be. Further, the artifact may be, for example, any image loss region that randomly occurs at each shooting on the medical image of the predetermined portion of the subject. Further, the display control unit 125 may display the value (distribution) of the parameter relating to the region including at least one of various artifacts (photo-loss regions) as described above on the display unit 103 as an analysis result. Further, the value (distribution) of the parameter relating to the region including at least one such as drusen, new blood vessel, vitiligo (hard vitiligo), and abnormal site such as pseudo-drusen may be displayed as an analysis result. Further, the comparison result obtained by comparing the standard value or standard range obtained by using the standard database with the analysis result may be displayed.

Further, the analysis result may be displayed in an analysis map, a sector showing statistical values corresponding to each divided area, or the like. The analysis result may be generated by using a trained model (analysis result generation engine, trained model for analysis result generation) obtained by learning the analysis result of the medical image as training data. .. At this time, the trained model is trained using training data including a medical image and an analysis result of the medical image, and training data including a medical image and an analysis result of a medical image of a type different from the medical image. It may be obtained by.

Further, the training data for performing image analysis may include a label image generated by using the trained model for image segmentation processing and an analysis result of a medical image using the label image. In this case, the control unit 102 can function as an example of an analysis result generation unit that generates an analysis result of a tomographic image from the result of image segmentation processing by using, for example, a trained model for generating an analysis result. .. Further, the trained model is a training data including input data including a set of a plurality of medical images of different types of predetermined parts, such as an En-Face image and a motion contrast front image (OCTA En-Face image) described later. It may be obtained by learning using.

Further, the analysis result obtained by using the high-quality image generated by using the high-quality model may be displayed. In this case, the input data included in the training data may be a high-quality image generated by using a trained model for high image quality, or may be a set of a low-quality image and a high-quality image. May be good. The training data may be an image in which at least a part of an image whose image quality has been improved by using the trained model has been manually or automatically modified.

Further, the training data includes, for example, at least the analysis value (for example, average value, median value, etc.) obtained by analyzing the analysis area, the table including the analysis value, the analysis map, the position of the analysis area such as the sector in the image, and the like. The information including one may be the data labeled (annotated) with the input data as the correct answer data (for supervised learning). It should be noted that the analysis result obtained by using the trained model for generating the analysis result may be displayed according to the instruction from the operator.

Further, the estimation unit 123 in the above-described embodiment and modification can output an accurate estimation result by using, for example, an image to which various artifact reduction processes as described above are applied for the estimation process. can. Further, the display control unit 125 may display the estimation result on the image at the position of the specified abnormal portion or the like, or may display the state of the abnormal portion or the like by characters or the like. Further, the display control unit 125 may display a classification result (for example, Cartin classification) of an abnormal site or the like as a diagnosis result separately from the estimation result for the disease. Further, as the classification result, for example, information indicating the certainty of each abnormal part (for example, a numerical value indicating a ratio) may be displayed. In addition, information necessary for the doctor to confirm the diagnosis may be displayed as a diagnosis result. As the necessary information, for example, advice such as additional shooting can be considered. For example, when an abnormal site is detected in the blood vessel region in the OCTA image, it may be displayed that fluorescence imaging using a contrast medium capable of observing the blood vessel in more detail than OCTA is performed. In addition, the diagnosis result may be information on the future medical treatment policy of the subject. In addition, the diagnosis result is, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Medical support information, etc.) and grounds for denying the diagnosis name (negative medical support information, etc.) may be included in the information. At this time, for example, a diagnosis result that is more probable than the diagnosis result such as a diagnosis name input in response to an instruction from the examiner may be displayed as medical support information. Further, when a plurality of types of medical images are used, for example, the types of medical images that can be the basis of the diagnosis result may be displayed in an identifiable manner. The basis of the diagnosis result is a map (attention map, activation map) that visualizes the features extracted by the trained model, for example, a color map (heat map) that shows the features in color. good. At this time, for example, the heat map may be superimposed and displayed on the medical image used as the input data. The heat map is, for example, a method for visualizing a region (a region having a large gradient) that contributes greatly to the output value of the predicted (estimated) class, such as Grad-CAM (Gradient-weighted Class Activity Mapping) or Guided Grade. -Can be obtained using CAM or the like.

The diagnosis result may be generated by using a trained model (diagnosis result generation engine, trained model for generation of diagnosis result) obtained by learning the diagnosis result of the medical image as training data. .. Further, the trained model is based on training using training data including a medical image and a diagnosis result of the medical image, and training data including a medical image and a diagnosis result of a medical image of a type different from the medical image. It may be obtained.

Further, the training data may include a label image generated by using the trained model for image segmentation processing and a diagnostic result of a medical image using the label image. In this case, the control unit 102 can function as an example of a diagnosis result generation unit that generates a diagnosis result of a tomographic image from the result of image segmentation processing by using, for example, a trained model for generating a diagnosis result. ..

Further, it may be configured to display the diagnostic result obtained by using the high-quality image generated by using the trained model for high image quality. In this case, the input data included in the training data may be a high-quality image generated by using a trained model for high image quality, or may be a set of a low-quality image and a high-quality image. May be good. The training data may be an image in which at least a part of an image whose image quality has been improved by using the trained model has been manually or automatically modified.

In addition, the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Information including at least one such as (general medical support information, etc.) and grounds for denying the diagnosis name (negative medical support information), etc. are labeled (annotated) as correct answer data (for supervised learning). Data may be used. In addition, in response to an instruction from the examiner, the diagnosis result obtained by using the trained model for generating the diagnosis result may be displayed.

A trained model may be prepared for each information used as input data or for each type of information, and the diagnosis result may be acquired using the trained model. In this case, the information output from each trained model may be statistically processed to determine the final diagnostic result. For example, the ratio of the information output from each trained model may be added for each type of information, and the information having a higher total ratio than the other information may be determined as the final diagnosis result. Note that the statistical processing is not limited to the calculation of the total, and may be the calculation of the average value or the median value. Further, for example, among the information output from each trained model, the information having a higher ratio than the other information (the information having the highest ratio) may be used to determine the diagnosis result. Similarly, the diagnosis result may be determined using the information of the ratio of the information output from each trained model that is equal to or higher than the threshold value.

Further, it may be configured so that the quality of the determined diagnosis result can be determined (approved) according to the instruction (selection) of the operator. Further, the diagnosis result may be determined from the information output from each trained model according to the instruction (selection) of the operator. At this time, for example, the display control unit 125 may display the information output from each trained model and the ratio thereof side by side on the display unit 103. Then, the operator may be configured to determine the selected information as a diagnosis result, for example, by selecting information having a higher ratio than other information. Further, the diagnosis result may be determined by using the machine learning model from the information output from each trained model. In this case, the machine learning algorithm may be a different type of machine learning algorithm from the machine learning algorithm used to generate the diagnostic result, for example, a neural network, a support vector machine, an adaboost, a Bayesian network, or a random. A forest or the like may be used.

The learning of the various trained models described above may be not only supervised learning (learning with labeled learning data) but also semi-supervised learning. In semi-supervised learning, for example, after multiple classifiers (classifiers) perform supervised learning, unlabeled learning data is identified (classified), and the reliability of the identification result (classification result) is determined. It is a method of automatically labeling (annotating) (for example, an identification result whose certainty is equal to or higher than the threshold value) and learning with the labeled learning data. Semi-supervised learning may be, for example, co-training (or Multiview). At this time, the trained model for generating the diagnosis result uses, for example, a first discriminator that identifies a medical image of a normal subject and a second discriminator that identifies a medical image containing a specific lesion. It may be a trained model obtained by semi-supervised learning (for example, co-training). It should be noted that the purpose is not limited to diagnostic purposes, but may be, for example, photography support or the like. In this case, the second classifier may, for example, identify a medical image including a partial region such as a region of interest or an artifact region.

Further, the display control unit 125 according to the various embodiments and modifications described above has an object recognition result (object detection) of a partial region such as a region of interest, an artifact region, and an abnormal region as described above on the report screen of the display screen. The result) and the segmentation result may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, colors and the like may be superimposed and displayed on an object in an image. The object recognition result and segmentation result are trained models (object recognition engine, for object recognition) obtained by learning learning data that labels (annotifies) medical images with information indicating object recognition and segmentation as correct answer data. It may be generated using a trained model, a segmentation engine, a trained model for segmentation). The above-mentioned analysis result generation and diagnosis result generation may be obtained by using the above-mentioned object recognition result and segmentation result. For example, analysis result generation or diagnosis result generation processing may be performed on a region of interest obtained by object recognition or segmentation processing.

Further, when detecting an abnormal portion, the control unit 102 may use a hostile generation network (GAN: Generative Adversarial Networks) or a variational auto-encoder (VAE: Variational Auto-Encoder). For example, a DCGAN (Deep Convolutional GAN) consisting of a generator obtained by learning the generation of a medical image and a classifier obtained by learning the discrimination between a new medical image generated by the generator and a real medical image. Can be used as a machine learning model.

When DCGAN is used, for example, the classifier encodes the input medical image into a latent variable, and the generator generates a new medical image based on the latent variable. After that, the difference between the input medical image and the generated new medical image can be extracted (detected) as an abnormal part. When VAE is used, for example, the input medical image is encoded by an encoder to be a latent variable, and the latent variable is decoded by a decoder to generate a new medical image. After that, the difference between the input medical image and the generated new medical image can be extracted as an abnormal part.

Further, the control unit 102 may detect an abnormal portion by using a convolutional autoencoder (CAE). When CAE is used, the same medical image is learned as input data and output data at the time of learning. As a result, when a medical image having an abnormal part is input to CAE at the time of estimation, a medical image having no abnormal part is output according to a learning tendency. After that, the difference between the medical image input to the CAE and the medical image output from the CAE can be extracted as an abnormal portion.

In these cases, the control unit 102 uses information on the difference between the medical image obtained by using the hostile generation network or the autoencoder and the medical image input to the hostile generation network or the autoencoder as information on the abnormal portion. Can be generated. As a result, the control unit 102 can be expected to detect the abnormal portion at high speed and with high accuracy. For example, even if it is difficult to collect many medical images including abnormal parts as learning data in order to improve the detection accuracy of abnormal parts, a relatively large number of normal medical images of subjects that are easy to collect are used as learning data. Can be used. Therefore, for example, learning for accurately detecting an abnormal portion can be performed efficiently. Here, the autoencoder includes VAE, CAE, and the like. Further, at least a part of the generation part of the hostile generation network may be composed of VAE. As a result, for example, it is possible to generate a relatively clear image while reducing the phenomenon of generating similar data. For example, the control unit 102 provides information on the difference between a medical image obtained from various medical images using a hostile generation network or an autoencoder and a medical image input to the hostile generation network or the autoencoder. It can be generated as information about the abnormal part. Further, for example, the display control unit 125 relates to a difference between a medical image obtained from various medical images using a hostile generation network or an auto encoder and a medical image input to the hostile generation network or the auto encoder. The information can be displayed on the display unit 103 as information on the abnormal portion.

Further, in particular, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a set of a plurality of medical images of different types of the predetermined part of the subject. good. At this time, as the input data included in the training data, for example, input data in which a motion contrast front image and a luminance front image (or a luminance tomographic image) of the fundus are set can be considered. Further, as the input data included in the training data, for example, input data in which a tomographic image (B scan image) of the fundus and a color fundus image (or a fluorescent fundus image) are set can be considered. Further, the plurality of medical images of different types may be anything as long as they are acquired by different modality, different optical systems, different principles, or the like.

Further, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a plurality of medical images of different parts of the subject. At this time, as the input data included in the training data, for example, input data in which a tomographic image of the fundus (B scan image) and a tomographic image of the anterior segment of the eye (B scan image) are considered as a set can be considered. Further, as the input data included in the training data, for example, input data in which a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic nerve head of the fundus are set as a set, etc. Is also possible.

The input data included in the learning data may be different parts of the subject and a plurality of different types of medical images. At this time, the input data included in the training data may be, for example, input data in which a tomographic image of the anterior segment of the eye and a color fundus image are set. Further, the trained model described above may be a trained model obtained by learning from training data including input data including a set of a plurality of medical images having different shooting angles of view of a predetermined portion of the subject. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a predetermined portion into a plurality of regions, such as a panoramic image. At this time, by using a wide angle of view image such as a panoramic image as training data, there is a possibility that the feature amount of the image can be accurately acquired because the amount of information is larger than that of the narrow angle of view image. The result of can be improved. Further, the input data included in the learning data may be input data in which a plurality of medical images of different dates and times of a predetermined part of the subject are set.

Further, the display screen on which at least one of the above-mentioned estimation result, analysis result, diagnosis result, object recognition result, and segmentation result is displayed is not limited to the report screen. Such a display screen is, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). It may be displayed in. For example, by displaying at least one result obtained by using the above-mentioned trained model on the shooting confirmation screen, the operator can confirm the accurate result even immediately after shooting.

Further, for example, when a specific object is recognized, a frame surrounding the recognized object may be configured to be superimposed and displayed on the live moving image. At this time, if the information indicating the certainty of the object recognition result (for example, a numerical value indicating the ratio) exceeds the threshold value, the color of the frame surrounding the object may be highlighted, for example. good. This allows the examiner to easily identify the object on the live video.

In addition, in the generation of the correct answer data used for learning the various trained models described above, the trained model for generating the correct answer data for generating the correct answer data such as labeling (annotation) may be used. At this time, the trained model for generating correct answer data may be obtained by (sequentially) additional learning of the correct answer data obtained by labeling (annotation) by the examiner. That is, the trained model for generating correct answer data may be obtained by additional training of training data in which the data before labeling is used as input data and the data after labeling is used as output data. Further, in a plurality of continuous frames such as a moving image, the result of the frame judged to have low accuracy is corrected in consideration of the results of object recognition and segmentation of the preceding and following frames. May be good. At this time, according to the instruction from the examiner, the corrected result may be additionally learned as correct answer data. Further, for example, for a medical image with low accuracy of the result, the feature amount is an example of a map (attention map, activation map) in which the examiner visualizes the feature amount extracted by the trained model on the medical image. It may be configured to additionally learn an image labeled (annotated) as input data while checking a color map (heat map) showing the above in color. For example, in a heat map on a layer just before outputting the result in the trained model, if the part to be noted is different from the intention of the examiner, the part to be noted by the examiner is labeled (annotation). You may additionally learn the medical image. Thereby, for example, the trained model is a partial region on the medical image, and the feature amount of the partial region having a relatively large influence on the output result of the trained model is prioritized over the other regions ( Additional learning can be done (with weighting).

Here, the various trained models described above can be obtained by machine learning using the training data. Machine learning includes, for example, deep learning consisting of a multi-layered neural network. Further, for example, a convolutional neural network can be used for at least a part of the multi-layered neural network. Further, a technique related to an autoencoder (self-encoder) may be used for at least a part of a multi-layered neural network. Further, a technique related to backpropagation (error backpropagation method) may be used for learning. Further, for learning, a method (dropout) in which each unit (each neuron or each node) is randomly inactivated may be used. Further, for learning, a method (batch normalization) may be used in which the data transmitted to each layer of the multi-layer neural network is normalized before the activation function (for example, the ReLu function) is applied. However, the machine learning is not limited to deep learning, and any learning using a model capable of extracting (expressing) the features of learning data such as images by learning may be used. Here, the machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. The trained model is a model in which a machine learning model by an arbitrary machine learning algorithm is trained (learned) in advance using appropriate learning data. However, the trained model does not require further learning, and additional learning can be performed. The learning data is composed of a pair of input data and output data (correct answer data). Here, the learning data may be referred to as teacher data, or the correct answer data may be referred to as teacher data.

The GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed a plurality of times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in this modification, the GPU is used in addition to the CPU for the processing by the control unit 102, which is an example of the learning unit (not shown). Specifically, when a learning program including a learning model is executed, learning is performed by the CPU and the GPU collaborating to perform calculations. In addition, the processing of the learning unit may be performed only by the CPU or the GPU. Further, the processing unit (estimation unit) that executes the processing using the various trained models described above may also use the GPU in the same manner as the learning unit. Further, the learning unit may include an error detection unit and an update unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error back propagation method is a method of adjusting the coupling weighting coefficient and the like between the nodes of each neural network so that the above error becomes small.

Further, as the machine learning model used for the above-mentioned object recognition, segmentation, high image quality, etc., the function of the encoder consisting of a plurality of layers including a plurality of downsampling layers and a plurality of layers including a plurality of upsampling layers A U-net type machine learning model having a decoder function consisting of is applicable. In the U-net type machine learning model, the position information (spatial information) obscured in a plurality of layers configured as an encoder is transferred to the same-dimensional layer (layers corresponding to each other) in a plurality of layers configured as a decoder. ) (For example, using a skip connection).

Further, as the machine learning model used for the above-mentioned object recognition, segmentation, high image quality, etc., for example, FCN (Full Convolutional Network), SegNet, or the like can be used. Further, a machine learning model that recognizes an object in a region unit according to a desired configuration may be used. As a machine learning model for performing object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Further, as a machine learning model for recognizing an object in a region unit, YOLO (You Only Look None) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used.

Further, the machine learning model may be, for example, a capsule network (Capsule Network; CapsNet). Here, in a general neural network, each unit (each neuron or each node) is configured to output a scalar value, for example, a spatial positional relationship (relative position) between features in an image. It is configured to reduce spatial information about. Thereby, for example, learning can be performed so as to reduce the influence of local distortion, translation, and the like of the image. On the other hand, in the capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is retained. This makes it possible to perform learning in which, for example, the spatial positional relationship between features in an image is taken into consideration.

[Modification 10]
In the preview screens in the various embodiments and modifications described above, the various trained models described above may be configured to be used for at least one frame of the live moving image. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the trained model corresponding to each live moving image may be used. As a result, for example, even if it is a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of shooting. Therefore, for example, it is possible to reduce the failure of re-imaging, and thus it is possible to improve the accuracy and efficiency of diagnosis.

The plurality of live moving images may be, for example, a moving image of the anterior eye portion for alignment in the XYZ direction, and a frontal moving image of the fundus for focusing adjustment or OCT focus adjustment of the fundus observation optical system. Further, the plurality of live moving images may be, for example, a tomographic moving image of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measured optical path length and the reference optical path length). When such a preview image is displayed, the above-mentioned various adjustments are performed so that the region detected by using the above-mentioned trained model for object recognition and the above-mentioned trained model for segmentation satisfies a predetermined condition. The control unit 102 may be configured as described above. For example, a value (for example, a contrast value or an intensity value) related to a predetermined retinal layer such as a vitreous region or an RPE detected by using a trained model for object recognition or a trained model for segmentation exceeds a threshold value (for example, a contrast value or an intensity value). Alternatively, it may be configured to perform various adjustments such as OCT focus adjustment so as to reach a peak value). Further, for example, the OCT so that a predetermined retinal layer such as a vitreous region or RPE detected by using a trained model for object recognition or a trained model for segmentation is at a predetermined position in the depth direction. Coherence gate adjustments may be configured to be made.

In these cases, the control unit 102 can generate a high-quality moving image by performing high-quality processing on the moving image using the trained model. Further, the drive control unit 124 has a shooting range of a reference mirror or the like so that a partial area such as a region of interest obtained by segmentation processing or the like is at a predetermined position in the display area while a high-quality moving image is displayed. It is possible to drive and control the optical member for changing the. In such a case, the drive control unit 124 can automatically perform the alignment process so that the desired area becomes a predetermined position in the display area based on the highly accurate information. The optical member that changes the photographing range may be, for example, an optical member that adjusts the coherence gate position, and specifically, a reference mirror that reflects reference light. Further, the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measured optical path length and the reference optical path length, and the optical member changes, for example, the optical path length of the measurement light (not shown). It may be a mirror or the like. The optical member for changing the photographing range may be, for example, a stage portion (not shown). Further, in response to the instruction regarding the start of shooting by the drive control unit 124, the scanning means is provided so that the partial region such as the artifact region obtained by the segmentation process or the like is photographed (rescanned) again during or at the end of the imaging. Drive control may be used. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, various adjustments, shooting start, etc. may be automatically performed. good. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, the start of imaging, etc. can be executed according to the instruction from the examiner. It may be configured to change to a new state (release the execution prohibited state).

Further, the moving image to which the various trained models described above can be applied is not limited to the live moving image, and may be, for example, a moving image stored (stored) in the storage unit 126. At this time, for example, the moving image obtained by aligning each frame of the tomographic moving image of the fundus stored (stored) in the storage unit 126 may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous body, first, a reference frame based on a condition such as the presence of the vitreous body on the frame as much as possible may be selected. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) sequentially generated by the trained model for high image quality may be continuously displayed for each at least one frame of the moving image.

As the above-mentioned alignment method between frames, the same method may be applied to the alignment method in the X direction and the alignment method in the Z direction (depth direction), or all different methods may be applied. May be applied. Further, the alignment in the same direction may be performed a plurality of times by different methods, and for example, a precise alignment may be performed after performing a rough alignment. Further, as a method of alignment, for example, a plurality of alignments obtained by segmenting a tomographic image (coarse in the Z direction) using a retinal layer boundary obtained by performing segmentation processing of a tomographic image (B scan image) are performed. Alignment using the correlation information (similarity) between the region and the reference image (precise in the X and Z directions), and the one-dimensional projection image generated for each tomographic image (B scan image) was used (in the X direction). ) Alignment, alignment (in the X direction) using a two-dimensional front image, and the like. Further, it may be configured so that the alignment is roughly performed in pixel units and then the precise alignment is performed in sub-pixel units.

Here, during various adjustments, it is possible that the imaged object such as the retina of the eye to be inspected has not yet been successfully imaged. Therefore, since there is a large difference between the medical image input to the trained model and the medical image used as the training data, there is a possibility that a high-quality image cannot be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the display of the high-quality moving image (continuous display of high-quality frames) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the image quality enhancement button may be configured to be changed to a state (active state) that can be specified by the examiner.

Further, for example, a trained model for high image quality is prepared for each shooting mode having a different scan pattern, and the trained model for high image quality corresponding to the selected shooting mode is selected. May be done. Further, one trained model for high image quality obtained by learning training data including various medical images obtained in different imaging modes may be used.

[Modification 11]
In the above-described embodiment and modification, when various trained models are executing additional learning, it may be difficult to output (infer / predict) using the trained model itself during additional learning. be. Therefore, it may be configured to prohibit the input of medical images other than the training data to the trained model during the execution of additional learning. Further, another trained model that is the same as the trained model before the execution of the additional learning may be prepared as another preliminary trained model. At this time, during the execution of the additional learning, it may be configured so that the input of the medical image other than the training data to the preliminary trained model can be executed. Then, after the additional learning is completed, the trained model after the additional learning is executed may be evaluated, and if there is no problem, the preliminary trained model may be replaced with the trained model after the additional learning is executed. Also, if there is a problem, a preliminary trained model may be used.

As an evaluation of the trained model after the execution of additional learning, for example, a trained model for classification for classifying a high-quality image obtained by the trained model for high image quality from another type of image is used. It may be used. The trained model for classification uses, for example, a plurality of images including a high-quality image and a low-quality image obtained by the trained model for high image quality as input data, and the types of these images are labeled (annotation). It may be a trained model obtained by training training data including the obtained data as correct answer data. At this time, the image type of the input data at the time of estimation (prediction) is displayed together with the information indicating the certainty of each type of image included in the correct answer data at the time of learning (for example, a numerical value indicating a ratio). May be good. In addition to the above images, the input data of the trained model for classification includes overlay processing of a plurality of low-quality images (for example, averaging processing of a plurality of low-quality images obtained by alignment) and the like. May include high-quality images such as those with high contrast and noise reduction. Further, as the evaluation of the trained model after the execution of the additional learning, for example, the trained model after the execution of the additional learning and the trained model before the execution of the additional learning (preliminary trained model) are used and the same. A plurality of high-quality images obtained from the above images may be compared, or the analysis results of the plurality of high-quality images may be compared. At this time, for example, the comparison result of the plurality of high-quality images (an example of change due to additional learning) or the comparison result of the analysis result of the plurality of high-quality images (an example of change due to additional learning) is within a predetermined range. It may be determined whether or not, and the determination result may be displayed.

In addition, the trained model obtained by learning for each imaging site may be selectively used. Specifically, the first trained model obtained by using the learning data including the first imaged portion (for example, the anterior eye portion, the posterior eye portion, etc.) and the second imaged portion different from the first imaging region. It is possible to prepare a second trained model obtained by using the training data including the imaged portion, and a plurality of trained models including the trained model. Then, the control unit 102 may have a selection means for selecting one of the plurality of trained models. At this time, the control unit 102 may have a control means for executing additional learning on the selected trained model. The control means searches for data in which the captured part corresponding to the selected trained model and the captured image of the captured portion are paired in response to an instruction from the examiner, and the data obtained by the search is learned data. Can be executed as additional training for the selected trained model. The imaged portion corresponding to the selected trained model may be acquired from the information in the header of the data or manually input by the examiner. Further, the data can be searched for, for example, from a server of an external facility such as a hospital or a research institute via a network. As a result, additional learning can be efficiently performed for each imaged part by using the photographed image of the imaged part corresponding to the trained model.

The selection means and the control means may be composed of a software module executed by a processor such as a CPU or an MPU of the control unit 102. Further, the selection means and the control means may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like.

In addition, when learning data for additional learning is acquired from a server of an external facility such as a hospital or research institute via a network, it is necessary to reduce reliability deterioration due to falsification or system trouble during additional learning. Is useful. Therefore, the legitimacy of the learning data for additional learning may be detected by confirming the consistency by digital signature or hashing. This makes it possible to protect the learning data for additional learning. At this time, if the validity of the training data for additional learning cannot be detected as a result of confirming the consistency by digital signature or hashing, a warning to that effect is given and additional learning is performed using the training data. Make it not exist. The server may be in any form, for example, a cloud server, a fog server, an edge server, or the like, regardless of its installation location.

Further, the data protection by confirming the consistency as described above can be applied not only to the learning data for additional learning but also to the data including the medical image. Further, the image management system may be configured so that the transaction of data including medical images between the servers of a plurality of facilities is managed by a decentralized network. Further, the image management system may be configured to connect a plurality of blocks in which the transaction history and the hash value of the previous block are recorded together in chronological order. As a technique for confirming consistency, even if cryptography that is difficult to calculate even using a quantum computer such as a quantum gate method (for example, lattice cryptography, quantum cryptography by quantum key distribution, etc.) is used. good. Here, the image management system may be a device and a system that receives and stores an image taken by a photographing device or an image processed image. In addition, the image management system may transmit an image in response to a request from the connected device, perform image processing on the saved image, or request another device to perform image processing. can. The image management system can include, for example, an image storage communication system (PACS). In addition, the image management system includes a database that can store various information such as information on the subject and shooting time associated with the received image. In addition, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with saved images in response to requests from other devices. ..

When additional learning is performed on various trained models, GPU can be used to perform high-speed processing. Since the GPU can perform efficient calculations by processing more data in parallel, it is possible to perform processing on the GPU when learning is performed multiple times using a learning model such as deep learning. It is valid. The GPU, CPU, and the like may cooperate in the additional learning process.

[Modification 12]
In the various embodiments and modifications described above, the instruction from the examiner may be an instruction by voice or the like in addition to the instruction by manual operation (for example, the instruction using the user interface or the like). At this time, for example, a machine learning model including a voice recognition model (speech recognition engine, trained model for voice recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by character input or the like using a keyboard, a touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (character recognition engine, trained model for character recognition) obtained by machine learning may be used. Further, the instruction from the examiner may be an instruction by a gesture or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, trained model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be the line-of-sight detection result of the examiner on the display screen of the display unit 103. The line-of-sight detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from the periphery of the display screen on the display unit 103. At this time, the object recognition engine as described above may be used for the pupil detection from the moving image. Further, the instruction from the examiner may be an instruction by an electroencephalogram, a weak electric signal flowing through the body, or the like.

In such a case, for example, as the training data, character data or voice data (waveform data) indicating an instruction for displaying the result by processing of various trained models as described above is used as input data, and various trained data have been trained. It may be learning data in which the execution instruction for actually displaying the result or the like obtained by the processing of the model on the display unit 103 is the correct answer data. Further, the learning data may be, for example, learning data in which an execution command for whether or not to automatically set shooting parameters and an execution command for changing the button for the command to the active state are correct data. good. The learning data may be any data as long as the instruction content and the execution command content indicated by the character data, the voice data, or the like correspond to each other. Further, the voice data may be converted into character data by using an acoustic model, a language model, or the like. Further, the waveform data obtained by the plurality of microphones may be used to perform a process of reducing the noise data superimposed on the voice data. Further, the instruction by characters or voice and the instruction by a mouse, a touch panel or the like may be configured to be selectable according to the instruction from the examiner. Further, the on / off of the instruction by characters or voice may be selectably configured according to the instruction from the examiner.

Here, machine learning includes deep learning as described above, and RNN can be used, for example, for at least a part of a multi-layer neural network. Further, LSTM or QRNN may be used. Further, the machine learning model is not limited to the neural network, and boosting, a support vector machine, or the like may be used. Further, when the instruction from the examiner is input by characters, voice, or the like, a technique related to natural language processing (for example, Sequence to Sequence) may be applied. At this time, as a technique related to natural language processing, for example, a model that is output for each input sentence may be applied. Further, the various trained models described above are not limited to the instructions from the examiner, and may be applied to the output to the examiner. Further, a dialogue engine (dialogue model, trained model for dialogue) that responds to the examiner with output by characters, voice, or the like may be applied.

Further, as a technique related to natural language processing, a trained model obtained by pre-learning document data by unsupervised learning may be used. Further, as a technique related to natural language processing, a trained model obtained by transfer learning (or fine tuning) of a trained model obtained by pre-learning may be used. Further, as a technique related to natural language processing, for example, BERT (Bidirectional Encoder Representations from Transfermers) may be applied. Further, as a technique related to natural language processing, a model capable of extracting (expressing) a context (feature amount) by itself by predicting a specific word in a sentence from both the left and right contexts may be applied. Further, as a technique related to natural language processing, a model capable of determining the relationship (continuity) between two sequences (sentences) in the input time series data may be applied. Further, as a technique related to natural language processing, a Transformer Encoder is used for the hidden layer, and a model in which a sequence of vectors is input and output may be applied.

Here, the instruction from the examiner to which this modification is applicable is for changing the display of various images and analysis results as described in the various embodiments and modifications described above, and for generating an En-Face image. Selection of depth range, selection of whether to use as training data for additional learning, selection of trained model, output (display, transmission, etc.) and storage of results obtained using various trained models, etc. Any instruction may be used as long as it is at least one instruction. Further, the instruction from the examiner to which this modification is applicable may be an instruction before shooting as well as an instruction after shooting, for example, an instruction regarding various adjustments and an instruction regarding setting of various shooting conditions. , It may be an instruction regarding the start of shooting. Further, the instruction from the examiner to which this modification is applicable may be an instruction regarding a change (screen transition) of the display screen.

The machine learning model may be a machine learning model that combines a machine learning model related to images such as CNN and a machine learning model related to time series data such as RNN. In such a machine learning model, for example, it is possible to learn the relationship between the feature amount related to an image and the feature amount related to time series data. When the input layer side of the machine learning model is CNN and the output layer side is RNN, for example, the medical image is used as input data, and the text related to the medical image (for example, the presence or absence of a lesion, the type of lesion, and the recommendation of the next examination). Etc.) may be used as output data for training. As a result, for example, medical information related to medical images is automatically explained in sentences, so that even an examiner with little medical experience can easily grasp medical information related to medical images. When the input layer side of the machine learning model is RNN and the output layer side is CNN, for example, texts related to medical treatment such as lesions, findings, and diagnoses are used as input data, and medical images corresponding to the texts related to the medical treatment are output. Learning may be performed using the training data as data. This makes it possible, for example, to easily search for a medical image related to a case that the examiner wants to confirm.

In addition, even if a machine translation engine (machine translation model, trained model for machine translation) that machine translates sentences such as characters and voices into any language is used for instructions from the examiner and output to the examiner. good. In addition, any language may be configured to be selectable according to an instruction from the examiner. Further, any language may be configured to be automatically selectable by using a trained model that automatically recognizes the type of language. Further, the automatically selected language type may be configured to be modifiable according to an instruction from the examiner. For example, the above-mentioned techniques related to natural language processing (for example, Sequence to Sequence) may be applied to the machine translation engine. For example, after the sentence input to the machine translation engine is machine-translated, the machine-translated sentence may be input to the character recognition engine or the like. Further, for example, the sentences output from the various trained models described above may be input to the machine translation engine, and the sentences output from the machine translation engine may be output.

Further, the various trained models described above may be used in combination. For example, the character corresponding to the instruction from the examiner is input to the character recognition engine, and the voice obtained from the input character is input to another type of machine learning engine (for example, a machine translation engine). May be done. Further, for example, characters output from other types of machine learning engines may be input to the character recognition engine, and the voice obtained from the input characters may be output. Further, for example, the voice corresponding to the instruction from the examiner is input to the voice recognition engine, and the characters obtained from the input voice are input to another type of machine learning engine (for example, a machine translation engine). It may be configured in. Further, for example, the voice output from another type of machine learning engine may be input to the voice recognition engine, and the characters obtained from the input voice may be displayed on the display unit 103. At this time, it may be configured so that the output to the examiner can be selected from the output by characters and the output by voice according to the instruction from the examiner. Further, it may be configured so that the instruction from the examiner can be selected from the input by characters and the input by voice according to the instruction from the examiner. Further, the various configurations described above may be adopted by selection according to an instruction from the examiner.

[Modification 13]
A label image, a high-quality image, or the like related to the image acquired by the main shooting may be stored in the storage unit 126 according to an instruction from the operator. At this time, for example, after an instruction from the operator for saving a high-quality image, when registering the file name, as a recommended file name, any part of the file name (for example, the first part, or In the last part), a file name containing information (for example, characters) indicating that the image is generated by processing using a trained model for high image quality (high image quality processing) is given by the operator. It may be displayed in an editable state according to the instruction. Similarly, for the label image and the like, the file name including the information which is the image generated by the process using the trained model may be displayed.

Further, when displaying a high-quality image on the display unit 103 on various display screens such as a report screen, the displayed image is a high-quality image generated by processing using a high-quality model. The display shown may be displayed together with a high-quality image. In this case, the operator can easily identify from the display that the displayed high-quality image is not the image itself acquired by shooting, thus reducing erroneous diagnosis and improving diagnostic efficiency. be able to. The display indicating that the image is a high-quality image generated by the process using the high-quality model is any mode as long as the input image and the high-quality image generated by the process can be distinguished. It may be one. Further, it is shown that not only the processing using the high image quality model but also the processing using various trained models as described above is the result generated by the processing using the trained model of that kind. The display may be displayed with the result. For example, when displaying the analysis result of the segmentation result using the trained model for image segmentation processing, the display indicating that the analysis result is based on the result using the trained model for image segmentation is analyzed. It may be displayed with the result.

At this time, the display screen such as the report screen may be stored in the storage unit 126 as image data in response to an instruction from the operator. For example, the report screen may be stored in the storage unit 126 as one image in which a high-quality image or the like and a display indicating that these images are images generated by processing using the trained model are arranged side by side. ..

In addition, regarding the display indicating that the image is a high-quality image generated by processing using the high-quality model, the display unit shows what kind of learning data the high-quality model has learned. It may be displayed in 103. The display may include an explanation of the types of input data and correct answer data of learning data, and an arbitrary display regarding correct answer data such as an imaging site included in the input data and correct answer data. It should be noted that even in the case of processing using the various trained models described above, such as image segmentation processing, the display unit 103 displays a display showing what kind of training data the trained model of that type was trained by. It may be displayed.

Further, information (for example, characters) indicating that the image is generated by processing using the trained model may be displayed or stored in a state of being superimposed on the image or the like. At this time, the portion superimposed on the image may be any region (for example, the edge of the image) that does not overlap with the region where the region of interest to be photographed is displayed. Further, the non-overlapping areas may be determined and superimposed on the determined areas. It should be noted that not only the processing using the high image quality model but also the image obtained by the processing using the various trained models described above such as the image segmentation processing may be processed in the same manner.

In addition, if the initial display screen of the report screen is set by default so that the high image quality processing button or the like is in the active state (high image quality processing is on), it is set to high according to the instruction from the examiner. The report image corresponding to the report screen including the image quality image may be configured to be transmitted to the server. Also, if the button is set to the active state by default, at the end of the inspection (for example, when the shooting confirmation screen or preview screen is changed to the report screen according to the instruction from the inspector). In addition, the report image corresponding to the report screen including the high-quality image may be configured to be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, whether or not the analysis map is superimposed, whether or not the image is a high-quality image, and the display screen for follow-up observation. The report image generated based on (settings related to at least one such as whether or not) may be configured to be sent to the server. It should be noted that the same processing may be performed even when the button represents the switching of the image segmentation processing.

[Modification 14]
In the above-described embodiments and modifications, among the various trained models described above, images obtained by the first type of trained model (for example, high-quality images, images showing analysis results such as analysis maps, etc.). An image showing a predetermined region detection result, an image showing a segmentation result) may be input to a trained model of a second type different from the first type. At this time, it may be configured to generate the result (for example, estimation result, analysis result, diagnosis result, predetermined area detection result, segmentation result) by the processing of the second kind of trained model.

Further, among the various trained models as described above, the results obtained by processing the first type of trained model (for example, estimation result, analysis result, diagnosis result, predetermined area detection result, segmentation result) are used. From the image input to the trained model of the first type, an image to be input to the trained model of the second type different from the first type may be generated. At this time, the generated image is likely to be an image suitable as an image to be processed using the second type of trained model. Therefore, an image obtained by inputting the generated image into the second type of trained model (for example, a high-quality image, an image showing an analysis result such as an analysis map, an image showing a predetermined area detection result, and a segmentation result). The accuracy of the image) can be improved.

By inputting a common image into the first type of trained model and the second type of trained model, the generation (or display) of each processing result using these trained models can be generated. It may be configured to run. At this time, for example, it may be configured to collectively (in conjunction with) generate (or display) each processing result using these trained models in response to an instruction from the examiner. In addition, the type of image to be input (for example, high-quality image, object recognition result, segmentation result, similar case image), and the type of processing result to be generated (or displayed) (for example, high-quality image, estimation result, diagnosis result, analysis). The result, the object recognition result, the segmentation result, the similar case image), the input type and the output type (for example, characters, voice, language) and the like may be configured to be selectable according to the instruction from the examiner. Further, the input type may be configured to be automatically selectable by using a trained model that automatically recognizes the input type. Further, the output type may be configured to be automatically selectable so as to correspond to the input type (for example, to be the same type). Further, the automatically selected type may be configured to be modifiable according to an instruction from the examiner. At this time, at least one trained model may be configured to be selected according to the selected type. At this time, when a plurality of trained models are selected, how to combine the plurality of trained models (for example, the order in which data is input) may be determined according to the selected type. It should be noted that, for example, the type of the image to be input and the type of the processing result to be generated (or displayed) may be configured to be differently selectable, or if they are the same, they may be selected differently. It may be configured to output prompting information to the examiner. Also, each trained model may be executed anywhere. For example, some of the plurality of trained models may be used in a cloud server, and the others may be configured to be used in another server such as a fog server or an edge server. In addition, when the network in the facility, the site including the facility, the area including multiple facilities, etc. is configured to enable wireless communication, for example, it is assigned only to the facility, the site, the area, etc. The reliability of the network may be improved by configuring to use radio waves in a dedicated wavelength band. Further, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and simultaneous connection of a large number of people. With these, for example, surgery such as vitreous body, cataract, glaucoma, corneal refraction correction, external eye, and treatment such as laser photocoagulation can be supported in real time even if it is remote. At this time, for example, information obtained by a fog server, an edge server, or the like that wirelessly receives at least one of various medical images obtained by these devices related to surgery or treatment using at least one of various trained models. May be configured to wirelessly transmit to a device for surgery or treatment. Further, for example, the information wirelessly received by the device related to surgery or treatment may be the amount of movement (vector) of the optical system or optical member as described above. In this case, the device related to surgery or treatment is automatically controlled. It may be configured to be. Further, for example, for the purpose of supporting the operation by the examiner, it may be configured as an automatic control (semi-automatic control) with the permission of the examiner.

Further, a similar case image search using an external database stored in a server or the like may be performed using the analysis result, the diagnosis result, etc. obtained by the processing of the trained model as described above as a search key. Further, a similar case image search using an external database stored in a server or the like may be performed using an object recognition result, a segmentation result, or the like obtained by processing various trained models as described above as a search key. If the plurality of medical images stored in the database are already managed by machine learning or the like with the feature amount of each of the plurality of medical images attached as incidental information, the medical image itself may be used. A similar case image search engine (similar case image search model, trained model for similar case image search) may be used as a search key. For example, the control unit 102 searches various medical images for similar case images related to the medical image by using the trained model for similar case image search (different from the trained model for high image quality). It can be carried out. Further, for example, the display control unit 125 can display a similar case image obtained from various medical images using a learned model for searching for a similar case image on the display unit 103. At this time, the similar case image is, for example, an image having a feature amount similar to the feature amount of the medical image input to the trained model. Further, the similar case image is, for example, an image of a feature amount similar to the feature amount of the partial area such as the abnormal part when the medical image input to the trained model includes a partial area such as an abnormal part. .. Therefore, for example, not only can the learning for accurately searching for similar case images be performed efficiently, but also when the medical image contains an abnormal part, the examiner efficiently diagnoses the abnormal part. be able to. Further, a plurality of similar case images may be searched, and a plurality of similar case images may be displayed so that the order in which the feature amounts are similar can be identified. In addition, a trained model for searching similar case images is additionally learned using learning data including an image selected according to an instruction from the examiner and a feature amount of the image among a plurality of similar case images. It may be configured to be.

Further, the training data of various trained models is not limited to the data obtained by using the ophthalmic apparatus itself that actually performs the imaging, and the data obtained by using the same type of ophthalmic apparatus or the same type according to the desired configuration. It may be data obtained by using an ophthalmic apparatus or the like.

It should be noted that various trained models according to the above-described embodiments and modifications can be provided in the control unit 102. The trained model may be configured by, for example, a CPU, a software module executed by a processor such as an MPU, GPU, FPGA, or the like, or may be configured by a circuit or the like that performs a specific function such as an ASIC. Further, these learned models may be provided in a device of another server connected to the control unit 102 or the like. In this case, the control unit 102 can use the trained model by connecting to a server or the like provided with the trained model via an arbitrary network such as the Internet. Here, the server provided with the trained model may be, for example, a cloud server, a fog server, an edge server, or the like. In addition, when the network in the facility, the site including the facility, the area including multiple facilities, etc. is configured to enable wireless communication, for example, it is assigned only to the facility, the site, the area, etc. The reliability of the network may be improved by configuring to use radio waves in a dedicated wavelength band. Further, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and simultaneous connection of a large number of people.

[Modification 15]
The medical image processed by the control unit 102 according to the various embodiments and modifications described above includes an image acquired by using an arbitrary modality (imaging device, imaging method). The medical image to be processed may include a medical image acquired by an arbitrary imaging device or the like, or an image created by a medical image processing device or a medical image processing method.

Further, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. In addition, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black-and-white image or a color image. Further, the medical image may be an image showing the structure (morphology) of a predetermined part or an image showing the function thereof. The image showing the function includes, for example, an OCTA image, a Doppler OCT image, an fMRI image, and an image showing hemodynamics (blood flow volume, blood flow velocity, etc.) such as an ultrasonic Doppler image. The predetermined site of the subject may be determined according to the subject to be imaged, and the human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, organs such as liver, head, chest, etc. Includes any part such as legs and arms. In particular, in the various embodiments and modifications described above, the medical image relating to the eye to be inspected was used for the estimation process. In this regard, the subject relating to the medical image used for the estimation process in the various embodiments and modifications described above is not limited to the subject to be examined, and may be a subject having symmetry in the left-right direction, the up-down direction, or the left-right up-down direction. It may be other organs such as lungs. However, the embodiment according to the third embodiment is not limited to the subject having symmetry. When the subject is an organ such as a lung, the imaging device 101 may have, for example, an endoscope or the like.

Further, the medical image may be a tomographic image of the subject or a frontal image. The frontal image is, for example, an SLO image of the fundus or anterior segment of the eye, a fundus image photographed by fluorescence, and data acquired by OCT (three-dimensional OCT data) in at least a part of the range in the depth direction of the imaged object. Includes En-Face images generated using. The En-Face image is an OCTA En-Face image (motion contrast front image) generated by using at least a part of the data in the depth direction of the shooting target for the three-dimensional OCTA data (three-dimensional motion contrast data). ) May be. Further, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Here, the motion contrast data is data showing a change between a plurality of volume data obtained by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data showing changes between a plurality of tomographic images obtained at substantially the same position at different positions. The motion contrast front image is also referred to as an OCTA front image (OCTA En-Face image) relating to OCTA angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also referred to as OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals. , Can be determined by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. When controlling the scanning means so that the measurement light is scanned multiple times at substantially the same position, the time interval (time interval) between one scan (one B scan) and the next scan (next B scan). ) May be modified (determined). Thereby, for example, even if the blood flow velocity may differ depending on the state of the blood vessel, the blood vessel region can be visualized with high accuracy. At this time, for example, the time interval may be changed so as to be instructed by the examiner. Further, for example, any motion contrast image may be configured to be selectable from a plurality of motion contrast images corresponding to a plurality of preset time intervals according to an instruction from the examiner. Further, for example, the time interval at which the motion contrast data is acquired may be associated with the motion contrast data so that the motion contrast data can be stored in the storage unit 126. Further, for example, the display control unit 125 may display the time interval when the motion contrast data is acquired and the motion contrast image corresponding to the motion contrast data on the display unit 103. Further, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. At this time, for example, using a machine learning model, the time interval may be determined (output) from the motion contrast image. In such a machine learning model, for example, a plurality of motion contrast images corresponding to a plurality of time intervals are used as input data, and the difference from the plurality of time intervals to the time interval when a desired motion contrast image is acquired is correctly answered. Learning as data It can be obtained by learning the data.

The En-Face image is, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions. At this time, the front image is at least a part of the depth range of the volume data (three-dimensional tomographic image) obtained by using optical interference, and is the data corresponding to the depth range determined based on the two reference planes. Is projected or integrated on a two-dimensional plane. The En-Face image is a frontal image generated by projecting the data corresponding to the depth range determined based on the detected retinal layer among the volume data onto a two-dimensional plane. As a method of projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on the two-dimensional plane. Techniques can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Further, the depth range related to the En-Face image is, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries relating to the detected retinal layer. May be good. Further, the depth range related to the En-Face image may be, for example, a range changed (offset) according to the instruction of the operator from the range between the two layer boundaries regarding the detected retinal layer. good.

The photographing device is a device for photographing an image used for diagnosis. The photographing device detects, for example, a device that obtains an image of a predetermined part by irradiating a predetermined part of the subject with radiation such as light or X-rays, electromagnetic waves, ultrasonic waves, or the like, or radiation emitted from a subject. This includes a device for obtaining an image of a predetermined part. More specifically, the imaging devices according to the various embodiments and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, and a fundus. Includes cameras, endoscopes, etc. It should be noted that the configuration according to each of the above-described embodiments and modifications can be applied to these imaging devices. In this case, the movement of the subject corresponding to the above-mentioned movement of the eye to be predicted may be, for example, the movement of the face or body, the movement of the heart (heartbeat), or the like.

The OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Further, the Fourier domain OCT apparatus may include a spectral domain OCT (SD-OCT) apparatus and a wavelength sweep type OCT (SS-OCT) apparatus. Further, the OCT apparatus may include a Line-OCT apparatus (or SS-Line-OCT apparatus) using line light. Further, the OCT apparatus may include a Full Field-OCT apparatus (or SS-Full Field-OCT apparatus) using area light. Further, the OCT device may include a Doppler-OCT device. Further, the SLO device and the OCT device may include a wave surface compensating SLO (AO-SLO) device using a wave surface compensating optical system, a wave surface compensating OCT (AO-OCT) device, and the like. Further, the SLO device and the OCT device may include a polarized SLO (PS-SLO) device, a polarized OCT (PS-OCT) device, and the like for visualizing information on polarization phase difference and polarization elimination. Further, the SLO apparatus and the OCT apparatus may include a pathological microscope SLO apparatus, a pathological microscope OCT apparatus, and the like. Further, the SLO device and the OCT device may include a handheld type SLO device, a handheld type OCT device, and the like. Further, the SLO device and the OCT device may include a catheter SLO device, a catheter OCT device and the like. Further, the SLO device and the OCT device may include a head-mounted SLO device, a head-mounted OCT device, and the like. Further, the SLO device and the OCT device may include a binocular type SLO device, a binocular type OCT device, and the like. Further, the SLO device and the OCT device may be capable of changing the shooting angle of view by a configuration capable of optical scaling. Further, the SLO device can acquire a color image or a fluorescent image by using each of the RGB light sources and having a configuration in which one light receiving element receives light in a time-division manner or a configuration in which a plurality of light receiving elements simultaneously receive light. May be good.

Further, in the above-described embodiment and modification, the control unit 102 is configured as a part of the OCT device 100, but the control unit 102 may be configured as a separate body from the OCT device 100. In this case, the control unit 102 may be connected to the photographing device 101 or the like of the OCT device 100 via the Internet or the like. Further, the configuration of the OCT apparatus 100 is not limited to the above configuration, and a part of the configuration included in the OCT apparatus may be a configuration in which, for example, an SLO photographing unit or the like is separate from the OCT apparatus.

In the learned model for estimation processing according to the third embodiment described above, since learning is performed using time-series data, the slope between the input continuous time-series data values is the feature amount. It is considered that it is extracted as a part and used for the estimation process. Similarly, in trained models for voice recognition, character recognition, gesture recognition, etc., the gradient between input continuous time-series data values is extracted as part of the feature quantity and used for estimation processing. it is conceivable that. It is expected that such a trained model can perform accurate estimation by using the influence of a specific numerical value over time in the estimation process. Further, even in the trained models for estimation processing, high image quality, segmentation processing, image analysis, and diagnosis result generation according to the above-described embodiments and modifications, the magnitude and brightness of the tomographic image are different. It can be considered that the order, inclination, position, distribution, continuity, etc. of the dark part are extracted as a part of the feature amount and used for the estimation process.

(Other examples)
The present invention is also a process in which a program that realizes one or more functions of the above-described embodiments and modifications is supplied to a system or device via a network or storage medium, and a computer of the system or device reads and executes the program. It is feasible. A computer may have one or more processors or circuits and may include multiple separate computers or a network of separate processors or circuits for reading and executing computer-executable instructions.

The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Although the present invention has been described above with reference to the embodiments and modifications, the present invention is not limited to the above embodiments and modifications. The present invention also includes inventions modified to the extent not contrary to the gist of the present invention, and inventions equivalent to the present invention. In addition, the above-described embodiments and modifications can be appropriately combined as long as they do not contradict the gist of the present invention.

102: Control unit (image processing device), 122: Image processing unit (generation unit), 123: Estimating unit (output unit)

Claims (28)

A generator that processes the medical image of the subject to generate a symmetry map showing the symmetry of the subject.
Using the trained model obtained by learning using the symmetry map, an output unit that outputs the estimation result regarding the disease estimated from the generated symmetry map, and the output unit.
An image processing device. The medical image is at least one of an analysis map showing information obtained by analyzing a three-dimensional tomographic image, a fundus photograph, an anterior segment photograph, an SLO image, an OCTA image, an En-Face image, and a medical image of a subject. The image processing apparatus according to claim 1, comprising one. The symmetry map generated by the generation unit is any one of a fundus photograph, an anterior segment photograph, an SLO image, an OCTA image, an En-Face image, and an analysis map showing information obtained by analyzing a medical image of a subject. The image processing apparatus according to claim 1 or 2, which is a symmetry map that evaluates the symmetry of at least one of the vertical direction and the horizontal direction. In addition to the symmetry map generated by the generation unit, the output unit is a measurement result by a perimeter, a two-dimensional tomographic image, a three-dimensional tomographic image, a fundus photograph, an anterior segment photograph, an SLO image, and an En-Face. Image, OCTA image, analysis map showing information obtained by analyzing the medical image of the subject, corner angle information, layer thickness profile or feature vector, feature map from which the feature part of the medical image is extracted, and patient data. The image processing apparatus according to any one of claims 1 to 3, which outputs an estimation result obtained by using at least one. The analysis map according to any one of claims 2 to 4, wherein the analysis map includes at least one of a thickness map showing the thickness of at least a part of the subject and a blood vessel density map showing the blood vessel density of at least a part of the subject. Image processing equipment. The image processing apparatus according to any one of claims 1 to 5, wherein the disease comprises at least one of glaucoma, diabetic retinopathy, and age-related macular degeneration. The image processing apparatus according to any one of claims 1 to 6, wherein the generation unit performs correction processing of the medical image before generating a symmetry map. The correction process is at least one of a smoothing process, a correction process based on the axial length, a layer thickness correction process based on a reference layer, a geometric correction process, an error correction process, an edge enhancement process, and a contrast adjustment process. 7. The image processing apparatus according to claim 7. The image processing apparatus according to any one of claims 1 to 8, wherein the generation unit generates a symmetry map using an image of a region extracted from the medical image. The trained model outputs the probability of the disease and
The image processing apparatus according to claim 9, wherein the output unit outputs an estimation result for the disease estimated based on the probability output by the trained model. The image processing apparatus according to any one of claims 1 to 10, wherein the generation unit selects an image to be image-processed according to a disease to be diagnosed. The image processing apparatus according to any one of claims 1 to 11, wherein the output unit outputs estimation results in a plurality of stages. The image processing apparatus according to any one of claims 1 to 12, wherein the estimation result includes an estimation result regarding the stage of the disease. The trained model includes a plurality of trained models.
One of claims 1 to 13, wherein the output unit outputs an estimation result estimated using a trained model selected from the plurality of trained models according to the disease to be diagnosed. The image processing apparatus according to. The image processing apparatus according to any one of claims 1 to 14, wherein the medical image includes a plurality of medical images obtained by photographing a subject at different times. The generation unit generates a correlation map showing the correlation between a plurality of time symmetry maps obtained by image processing the plurality of medical images.
The image processing apparatus according to claim 15, wherein the output unit outputs an estimation result of estimating the progression of the disease from the correlation map. The image processing apparatus according to claim 16, wherein the generation unit generates the correlation map by comparing a plurality of time symmetry maps. The image processing apparatus according to claim 16 or 17, wherein the output unit estimates the progression of the disease from the time-series data of the correlation map. The trained model estimates the progression of the disease based on a trained model for estimating the disease from a single-time symmetry map and a correlation map generated from time-series data of the symmetry map. The image processing apparatus according to any one of claims 16 to 18, including the trained model of the above. The image processing apparatus according to any one of claims 1 to 19, wherein the output unit outputs a caution map in which the area of interest of the trained model is mapped. The acquisition unit that acquires the medical image of the subject,
An output unit that outputs the measurement result of the subject's perimeter and the estimation result of the disease estimated from the acquired medical image using the learned model obtained by the measurement result by the perimeter and the learning using the medical image.
An image processing device. An analysis result obtained by analyzing a medical image of a subject, and a generation unit that generates a comparison map comparing the analysis results of positions related to each other.
Using the trained model obtained by learning using the comparison map, an output unit that outputs the estimation result regarding the disease estimated from the generated comparison map, and the output unit.
An image processing device. A generator that generates a correlation map showing the correlation between images by image processing a plurality of medical images obtained by photographing a subject at different times.
Using the trained model obtained by learning using the correlation map, an output unit that outputs the estimation result regarding the disease estimated from the generated correlation map, and the output unit.
An image processing device. To generate a symmetry map showing the symmetry of the subject by image processing the medical image of the subject,
Using the trained model obtained by learning using the symmetry map, the estimation result regarding the disease estimated from the generated symmetry map can be output.
Image processing methods, including. Obtaining a medical image of the subject and
Using the measurement result by the perimeter and the learned model obtained by learning using the medical image, the measurement result by the perimeter of the subject and the estimation result regarding the disease estimated from the acquired medical image can be output.
Image processing methods, including. It is an analysis result obtained by analyzing the medical image of the subject, and it is possible to generate a comparison map comparing the analysis results of the positions related to each other.
Using the trained model obtained by learning using the comparison map, the estimation result regarding the disease estimated from the generated comparison map can be output.
Image processing methods, including. Multiple medical images obtained by photographing a subject at different times are image-processed to generate a correlation map showing the correlation between the images.
Using the trained model obtained by learning using the correlation map, the estimation result regarding the disease estimated from the generated correlation map can be output.
Image processing methods, including. A program that, when executed by a computer, causes the computer to execute each step of the image processing method according to any one of claims 24 to 27.

Images