JP2020058615A - Image processing device, learned model, image processing method, and program - Google Patents

Abstract

To provide information more appropriate for assisting eye diagnostic by an examiner than that in a prior art.SOLUTION: An image processing device includes acquisition means which acquires data related to tomography containing a plurality of types of analysis values of an optometry object eye and being related to first tomography of the optometry object eye associated to a first time point, and data of second tomography of the optometry object eye associated to a second time point after the first time point; and generation means which generates, from the required data related to the first tomography and the required data related to the second tomography, at least one of the plurality of types of analysis values of the optometry object eye at a third time point after the second time point using a learned model.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an image processing device, a learned model, an image processing method, and a program.

  A tomographic image capturing apparatus of an eye such as an apparatus (OCT apparatus) using an optical coherence tomography (OCT: Optical Coherence Tomography) is capable of three-dimensionally observing a state inside a retinal layer. This tomographic imaging apparatus has been attracting attention in recent years because it is useful for more accurate diagnosis of diseases.

  As a form of OCT, for example, there is TD-OCT (Time domain OCT) in which a broadband light source and a Michelson interferometer are combined. This is configured to scan the delay of the reference arm, measure the interference light with the backscattered light of the signal arm, and obtain the information in the depth direction. However, high-speed image acquisition is difficult with such TD-OCT.

  On the other hand, as a method for acquiring an image at a higher speed, SD-OCT (Spectral domain OCT) is known which uses a broadband light source and acquires an interferogram by a spectroscope. In addition, SS-OCT (Swept Source OCT) is known, which uses a high-speed wavelength swept light source as a light source and measures spectral interference with a single-channel photodetector.

  By measuring the thickness of the nerve fiber layer using a tomographic image taken by OCT, it is possible to quantitatively diagnose the degree of progress of diseases such as glaucoma and the degree of recovery after treatment. Patent Literature 1 discloses an ophthalmic analysis apparatus that creates a time-series graph of analysis values such as film thickness obtained from a tomographic image of the fundus. In this ophthalmological analysis device, future analysis values are predicted from the regression line obtained from the time series graph.

JP, 2017-170193, A

  However, depending on the type of lesion or abnormality that occurs in the eye, the eye may show multiple features rather than a single feature, and prediction based on specific analysis values may not be sufficient for diagnostic assistance. There is.

  One of the objects of the present invention has been made in view of the above problems, and provides information more suitable than before when assisting an eye diagnosis by an examiner.

In order to achieve the above object, an image processing device according to an aspect of the present invention is
Data relating to a slice including multiple types of analysis values of the eye to be inspected, data relating to the first slice of the eye to be examined associated with a first time, and a second time after the first time. Data relating to the second cross-section of the associated eye to be inspected, and acquisition means,
Using a learned model, from the acquired data concerning the first tom and the acquired data concerning the second tom, a plurality of types of the eye to be examined at a third time after the second time. Generating means for generating at least one of the analysis values of.

  According to one aspect of the present invention, it is possible to provide more suitable information than before when assisting the eye diagnosis by the examiner.

1 shows an example of a schematic configuration of an image processing system according to a first embodiment. It is a figure for explaining the structure of an eye part, a tomographic image, and a fundus image. 5 is a flowchart showing a series of processes according to the first embodiment. It is a figure for explaining an example of learning data. It is a figure for explaining an example of a machine learning model. It is a figure for explaining an example of a machine learning model. FIG. 6 is a diagram for explaining a display screen of a processing result according to the first embodiment. 1 shows an example of a schematic configuration of an image processing system according to a second embodiment. 9 shows an example of a schematic configuration of an image processing system according to a third embodiment. FIG. 11 is a diagram for explaining a display screen of processing results according to the third embodiment.

  Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of constituent elements, and the like described in the following embodiments are arbitrary, and can be changed according to the configuration of the apparatus to which the present invention is applied or various conditions. Also, in the drawings, the same reference numbers are used in the drawings to denote the same or functionally similar elements.

(Example 1)
Hereinafter, with reference to FIGS. 1 to 7, an image processing system according to a first embodiment of the present invention that presents information that will be diagnostic assistance for an eye to be inspected will be described. In the present embodiment, a future analysis value of the target eye to be inspected is generated using the learned model related to the machine learning model. Specifically, a future analysis value is predicted using a learned model obtained by learning using a plurality of types of analysis values obtained from a tomographic image of the eye to be examined, and the result is presented.

  Here, the machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. Further, the learned model is a model obtained (trained) by preliminarily training a machine learning model by an arbitrary machine learning algorithm using appropriate learning data. However, although the learned model is obtained in advance by using appropriate learning data, it is not the case that further learning is not performed but additional learning can be performed.

  FIG. 1 shows an example of a schematic configuration of an image processing system 1 including an image processing device 300 according to this embodiment. As shown in FIG. 1, the image processing system 1 includes an OCT apparatus (hereinafter referred to as OCT apparatus 200) as an example of a tomographic image capturing apparatus, an image processing apparatus 300, a fundus image capturing apparatus 400, an external storage device 500, and a display unit 600. , And an input unit 700 are provided.

  The OCT apparatus 200 is used as an apparatus that captures a tomographic image of the subject's eye. Any type of OCT device can be used, for example, SD-OCT or SS-OCT can be used.

  The image processing apparatus 300 is connected to the OCT apparatus 200, the fundus image capturing apparatus 400, the external storage device 500, the display unit 600, and the input unit 700 via an interface, and can control these. The image processing apparatus 300 generates various images such as a tomographic image of an eye to be examined and an En-Face image (front image) based on various signals acquired from the OCT apparatus 200, the fundus image capturing apparatus 400, and the external storage device 500. can do. Further, the image processing apparatus 300 can perform image processing on these images. The image processing apparatus 300 may be configured by a general-purpose computer or a computer dedicated to the image processing system 1.

  The fundus image capturing device 400 is a device for capturing a fundus image of an eye to be inspected, and as the device, for example, a fundus camera, an SLO (Scanning Laser Ophthalmoscope), or the like can be used. The OCT device 200 and the fundus image capturing device 400 may be integrated or separated.

  The external storage device 500 holds information relating to the eye to be inspected (name, age, sex, etc. of the patient), various image data captured, imaging parameters, image analysis parameters, and parameters set by the user in association with each other. . The external storage device 500 may be configured by any storage device, for example, a storage medium such as an optical disk or a memory.

  The display unit 600 is configured by an arbitrary display and can display information regarding the eye to be inspected and various images under the control of the image processing device 300.

  The input unit 700 includes, for example, a mouse, a keyboard, a touch operation screen, or the like. The user can input an instruction to the image processing apparatus 300, the OCT apparatus 200, and the fundus image capturing apparatus 400 to the image processing apparatus 300 via the input unit 700. When the input unit 700 is a touch operation screen, the input unit 700 can be integrated with the display unit 600.

  Note that these constituent elements are shown as separate bodies in FIG. 1, but some or all of these constituent elements may be integrally configured.

  Next, the OCT device 200 will be described. The OCT device 200 is provided with a galvanometer mirror 201, a drive control unit 202, a focus lens stage 203, an internal fixation lamp 204, a coherence gate stage 205, a light source 206, and a detector 207. Since the OCT apparatus 200 is a known apparatus, detailed description thereof will be omitted, and here, the imaging of a tomographic image performed according to an instruction from the image processing apparatus 300 will be described.

  When a shooting instruction is transmitted from the image processing device 300, the light source 206 emits light. The light from the light source 206 is split into measurement light and reference light using a splitting unit (not shown). The OCT apparatus 200 can generate an interference signal including tomographic information of the eye to be inspected by irradiating the eye to be inspected with the measurement light and detecting the interference light between the return light from the subject and the reference light.

  The galvanometer mirror 201 is used to scan the fundus of the subject's eye with the measurement light, and the scanning range of the measurement light by the galvanometer mirror 201 can define the imaging range of the fundus during tomographic imaging. By controlling the drive range and speed of the galvano mirror 201, the image processing apparatus 300 can define the imaging range in the plane direction and the number of scanning lines (scanning speed in the plane direction) at the fundus. In FIG. 1, the galvano mirror 201 is shown as one unit for the sake of simplification of description, but the galvano mirror 201 is actually composed of a mirror for X scanning and two mirrors for Y scanning, The desired range above can be scanned with the measuring light. The configuration of the scanning unit for scanning the measurement light is not limited to the galvanometer mirror, and any other deflection mirror can be used. Further, as the scanning unit, for example, a deflection mirror capable of scanning the measurement light in a two-dimensional direction with one sheet such as a MEMS mirror may be used.

  The focus lens stage 203 is provided with a focus lens (not shown). By moving the focus lens stage 203, the focus lens can be moved along the optical axis of the measurement light. Therefore, the focus lens can focus the measurement light on the retinal layer of the fundus through the anterior segment of the subject's eye. The measurement light emitted to the fundus of the eye is reflected and scattered by each retinal layer and returns to the optical path as return light.

  The coherence gate stage 205 is used to adjust the length of the optical path of the reference light or the measurement light in order to deal with the difference in the axial length of the eye to be inspected and the like. In this embodiment, the coherence gate stage 205 is composed of a stage provided with a mirror, and the optical path length of the reference light can be made to correspond to the optical path length of the measurement light by moving in the optical axis direction in the optical path of the reference light. it can. Here, the coherence gate represents a position where the optical distances of the measurement light and the reference light in OCT are equal. The coherence gate stage 205 can be controlled by the image processing device 300. The image processing apparatus 300 can control the imaging range in the depth direction of the eye to be inspected by controlling the position of the coherence gate by the coherence gate stage 205, and the imaging on the retina layer side or on the side deeper than the retina layer. It is possible to control shooting and the like.

  The detector 207 detects the interference light between the return light of the measurement light from the eye to be inspected and the reference light, which is generated in the interference section (not shown), and generates an interference signal. The image processing apparatus 300 can generate a tomographic image of the eye to be inspected by acquiring the interference signal from the detector 207 and performing Fourier transform or the like on the interference signal.

  The internal fixation lamp 204 is provided with a display unit 241 and a lens 242. In this embodiment, as an example of the display unit 241, a plurality of light emitting diodes (LD) arranged in a matrix is used. The lighting position of the light emitting diode is changed under the control of the image processing apparatus 300 in accordance with the part to be photographed. The light from the display unit 241 is guided to the subject's eye via the lens 242. The light emitted from the display unit 241 has a wavelength of 520 nm, for example, and is displayed in a desired pattern under the control of the image processing device 300.

  The drive control unit 202 is connected to the galvano mirror 201, the focus lens stage 203, the internal fixation lamp 204, the coherence gate stage 205, the light source 206, and the detector 207 described above. The drive control unit 202 controls the drive of each of these components based on the control by the image processing apparatus 300 via the instruction unit 304 described later. The drive control unit 202 may not be provided, and the image processing apparatus 300 may directly control the drive of these components using the instruction unit 304.

  Next, with reference to FIGS. 2A to 2C, the structure and image of the eye acquired by the image processing system 1 will be described. FIG. 2A is a schematic diagram of an eyeball. In FIG. 2A, the cornea C, the lens CL, the vitreous body V, the macula M (the center of the macula represents the fovea), and the optic disc D are shown. In the present embodiment, a case will be mainly described in which the posterior pole of the retina including the vitreous body V, the macula M, and the optic papilla D is imaged. Although not described below, the OCT apparatus 200 can also image the anterior ocular segment such as the cornea and the crystalline lens.

  FIG. 2B shows an example of a tomographic image acquired by imaging the retina using the OCT apparatus 200. In FIG. 2B, AS indicates an image unit acquired by one A scan. Here, the A scan refers to acquisition of tomographic information in the depth direction at one point of the eye to be inspected by a series of operations of the OCT apparatus 200. In addition, acquiring the two-dimensional tomographic information of the eye to be examined in the transverse direction and the depth direction by performing the A scan a plurality of times in an arbitrary transverse direction (main scanning direction) is referred to as a B scan. One B-scan image can be constructed by collecting a plurality of A-scan images acquired by the A-scan. Hereinafter, this B-scan image will be referred to as a tomographic image.

  In FIG. 2B, the vitreous body V, the macula M, the optic disc portion D, and the lamina cribrosa La are shown. The boundary line L1 is the boundary between the inner limiting membrane (ILM) and the nerve fiber layer (NFL), the boundary line L2 is the boundary between the nerve fiber layer and the ganglion cell layer (GCL), and the boundary line L3 is the photoreceptor inner node. Represents an outer segment joint (ISOS). Further, the boundary line L4 represents the retinal pigment epithelium layer (RPE), the boundary line L5 represents the Bruch's membrane (BM), and the boundary line L6 represents the choroid. In the tomographic image, the horizontal axis (main scanning direction of OCT) is the x axis, and the vertical axis (depth direction) is the z axis.

  FIG. 2C shows an example of the fundus image acquired by photographing the fundus of the subject's eye using the fundus image photographing device 400. In FIG. 2C, the macula M and the optic papilla D are shown, and the blood vessels of the retina are shown by thick curves. In the fundus image, the horizontal axis (the main scanning direction of the OCT apparatus) is the x axis, and the vertical axis (the sub scanning direction of the OCT apparatus) is the y axis.

  Next, the image processing apparatus 300 will be described. The image processing apparatus 300 includes an acquisition unit 301, a storage unit 302, a processing unit 303, an instruction unit 304, and a display control unit 305.

  The acquisition unit 301 can acquire the data of the interference signal of the subject's eye from the OCT apparatus 200. The interference signal data acquired by the acquisition unit 301 may be an analog signal or a digital signal. When the acquisition unit 301 acquires an analog signal, the image processing apparatus 300 can convert the analog signal into a digital signal. Further, the acquisition unit 301 includes a tomographic image generation unit 311, and can acquire various images such as tomographic data and tomographic images generated by the tomographic image generation unit 311 and En-Face images. Here, the tomographic data is data including information about a tomographic image of a subject, data based on an interference signal by OCT, and data obtained by performing a fast Fourier transform (FFT: Fast Fourier Transform) or arbitrary signal processing on the data. Is meant to include.

  The tomographic image generation unit 311 can generate tomographic data by performing processing such as Fourier transform on the interference signal acquired by the acquisition unit 301, and can generate a tomographic image based on the tomographic data. Any known method may be used as the method of generating the tomographic image, and detailed description thereof will be omitted.

  Furthermore, the acquisition unit 301 includes a group of imaging conditions of a tomographic image to be image-processed (for example, imaging date / time, imaging site name, imaging region, imaging angle of view, imaging method, image resolution and gradation, image pixel size, image Get information such as filters and image data format). The shooting condition group is not limited to the illustrated one. Further, the shooting condition group does not need to include all of the exemplified ones, and may include some of them.

  The acquisition unit 301 can also acquire data including the fundus information acquired by the fundus image capturing apparatus 400. Furthermore, the acquisition unit 301 can acquire information for identifying the eye to be inspected, such as the subject identification number, from the input unit 700 or the like. The acquisition unit 301 can store various acquired data and images in the storage unit 302.

  Next, the processing unit 303 will be described. The processing unit 303 is provided with an image processing unit 331, a calculation processing unit 322, a shooting condition acquisition unit 333, and a selection unit 334.

  The image processing unit 331 generates a tomographic image, an En-Face image, or the like from the data acquired by the acquisition unit 301 or the data stored in the storage unit 302, and from the generated or acquired images, a plurality of images related to the eye to be described later is generated. Calculate the analysis value for the type. Note that the plurality of types of analysis values may be obtained by being calculated in advance and stored in the storage unit 302 or the external storage device 500.

  The arithmetic processing unit 332 includes a learned model created by giving learning data to an arbitrary machine learning model such as deep learning and learning. Specific contents of learning will be described later. The arithmetic processing unit 332 uses the learned model to capture, for example, tomographic data obtained by imaging the eye to be inspected at the first time and the eye to be inspected at the second time after the first time. A tomographic image captured at a third time after the second time is generated from the tomographic data obtained by performing the above. The image processing unit 331 performs analysis processing on the generated tomographic image to calculate a plurality of types of analysis values in the tomographic image. The calculation of the plurality of types of analysis values may be performed by the arithmetic processing unit 332, and may be directly generated from the two tomographic data by using a learned model instead of the calculation.

  The imaging condition acquisition unit 333 acquires, from the storage unit 302, the imaging conditions when the tomographic image to be image-processed was captured. Note that the imaging conditions to be acquired may be the above-described imaging conditions acquired by the acquisition unit 301 at the time of imaging the tomographic image, and are stored in the storage unit 302 or the like in association with the tomographic image previously captured. It may be a shooting condition.

  The selection unit 334 selects an appropriate learned model in accordance with the imaging conditions acquired by the imaging condition acquisition unit 333 or the imaging conditions such as the anterior eye segment and the posterior eye segment in addition to the above-described conditions. A plurality of learned models are generated in advance according to the imaging conditions and stored in the storage unit 302, and the arithmetic processing unit 332 uses the learned model selected by the selection unit 334 according to the imaging conditions to analyze a tomographic image and its analysis. Generate a value.

  The instruction unit 304 can control the driving of each component of the OCT apparatus 200 and the fundus image capturing apparatus 400. The storage unit 302 can store the tomographic data acquired by the acquisition unit 301 and various images and data such as the tomographic image generated and processed by the processing unit 303. The storage unit 302 can also store a program or the like for executing the functions of the respective constituent elements of the image processing apparatus 300 by being executed by the processor.

  The display control unit 305 can control display on the display unit 600 of various information acquired by the acquisition unit 301, a tomographic image generated and processed by the processing unit 303, and information input by a user.

  Each component other than the storage unit 302 of the image processing apparatus 300 may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). In addition, each of the constituent elements may be configured by a circuit or the like that performs a specific function such as an ASIC. The storage unit 302 may be configured by any storage medium such as an optical disk or a memory, for example.

  Next, a series of processes according to this embodiment will be described with reference to FIG. FIG. 3 is a flowchart of a series of processes according to this embodiment. When the series of processes according to this embodiment is started, the process proceeds to step S301.

  In step S301, the acquisition unit 301 acquires a subject identification number, which is an example of information for identifying the eye to be inspected, from the outside of the image processing apparatus 300 using the input unit 700 or the like. The acquisition unit 301 acquires information regarding the subject's eye held by the external storage device 500 based on the subject identification number and stores the information in the storage unit 302.

  In step S <b> 302, the drive control unit 202 controls the OCT apparatus 200 to scan the eye to be inspected with measurement light to perform imaging, and the acquisition unit 301 acquires an interference signal including tomographic information of the eye to be inspected from the OCT apparatus 200. . The scanning with the measurement light is performed by the instruction unit 304 controlling the OCT apparatus 200 and operating the light source 206, the galvanometer mirror 201, or the like in response to a user's instruction to start scanning.

  The galvanometer mirror 201 includes an X scanner for horizontal direction and a Y scanner for vertical direction. The drive control unit 202 can scan the measurement light in each of the horizontal direction (X) and the vertical direction (Y) in the apparatus coordinate system by changing the orientations of these scanners. The drive control unit 202 can also scan the measurement light in a direction in which the horizontal direction and the vertical direction are combined by changing the directions of these scanners at the same time. Therefore, the drive control unit 202 can scan the measurement light in an arbitrary direction on the fundus plane.

  The drive control unit 202 adjusts various shooting parameters when shooting. Specifically, the drive control unit 202 sets at least the position of the pattern displayed by the internal fixation lamp 204, the scan range and scan pattern of the galvano mirror 201, the coherence gate position, and the focus lens position.

  The drive control unit 202 controls the light emitting diode of the display unit 241 to control the position of the pattern displayed by the internal fixation lamp 204 so that, for example, the center of the macula of the eye to be inspected or the optic disc is imaged. Further, the drive control unit 202 sets, as the scanning pattern by the galvano mirror 201, a scanning pattern such as raster scan, radial scan, or cross scan for photographing a three-dimensional volume. It should be noted that no matter which scanning pattern is selected, scanning is repeated a plurality of times on one line, and imaging is performed a plurality of times (the number of repetitions is 2 or more). In the present embodiment, the scanning pattern is a cross scan, and the same portion is repeatedly photographed 150 times. After the adjustment of these imaging parameters is completed, the instruction unit 304 controls the OCT apparatus 200 to perform imaging of the eye to be inspected in response to a user's instruction to start imaging.

  Although not described in this disclosure, the OCT apparatus 200 can perform tracking of the eye to be inspected in order to capture the same portion for arithmetic averaging. As a result, the OCT apparatus 200 can scan the eye to be inspected while reducing the influence of the involuntary eye movement.

  In step S303, the tomographic image generation unit 311 generates a tomographic image based on the interference signal acquired by the acquisition unit 301. The tomographic image generation unit 311 can generate a tomographic image by performing a general reconstruction process on each interference signal.

  First, the tomographic image generation unit 311 removes fixed pattern noise from the interference signal. Fixed pattern noise removal is performed by averaging a plurality of acquired A scan signals to extract fixed pattern noise and subtracting this from the input interference signal. After that, the tomographic image generation unit 311 performs a desired window function process in order to optimize the depth resolution and the dynamic range that have a trade-off relationship when Fourier transform is performed in a finite section. The tomographic image generation unit 311 generates tomographic data by performing a fast Fourier transform (FFT) process on the interference signal subjected to the window function process.

  The tomographic image generation unit 311 obtains each pixel value of the tomographic image based on the generated tomographic data and generates a tomographic image. The method of generating the tomographic image is not limited to this, and any known method may be used.

  In this embodiment, the tomographic image generation unit 311 generates a tomographic image called a so-called intensity image based on the intensity of the interference light. However, the image generated by the tomographic image generation unit 311 is not limited to the intensity image and may be, for example, a motion contrast image. The motion contrast image or the motion contrast data used when generating the motion contrast image is also included in one aspect of the above-described tomographic data. Further, from the motion contrast data, information about the blood vessel such as the blood vessel diameter and the blood vessel density can be obtained, but this information is included in the data regarding the tomographic section described later in this embodiment.

  Blood cells and the like that flow in blood vessels exist as an object that causes fluctuations in the intensity of interference light and the like in the eye to be examined in an extremely short time. By obtaining the motion contrast data based on the movement of such blood cells, the blood vessel can be imaged without using a contrast agent. By integrating the three-dimensional motion contrast data acquired based on the depth information of the eye to be examined in the depth direction and projecting it on a two-dimensional plane, a front blood vessel image called an OCTA image can be generated.

  Here, a specific example will be described when the tomographic image generation unit 311 generates motion contrast data. A plurality of tomographic images obtained by photographing substantially the same positions where the photographing times are continuous with each other, and a plurality of tomographic images having been aligned are acquired. Note that various known methods can be used for alignment. For example, one of the plurality of tomographic images is selected as a reference image, the similarity with other tomographic images is calculated while changing the position and angle of the reference image, and the position of each tomographic image with respect to the reference image is calculated. The shift amount is calculated. Positioning of a plurality of tomographic images is performed by correcting each tomographic image based on the calculation result. Note that the alignment method is not limited to this, and any known method may be used.

The tomographic image generation unit 311 calculates a decorrelation value for each of the two tomographic images of which the photographing times are continuous among the plurality of tomographic images that have been aligned, by using the following mathematical expression 1.

  Here, A (x, z) indicates the amplitude at the position (x, z) of the tomographic image A, and B (x, z) indicates the amplitude at the same position (x, z) of the tomographic image B. The resulting decorrelation value M (x, z) takes a value from 0 to 1, and becomes closer to 1 as the difference between the two amplitude values is larger. Although the present embodiment has described the case of using the two-dimensional tomographic image of the XZ plane, for example, the two-dimensional tomographic image of the YZ plane or the like may be used. In this case, the position (x, z) may be replaced with the position (y, z) or the like. The decorrelation value may be obtained based on the brightness value of the tomographic image, or may be obtained based on the value of the tomographic signal corresponding to the tomographic image.

  The tomographic image generation unit 311 determines the pixel value of the motion contrast image based on the decorrelation value M (x, z) at each position (pixel position), and generates the motion contrast image. In the present embodiment, the tomographic image generation unit 311 calculates the decorrelation value for the tomographic images whose imaging times are continuous with each other, but the method of calculating the motion contrast data is not limited to this. The two tomographic images for which the decorrelation value M is obtained need only have a shooting time within a predetermined time interval with respect to each corresponding tomographic image, and the shooting times do not have to be continuous. Therefore, for example, for the purpose of extracting an object with a small temporal change, two tomographic images having an imaging interval longer than a normal specified time are extracted from the acquired tomographic images to calculate the decorrelation value. May be. Further, instead of the decorrelation value, a variance value, a value obtained by dividing the maximum value by the minimum value (maximum value / minimum value), or the like may be obtained.

  The method of generating the motion contrast image is not limited to the above method, and any other known method may be used. As described above, the motion contrast data includes, for example, a difference, a ratio, or a temporal change in the phase or vector or intensity of the complex OCT signal between a plurality of tomographic images obtained by repeatedly photographing substantially the same portion. It is obtained by calculating from the correlation or the like. In this case, the “substantially the same position” means a position that is the same as that which is acceptable for generating the motion contrast data, and also includes a position that is slightly deviated from the strictly same position.

  In step S304, the image processing unit 331 detects the retinal layer and obtains a plurality of analysis values exemplified by the thickness, volume, curvature, and shape of the retinal layer. First, the detection of the retinal layer will be described. The image processing unit 331 detects the boundary of the retinal layer in the plurality of tomographic images captured by the OCT apparatus 200.

  Here, details of the process of detecting the retinal layer will be described. The image processing unit 331 performs noise removal and edge enhancement processing on the tomographic image to be processed. As the noise removing process, for example, a median filter or a Gaussian filter is applied. A Sobel filter or a Hessian filter is applied as the edge enhancement processing.

An edge enhancement process for a two-dimensional tomographic image using a two-dimensional Hessian filter will be described. The Hessian filter can emphasize the secondary local structure of a two-dimensional grayscale distribution based on the relationship between _two eigenvalues (λ ₁ , λ ₂ ) of the Hessian matrix. The relationship between the eigenvalues of the Hessian matrix and the eigenvectors (e ₁ , e ₂ ) is used to emphasize the two-dimensional line structure. Since the line structure in the two-dimensional tomographic image of the eye to be examined corresponds to the retinal layer, the structure of the retinal layer can be emphasized by applying the Hessian filter.

  Note that in order to detect retinal layers having different thicknesses, the resolution of smoothing by a Gaussian function performed when calculating the Hessian matrix may be changed. When applying the two-dimensional Hessian filter, the data can be applied after transforming the data so that the physical size of XZ of the image is matched. In the case of general OCT, the physical sizes in the XY direction and the Z direction are different. Therefore, the filter is applied by matching the physical size of the retinal layer for each pixel. Since the physical sizes in the XY direction and the Z direction can be grasped from the design / configuration of the OCT apparatus 200, the tomographic image data can be transformed based on the physical size. In addition, when the physical size is not normalized, it can be approximately handled by changing the resolution of smoothing by the Gaussian function.

Although the processing on the two-dimensional tomographic image has been described above, the target to which the Hessian filter is applied is not limited to this. When the data structure when the tomographic image is captured is a three-dimensional tomographic image by raster scanning, it is possible to apply a three-dimensional Hessian filter. In this case, the image processing unit 331 performs the alignment process in the XZ direction between the adjacent tomographic images, and then the cubic function is performed based on the relationship between the _three eigenvalues (λ ₁ , λ ₂ , λ ₃ ) of the Hessian matrix. It is possible to emphasize the secondary local structure of the original gray distribution. Therefore, by emphasizing the three-dimensional layered structure by using the relation between the eigenvalues of the Hessian matrix and the eigenvectors (e ₁ , e ₂ , e ₃ ), it is possible to emphasize the edges in three dimensions. In addition, here, although the configuration in which the edge emphasis processing is performed using the Hessian filter has been described, the processing method of the edge emphasis processing is not limited to this, and any existing method may be used.

  The image processing unit 331 detects the edge-enhanced boundary from the tomographic image subjected to the edge enhancement processing. In this embodiment, the boundary between ILM and NFL, RPE, ISOS, and the boundary between NFL and GCL are detected. Although not shown, as other boundaries, a boundary between the outer reticular layer (OPL) and the outer granular layer (ONL), a boundary between the inner reticular layer (IPL) and the inner granular layer (INL), and a boundary between the INL and the OPL. A boundary, a boundary between GCL and IPL, etc. can also be detected.

  As a boundary detection method, a plurality of locations having strong edge strengths are detected as boundary candidates in each A-scan, and points (edge strength image strong locations) are formed as lines based on the continuity between the boundary candidates in adjacent A-scans. Perform the connecting process. In addition, the image processing unit 331 can remove the outlier that is displaced in the depth direction by evaluating the smoothness of the line when the points are connected as a line. More specifically, for example, the positions of the connected points in the Z direction are compared, and if the difference in the Z direction position is larger than a predetermined threshold value, the newly connected point is determined as an outlier, It can be excluded from the splicing process. When the outliers are removed, the boundary candidates in the A scan adjacent to the A scan position of the excluded point may be connected as a line. The method of removing outliers is not limited to this, and any existing method may be used.

  The image processing unit 331 determines, for each line formed by connecting points, a corresponding boundary based on the vertical distance and the positional relationship in the Z direction of the boundary of the retina layer. If there is no boundary detected as a result of removing the outlier in each A scan, this may be obtained by interpolation from the surrounding boundary. Alternatively, the boundary candidates may be searched in the horizontal direction (X or Y direction) from the peripheral boundary depending on the edge, and the boundary may be determined again based on the boundary candidates searched from the peripheral boundary. .

  After that, the image processing unit 331 executes a process for smoothly correcting the shape of the detected boundary. For example, an active contour model such as the Snakes or Level Set method may be used to smooth the shape of the boundary by using the image feature and the shape feature. Alternatively, the coordinate values of the boundary shape may be regarded as time-series data of a signal, and the shape may be smoothed by a smoothing process such as a Savitzky-Golay filter, a simple moving average, a weighted moving average, or an exponential moving average.

  The detection of the retinal layer may be performed using a dedicated learned model. In that case, a learned model for layer detection in which a label image created as teacher data and a tomographic image used to create the label image as an input image are learned in pairs is created in advance. The label image refers to an image in which a label for an image appearing (captured) in each pixel is given to each pixel in the tomographic image. For example, a label on the shallower layer side (vitreous body side) than the retina, a label on the inner layer of the retina, and a label on the deeper layer side (choroid side) than the retina can be exemplified as the given labels.

  Next, the image processing unit 331 performs measurement processing for obtaining a plurality of analysis values. An example of the analysis value is the thickness of the retina defined by two arbitrary boundaries obtained by the boundary detection. For example, in glaucoma, the retina becomes thin, and the layer thickness tends to be asymmetrical in the upper and lower parts. When the eye to be examined has an abnormality accompanied by bleeding such as molecular retinal vein occlusion, the entire retina becomes thick due to bleeding. Therefore, by measuring the thickness of the retina and obtaining an analysis value thereof, it is possible to know the feature of the layer thickness change that appears due to a disease.

  In addition, the layer thickness in the retina has different thickness standards depending on whether the measurement site is the macula or the optic papilla. Therefore, it is preferable to measure the thickness together with the position information of the measurement site. When the imaging site is the optic papilla, the C / D ratio, which is the ratio of Cup to Disc, may be measured. In addition, the shape feature of an arbitrary boundary obtained by the boundary detection may be measured. For example, in an intense myopic eye, the retina is deformed to be stretched backward (downward). Moreover, in age-related macular degeneration, the RPE is deformed into a shape protruding upward. Therefore, as a shape feature, it is advisable to measure the local curvature as an analysis value by using the boundary such as RPE or Bruch's membrane. From such an analysis value, it is possible to grasp the characteristic of the shape that appears due to the disease.

  As described above, the disease causes characteristics in the layer thickness and the shape of the retina. Moreover, not only one of these features may appear in one eye, but multiple features may appear. For example, as a certain disease progresses thinning of a specific layer, a change in shape such as the extreme of the bay also progresses, and due to the progress of the medical condition, the thinning may become rapid due to the influence of curvature. It can happen. In such a case, if only the progress of thinning is assumed from the analysis value, a rapid change cannot be expected. Therefore, it is advisable to obtain a plurality of types of assumed analytical values such as layer thickness and bay extremes.

  Further, as described above, the image processing unit 331 may generate the motion contrast data. In this case, the image processing unit 331 obtains the analysis value using the OCTA image which is the front image of the motion contrast image obtained by performing the average value projection (AIP) or the maximum value projection (MIP) of the motion contrast data. May be. For example, features such as the density of the portion corresponding to the blood vessel and the shape (length or shape) of the blood vessel may be obtained from the image obtained by binarizing the OCTA image or thinning the binarized image. These characteristics can also be used as analysis values.

  In step S304, when the analysis value measurement process is performed by the image processing unit 331, the process proceeds to step S305. In step S305, the detected boundary and tomographic image are displayed on the display unit 600. Here, the process of displaying a screen on the display unit 600 will be described with reference to FIG. 3B and FIGS. 4 to 6. Note that FIG. 3B is a flowchart of a series of processes executed in the display process of step S305. When the display process according to the present embodiment is started, the process proceeds to step S351.

  In step S351, the imaging condition acquisition unit 333 acquires from the storage unit 302 the imaging conditions used when the interference signal was acquired in step S302. Further, the imaging condition acquisition unit 333 also acquires the eye information acquired in step S301 from the storage unit 302.

  In step S352, the selection unit 334 determines whether or not the storage unit 302 or the external storage device 500 stores information obtained by imaging the same eye in the same scan pattern in the past from the information acquired in step S351. Is determined by. If there is data captured in the same scan pattern in the past, the process proceeds to step S353. If there is no such data, the process proceeds to step S356.

  The flow may be divided according to the number of past examinations as well as the presence or absence of photographed data. For example, when the number of data captured in the past is only 1, there is only 2 data including the current data, so there is little information to use the learned model described later. Therefore, a threshold value Th (for example, Th> 2) may be set as the number of past examinations, and the process may move to step S353 when the number of past examinations is a certain number or more.

  In step S353, the learned model used by the selection unit 334 is selected based on the shooting conditions acquired by the shooting condition acquisition unit 333. The imaging conditions related to the imaging region include, for example, whether the imaging region is the anterior segment or the posterior segment of the subject's eye. Since the analysis values output from the image data are different between the anterior segment and the posterior segment, separate learned models are obtained in advance. In addition, in the present embodiment, the example in which the learned model to be used is selected depending on which of the anterior segment and the posterior segment is imaged has been described. However, the imaged region to be referred to is not limited to this, and the learned model may be selected also in the posterior segment of the eye according to the macular region or the optic papilla.

  Note that, for example, when there is a learned model of the posterior segment of the eye and no learned model of the anterior segment exists, and the input data is an image of the anterior segment of the eye, the flow proceeds to step S356 at the time of step S352. You may do it. By such processing, even if the learned model does not correspond at the beginning, it is possible to add a learned model that can be supported by updating the software so that it can be supported even when the anterior segment of the eye is photographed. Become.

  In step S354, the arithmetic processing unit 332 uses the learned model to generate a tomographic image that is likely to be obtained when the subject eye is imaged under the same imaging condition in the future. In addition, the image processing unit 331 uses the generated tomographic image to calculate the above-described plurality of types of analysis values. In the present embodiment, the arithmetic processing unit 332 uses the learned model to generate a tomographic image. However, the arithmetic processing unit 332 may directly generate each analysis value that is likely to be obtained in the future from the plurality of types of analysis values obtained by the image processing unit 331 described above.

  Note that, as described above, the learned model in the description of the present invention is a model trained in advance using appropriate learning data for a machine learning model according to an arbitrary machine learning algorithm such as deep learning. . Here, the learning data used for training when obtaining the learned model according to the present embodiment will be described. The learning data is composed of a pair of input data corresponding to the data actually input to the learned model and teacher data corresponding to the data generated by the learned model. In the present invention, as input data, at least two times of the first time and the second time after the first time, two tomographic images obtained by imaging the same eye under the same imaging conditions, Alternatively, two inspection results composed of plural kinds of analysis values obtained from the tomographic image are used. Further, as teacher data, it is composed of a tomographic image obtained by photographing the same eye under the same photographing condition at a third time after the second time or a plurality of types of analysis values obtained from the tomographic image. Use the test results.

  When obtaining the learned model, learning is performed using the tomographic images and the inspection results previously obtained at at least three past times as the learning data, as described here. When this learned model is used, it is obtained at the same time interval as the time interval between the first time and the second time in the past corresponding to the input data used when the learned model is generated. Enter the two test results. The teacher data corresponds to the analysis value and the like generated from the learned model. That is, by using the learned model, in the several months to several years ahead corresponding to the time interval between the second time and the third time described above, the analysis value that is highly likely to be obtained from this eye is calculated. Obtainable.

  Next, a machine learning algorithm using a plurality of types of analysis values will be described with reference to FIG. Here, the learning data used for training when obtaining the learned model will be described. As described above, the learning data includes numerical value data obtained from the same eye to be inspected at different times in the past, for example, including numerical values of a plurality of types of analysis values. An example of the learning data is shown in FIG. FIG. 4 shows an example of analysis values (numerical values) of N measurement parameters (Params1 to ParamsN) on the time series (date T). Here, for example, when the imaging region is a posterior segment image, the plurality of types of analysis values include the thickness of the entire retina, the thickness between arbitrary layers, the thickness of the choroid, and the shape characteristics of the retina (such as RPE and Bruch's membrane). Curvature) etc. are included. When the imaging site is the optic papilla, a plurality of types of analysis values include the C / D ratio, which is the ratio of Cup to Disc, in addition to the thickness of the retina. The C / D ratio may be obtained not from a tomographic image but from a fundus image acquired from the fundus image capturing apparatus 400 or a plane image such as SLO. When the imaged part is the anterior segment, the corneal thickness, the corneal curvature, the anterior chamber depth, the entire lens curvature, the posterior lens curvature, and the like are included in a plurality of types of analysis values.

  FIG. 4 is a time-series graph including a horizontal axis that is a time axis T that indicates the date and time when the image was captured and a vertical axis that is an analysis value of a specific measurement parameter. The numerical value of the analysis value may be used as it is as the learning data. However, for easier handling, a difference from the analysis value of the previous time may be obtained and used as the learning data. It should be noted that an example in which two data on a time series are used as input data used when generating the learned model described above and two data corresponding to this are also used as data input to the learned model is described above. . However, the number of input data and the number of input data is not limited to two, and a larger number of data may be used. The time required for learning becomes long by using a lot of data on the time series when the trained model is generated. However, it becomes possible to match the interval of the input data with the interval of the arbitrary data on the time series, and the versatility may be enhanced.

  Further, at the time of learning, information regarding the degree of progress of the disease such as whether the disease has progressed, has not progressed, or has stopped may be added to the learning data as the evaluation value by totalizing the data including the plurality of analysis values. In this case, the degree of disease progression is defined as a numerical value. For example, a doctor or the like determines time-series data of analysis values or a tomographic image to determine a medical condition, and an evaluation value of 100 when the disease progresses, an evaluation value of 50 when the disease has not changed, and an improvement when the disease has improved. The evaluation value is 0 or the like. It should be noted that this numerical value is merely an example, and the numerical value is not limited to this value, and the evaluation value is appropriately determined according to the type of disease and the item to be judged. Although the evaluation value is treated as a numerical value here as an example, the quality may be ranked, for example, with respect to the disease progression state or the eye state. That is, the evaluation value described here is an example of the evaluation information that the doctor or the like evaluates and determines the eye condition and inputs as data.

  Next, as an example of the machine learning model according to the present embodiment, a Recurrent Natural Network (hereinafter, RNN) that is a neural network that handles time series information will be described with reference to FIG. In addition, a long short-term memory (hereinafter, LSTM), which is a type of RNN and used by the arithmetic processing unit 332, will be described with reference to FIG. 6.

FIG. 5A shows the structure of the RNN, which is a machine learning model in the arithmetic processing unit 332. The RNN 520 shown in FIG. 5 has a loop structure in the network, inputs data x ^t 510 at time t, and outputs data h ^t 530. Since the RNN 520 has a loop function in the network, it is possible to take over the state at the current time to the next state, so that the time series information can be handled. FIG. 5B shows an example of input / output of the parameter vector at time t. The data x ^t 510 includes analysis values regarding N (Params 1 to Params N) measurement parameters. The data h ^t 530 output from the RNN 520 includes N (Params 1 to Params N) analysis values corresponding to the input data.

However, the RNN cannot handle long-term information during error back propagation. Therefore, the LSTM is used in this embodiment. The LSTM can learn long-term information by including a forget gate, an input gate, and an output gate. Here, the structure of the LSTM is shown in FIG. In the LSTM 620, the information that the network takes over at the next time t is the internal state ct ^{-1 of the} network called a cell and the output data ht ^-1 . The lower case letters (c, h, x) in the figure represent vectors. In this embodiment, this vector is composed of the analysis values shown in FIG.

  Next, FIG. 6B shows details of the LSTM 620. In FIG. 6B, FG is a forgetting gate network, IG is an input gate network, OG is an output gate network, and each is a sigmoid layer. Therefore, a vector in which each element has a value of 0 to 1 is output. The forgetting gate network FG determines how much past information is retained, and the input gate network IG determines which value is updated. The CU is a cell update candidate network and is an activation function tanh layer. This creates a new vector of candidate values that will be added to the cell. The output gate network OG selects a cell candidate element and selects how much information is transmitted at the next time.

  The above-described LSTM model is a basic form, and is not limited to the network shown here. The connection between networks may be changed. QRNN (Quasi Current Neural Network) may be used instead of the LSTM. Further, the machine learning model is not limited to the neural network, and a machine learning model such as boosting or support vector machine may be used. Further, in the RNN and the LSTM illustrated here, the case where the numerical value illustrated as the analysis value is input as the data is described. However, as described above, the RNN or the LSTM may be configured to input the tomographic data such as the tomographic image and output the tomographic data.

  When data is input to the learned model, the data according to the design of the learned model is output. For example, output data that is highly likely to correspond to the input data is output according to the training tendency from the learning data. For example, when a tomographic image captured by OCT is input to a learned model trained with the above-described learning data, a tomographic image predicted to be highly likely to be captured in the future is output. If a learned model that uses analysis values instead of tomographic images is used, if an analysis value obtained from a tomographic image captured by OCT is input to the learned model, a tomographic image captured in the future will be used. The analysis value predicted to be measured is output.

In the learned model of the arithmetic processing unit 332 according to the present embodiment, when the tomographic image is input as the data x ^t 510, the tomographic image is output as the data h ^t 530 according to the tendency learned using the learning data. Further, when the analysis values of a plurality of measurement parameters are input as the data x ^t 510, a plurality of analysis values of the corresponding measurement parameters are output as the data h ^t 530 according to the tendency learned using the learning data.

  In step S355 or S356, the display unit 600 displays the analysis result according to the instruction from the display control unit 305. It should be noted that in step S355 and step S356, only the result generated by the arithmetic processing unit 332 is displayed or the analysis value obtained in step S351 is displayed as it is, and the other display information is the same. Is. Therefore, the screens displayed on the display unit 600 will be collectively described below.

  FIG. 7 shows an example of a screen displayed on the display unit 600. On the display screen 710, a patient tab 701, an imaging tab 702, a report tab 703, and a setting tab 704 are displayed. On the left side of the display screen, a tomographic image group obtained by photographing the same eye to be inspected in the past is displayed as a thumbnail image 705. It should be noted that although a diagonal line is added to the report tab 703 in the figure, this diagonal line indicates that the report screen is in an active state. Hereinafter, a case of displaying a report screen in the present embodiment will be described. It should be noted that the diagonal lines are used here to indicate the active state, but the coloring of the display area can also indicate the active state, for example, by changing the color of the display character.

  On the report screen, a tomographic image 711 obtained by scanning the eye to be examined in the horizontal direction and a tomographic image 712 obtained by scanning in the vertical direction are shown. In addition, an SLO image 713 that is a front image of the fundus is displayed on the report screen. On this SLO image 713, for example, a color map 714 expressing the thickness of the retina is superimposed. Further, an ETDRS grid 715 that displays the retinal thickness of each region of the sector that is superimposed and displayed on the SLO image 713, and a table 716 that represents the retinal thickness in the ETDRS grid are also displayed.

  The graph 720 is obtained by arranging the thickness of the retina calculated from the tomographic images 711 and 712 in time series. If there is an estimated analysis value for the thickness of the retina generated by the arithmetic processing unit 332, it is displayed here. The selector 721 is used to select the measurement parameter for which the analysis value shown in the graph 720 is obtained. Here, Params1 having the thickness of the retina as a measurement parameter is selected.

  In the graph 720, a plurality of black circles 722 indicate time series analysis values of retinal thickness obtained from tomographic images captured in the past, and white circles 723 indicate latest analysis values. Furthermore, an asterisk 724 indicates a predicted value of the future retinal thickness generated by the arithmetic processing unit 332. The probability 725 corresponds to the evaluation value input by the above-mentioned doctor or the like. When the evaluation value is input together with the past analysis value, the arithmetic processing unit 332 generates this evaluation value, and this is displayed as the probability 725 on the report screen.

  In the present embodiment, the analysis values of the plurality of measurement parameters are input, and the arithmetic processing unit 332 outputs the corresponding predicted progress value as the analysis value, and the result is displayed as the graph 720. Therefore, the time-series analysis value displayed in the graph 720 changes by selecting the measurement parameter displayed by the selector 721, and the future predicted value indicated by the star 724 is also changed accordingly. However, the probability 725 is generated based on the evaluation value comprehensively input by the doctor or the like using the entire parameters, and therefore the result does not change even if the measurement parameter displayed by the selector 721 is changed.

  Further, here, an example is shown in which the progress prediction value of the probability 725 is simply displayed as a numerical value. However, as another display example, if the symptom improves, it is blue or green, if there is not much change, it is black, if the symptom is likely to worsen, orange or red. You may change and display the color of a numerical value so that it may be easy. Further, instead of displaying as the probability 752, each point displayed in the graph 720 may be displayed in the above-described color to show the progress state of the symptom to the examiner. Further, here, the doctor or the like evaluates the eye to be inspected and numerically inputs the evaluation result as a probability, but the evaluation result is not limited to a numerical value, and may be ranked, for example. That is, the target displayed here may be evaluation information obtained by a doctor or the like evaluating the state of the eye to be inspected.

  In step S355, the report screen including the graph 720 including the result generated by the arithmetic processing unit 332 and the probability 725 is displayed on the display screen 710. After the display, the flow moves to step S306. In step S356, the arithmetic processing unit 332 cannot generate an analysis value or the like, so the result obtained in step S351 is displayed on the display unit 600 as it is. In this case, the star 724 is not displayed on the graph 720, the probability 725 is not displayed, or a blank is displayed. After the display, the flow moves to step S306. In addition, in the display in step S336, a doctor or the like can input the evaluation value as the probability 725 via the input unit 700.

  As described above, in the present embodiment, the form in which the arithmetic processing unit 332 generates a future tomographic image and an analysis value in the process of displaying the report screen has been described. However, the timing at which the arithmetic processing unit 332 generates a future tomographic image or analysis value is not limited to this. At the stage where the measurement process in step S304 is completed, if there is a past examination, a tomographic image or an analysis value may be generated and stored in the external storage device 500. In this case, in the display process of step S305, the stored tomographic image or analysis value may be read and displayed.

  In step S306, the acquisition unit 301 externally acquires an instruction as to whether or not to end a series of processes related to imaging of a tomographic image by the image processing system 1. This instruction can be input by the user using the input unit 700. When the instruction to end the process is acquired, the image processing system 1 ends the series of processes according to the present embodiment. On the other hand, when the acquisition unit 301 acquires an instruction not to end the process, the flow is returned to step S302 to continue shooting.

  As described above, the image processing apparatus 300 according to the present embodiment includes the acquisition unit (acquisition unit 301) and the generation unit (arithmetic processing unit 332). The acquisition means, data relating to the first tomography of the eye to be examined associated with the first time, and data relating to the second tomography of the eye to be examined associated with the second time after the first time, get. The data regarding the first tomographic data is at least one of a tomographic image obtained by photographing the eye to be inspected at the first time and a plurality of types of analysis values (first analysis values) obtained from the tomographic image. Including. The data regarding the second tomographic image is at least one of a tomographic image obtained by photographing the eye to be inspected at the second time and a plurality of types of analysis values (second analysis values) obtained from the tomographic image. Including one.

  The arithmetic processing unit 332 uses the learned model to extract the eye to be inspected at the third time after the second time from the data regarding the first tomography and the data regarding the second tomography acquired by the acquisition unit 301. At least one of a plurality of types of analysis values of is generated. At this time, the data regarding the fault includes a plurality of types of analysis values. The data regarding the fault includes not only the case where a plurality of types of analysis values are handled as one data set, but also the case where a plurality of types of analysis values are separately handled. Further, at least one analysis value can also be calculated / generated from a tomographic image of the eye to be inspected (data relating to the third tomographic image) generated using the learned model. That is, the data regarding the tomographic image may be a tomographic image or tomographic data. In this case, the tomographic image associated with the first time and the tomographic image associated with the second time described above are acquired, the tomographic image is generated from these tomographic images using the learned model, and the tomographic image is generated. A specific analysis value will be obtained from the tomographic image. When comparing tomographic images of the same eye to be inspected at different times, the characteristic layer may be thinned due to the progress of disease, for example, and at the same time, the curvature may be changed in the layer or the surrounding layers. That is, by using the tomographic images themselves acquired at different times from the same eye as the data to be input to the learned model, future tomographic images in which changes in multiple types of analysis values for the eye to be examined are collectively considered. Will be obtained. As described above, the present invention aims to provide a diagnostic aid using the complex features of the eye, and is data regarding a tomographic image including a plurality of types of analysis values of an eye to be examined, and two data associated with different times. A specific analysis value is to be obtained from the data using a trained model. Therefore, obtaining a specific analysis value by analyzing the tomographic images generated by using the learned model from the two tomographic images obtained by photographing the eye to be inspected at different times is related to the different times described above. The same effect can be obtained by obtaining a specific analysis value from data on two different faults. Here, a tomographic image associated with the first time and a tomographic image associated with the second time are acquired, a tomographic image is generated from these tomographic images using the learned model, and the tomographic image is generated. An example of obtaining a specific analysis value from a tomographic image was described. However, instead of using a learning model that generates a tomographic image from two tomographic images, a learned model that directly generates an analysis value from two tomographic images may be used. In this case, obtaining a tomographic image associated with the first time and a tomographic image associated with the second time, and obtaining a specific analysis value directly from these tomographic images using a trained model. Becomes

  The learned model according to the present embodiment is obtained using learning data including input data and teacher data. The input data includes, for example, a tomographic image obtained by photographing the eye to be inspected at a fourth time before the first time and a fifth time before the first time and after the fourth time. And a tomographic image obtained by photographing the eye to be inspected. Further, as the teacher data, a tomographic image obtained by photographing the eye to be inspected at a sixth time before the first time and after the fifth time is used.

  The data input to the learned model in the arithmetic processing unit 332 may be each of a plurality of types of analysis values obtained from the tomographic image of the subject's eye instead of the tomographic image. In this case, one of the input data in the learned model is a plurality of types of fourth analysis values obtained from the tomographic image obtained by photographing the subject's eye at the fourth time before the first time. Further, another input data is a plurality of types of fifth analysis obtained from a tomographic image obtained by photographing the eye to be inspected at a fifth time before the first time and after the fourth time. It becomes a value. Then, the teacher data becomes a plurality of types of sixth analysis values obtained from the tomographic image obtained by imaging the subject's eye at the sixth time before the first time and after the fifth time. Learning by using these input data and teacher data, the learned model described above is obtained.

  In the data input to the learned model described above and the input data used to obtain the learned model, the elapsed time between the second time and the third time, the fifth time, and the Elapsed time between six times is almost equal. In this case, the elapsed times do not have to be exactly the same, but the amount of difference that the user can tolerate can be appropriately determined depending on, for example, the medical condition or abnormality of the eye to be examined.

  Further, the acquisition unit 301 of the image processing apparatus 300 according to the present embodiment can acquire the above-described evaluation information regarding the state of the eye to be inspected at the first time and the second time. In this case, the arithmetic processing unit 332 can generate the evaluation information at the third time in combination with at least one of the plurality of types of analysis values at the third time. Further, the display control means (display control unit 305) can display the generated evaluation information on the display unit 600.

  Here, consider a case where the data regarding the first slice and the data regarding the second slice are tomographic images (tomographic data). In this case, the data relating to these faults provides a first analysis value and a second analysis value each associated with a first time. The display control unit 305 generates the graph 720 using the first and second analysis values of the same type as the at least one analysis value at the third time and the at least one analysis value at the third time. The display control unit 305 causes the display unit 600 to display the generated graph 720.

  Further, the image processing apparatus 300 according to the present embodiment selects the learned model based on the acquisition condition for acquiring the data regarding the first slice and the data regarding the second slice from the eye to be examined (selecting unit 334). Is further provided. In this case, the acquisition condition includes, for example, at least one of a shooting date and time, a shooting region name, a shooting region, a shooting angle of view, and a shooting method.

  According to this example, the eye to be inspected was imaged in the future using the learned model from the tomographic data obtained by imaging the eye to be examined at least two different times and the analysis values obtained from the tomographic data. It is possible to obtain fault data or analysis values that are likely to be obtained in some cases. Therefore, when diagnosing the eye to be inspected, it is possible to present information effective for assisting diagnosis for follow-up observation, treatment policy, and the like.

(Example 2)
In the first embodiment, an example of generating a plurality of analysis values for each measurement parameter of the eye to be examined using the tomographic image of the eye to be examined and various analysis values obtained from the tomographic image and the fundus plane image is shown. . However, the numerical values relating to the eye to be used when inspecting the eye are not limited to the analysis values and the like measured from the tomographic image described above. In addition to these, it is possible to know the transition of the condition such as the medical condition of the eye to be inspected by the measurement value measured by the eye to be inspected. In the second embodiment, a learned model that can be input including the results of other examinations (for example, axial length and intraocular pressure) is used, whereby various analysis values and measured values, and further the evaluation values described above are used. To generate.

  Hereinafter, the image processing system 8 according to the present embodiment will be described with reference to FIG. In the following description, differences from the image processing system 1 according to the first embodiment will be mainly described. It should be noted that the configurations and processes of the image processing system according to the present embodiment that are the same as the configurations and processes of the image processing system 1 according to the first embodiment are denoted by the same reference numerals, and description thereof will be omitted here. .

  FIG. 8 shows an example of a schematic configuration of the image processing system 8 according to this embodiment. In the image processing system 8 according to the present embodiment, in the processing unit 803 of the image processing apparatus 800, the arithmetic processing unit 832 according to the present embodiment, which is replaced with the arithmetic processing unit 332, and the newly provided measurement value acquisition unit 835. Are arranged.

  The measurement value acquisition unit 835 acquires other inspection information regarding the eye to be inspected. Specifically, the measurement value acquisition unit 835 may acquire the measurement value input by the user using, for example, a user interface. Further, the numerical data registered in the electronic medical chart system may be acquired by cooperating with the electronic medical chart system via a network. Alternatively, the information may be acquired from an external reader such as a barcode reader or an RFID reader.

  In this embodiment, the contents of the input / output parameter vectors (c, h, x) shown in FIGS. 5 and 6 of the first embodiment are different. That is, the data input / output to / from the arithmetic processing unit 832 is different. In the first embodiment, a tomographic image or an analytical value obtained from the tomographic image is used as Params in the contents of the vector used during learning and generation of the analytical value. On the other hand, in the present embodiment, not only the tomographic image or the analysis value obtained from the tomographic image, but also the input / output parameter vector of the inspection data regarding the eye part such as the axial length test and the intraocular pressure test that does not use the image. To generate analysis values.

  For example, in Example 1, an example in which N analysis values are used as Params1 to ParamsN is shown. On the other hand, in the present embodiment, if the number of measurement values acquired by the measurement value acquisition unit 835 is M, learning is performed using the analysis values of Params1 to ParamsN and the measurement values of Params1 'to ParamsM. The arithmetic processing unit 832 acquires learning data from these N analysis values and M measurement values, and uses the learned model obtained by using these learning data.

  The arithmetic processing unit 832 according to the present embodiment includes a learned model created by giving learning data to an arbitrary machine learning model such as deep learning. The arithmetic processing unit 832 sets the tomographic data and the measurement value obtained by imaging the eye to be inspected at the first time and the eye to be inspected at the second time after the first time for the learned model. The tomographic data and the measurement value obtained by photographing are input. The arithmetic processing unit 832 generates a tomographic image and a measured value from the tomographic data and the measured value using the learned model. The image processing unit 331 performs analysis processing on the generated tomographic image to calculate a plurality of types of analysis values in the tomographic image. Note that the calculation of the plurality of types of analysis values may be performed by the arithmetic processing unit 832 using the above-described edge enhancement processing or the like, and may be directly generated from the analysis values by using learning model completion instead of calculation.

  Here, the learning data used for training when obtaining the learned model in the present embodiment will be described. The learning data is composed of input data corresponding to the data actually input to the learned model and teacher data corresponding to the data generated by the learned model. In the first embodiment, as input data, two tomographic images obtained by photographing the same eye under the same photographing condition at least two times of the first time and the second time after the first time. Alternatively, two inspection results including a plurality of types of analysis values obtained from the tomographic image are used. In the present embodiment, the measurement value obtained by measuring the same eye to be examined at the first time and the measurement value obtained by measuring at the second time are also used as input data. Further, as teacher data, it is composed of a tomographic image obtained by photographing the same eye under the same photographing condition at a third time after the second time or a plurality of types of analysis values obtained from the tomographic image. The inspection result and the third measurement value obtained by measuring at the third time are used.

  When obtaining a learned model, learning is performed using the inspection results and measurement results obtained in advance at least at three points in the past, as learning data, as described here. Then, when using this learned model, two inspection results obtained at the time interval between the first time and the second time in the past corresponding to the input data used when the learned model is generated, and Enter the measurement result. The teacher data corresponds to analysis values and the like and measurement results generated from the learned model. That is, by using the learned model, in the several months to several years ahead corresponding to the time interval between the second time and the third time described above, with an analysis value that is likely to be obtained from this eye, The measured value can be obtained.

  It should be noted that, here, the acquisition of the tomographic image and the acquisition of the measurement value are said to be the same for the first time, for example. However, for example, when the subject eye such as glaucoma in which the lesion gradually progresses is targeted, the acquisition of the tomographic image and the measurement value may not be exactly the same time. For example, even if there is an interval of several days, it may be acceptable in view of the transition of the medical condition.

  For example, in an examination for glaucoma, in addition to the information obtained from the analysis value from the tomographic image, the intraocular pressure value, the axial length of the intense myopic eye, and the like are effective. According to the present embodiment, not only the analysis value obtained from the tomographic image but also the measurement value for the other type of measurement parameter regarding the eye to be examined is used to generate the analysis value or the measurement value for the future eye to be examined. be able to. Further, the doctor or the like determines the above-described evaluation value by also referring to these measured values, and the arithmetic processing unit 832 uses the learned model to generate an evaluation value expected in the future. Therefore, in the case of glaucoma as an example, by referring to the measured value that can more appropriately determine the symptom, it is possible to provide more accurate diagnosis assistance data when making a diagnosis by a doctor or the like.

  As the measured values, the axial length and the intraocular pressure are exemplified here. However, the measurement parameters for obtaining the measurement values used in this embodiment are not limited to these, and visual field data, visual acuity data, and the like can be used. Further, here, the example in which both the analysis value obtained from the tomographic image and the measurement value obtained by means other than the analysis of the tomographic image are used as the input data have been described. However, the input data or the input data is not limited to this. For example, the arithmetic processing unit 832 may generate a measurement value that can be measured in the future using only the measurement value without using the analysis value obtained from the tomographic image.

  As described above, the acquisition unit 301 according to the present embodiment, in addition to the analysis value described in the first embodiment, the first measurement value of the eye to be examined associated with the first time, and the second time The second measurement value of the eye to be inspected is further acquired. The arithmetic processing unit 832 determines at least one of a plurality of types of analysis values at the third time from the data regarding the first fault, the first measurement value, the data regarding the second fault, and the second measurement value. Generate one. The arithmetic processing unit 832 may generate at least one measurement value. Further, as in the first embodiment, the evaluation information may be added to the data input to the learned model, and the arithmetic processing unit 832 may also generate the evaluation information. In this case, only the evaluation information can be generated and displayed.

  Alternatively, the acquisition unit 301 according to the present embodiment acquires the first measurement value of the eye to be examined associated with the first time and the second measurement value of the eye to be examined associated with the second time. It can also be an aspect. In this case, the arithmetic processing unit 832 obtains a learned model that uses only the measured values in advance, and inputs the first measured value and the second measured value into the learned model, thereby obtaining a plurality of types in the third time period. It is also possible to generate at least one of the measured values of. In this case, the evaluation information can be further included as input data and generated data. Further, although the case where the tomographic data (tomographic image) is used as the input data or the input data for generating the learned model is described here, a plurality of analysis values can be used.

  According to the present embodiment, not only the tomographic data and the analysis value thereof used in the first embodiment but also other measured values of the eye to be inspected such as the axial length and the intraocular pressure are used, and the eye to be examined in the future by the learned model It is possible to obtain tomographic data that is highly likely to be obtained, its analysis value, and the measurement value. Therefore, in diagnosing the eye to be inspected, it is possible to present more effective information as a diagnostic aid for follow-up observation, treatment policy, and the like.

(Example 3)
In Example 1 and Example 2, diagnosis of the eye to be examined is performed using the tomographic image obtained by imaging the eye to be examined, the analysis value obtained from the tomographic image, and the measurement value obtained by measuring the eye to be examined. An example of presenting information to assist the above was shown. However, when there is a lesion or abnormality in the eye to be inspected, it is rare for a doctor or the like to leave it as it is and observe the progress, and treatment such as medication and surgery is given to the eye to be inspected depending on the situation. In the third embodiment, the learned model is used including the presence / absence of treatment, and various analysis values are generated with reference to the treatment. In addition, for example, for the purpose of confirming the effect of the treatment, an evaluation value relating to a future state of the eye to be examined such as a medical condition is generated.

  The image processing system 9 according to the present embodiment and the report screen displayed on the display unit will be described below with reference to FIGS. 9 and 10. In the following description, differences from the image processing systems according to the first and second embodiments will be mainly described. It should be noted that the configuration and processing of the image processing system 9 according to the present embodiment, which are similar to the configurations and processing of the image processing systems 1 and 8 according to the first and second embodiments, are denoted by the same reference numerals, The description here is omitted.

  FIG. 9 shows an example of a schematic configuration of the image processing system 9 according to the present embodiment. In the image processing system 9 according to the present embodiment, in the processing unit 903 of the image processing apparatus 900, the arithmetic processing unit 932 according to the present embodiment, which is replaced with the arithmetic processing unit 332, and the newly provided treatment information acquisition unit 935. Are arranged.

  The treatment information acquisition unit 935 acquires information regarding treatment of the eye to be inspected. Specifically, the treatment information acquisition unit 935 may acquire the measurement value input by the user through, for example, the user interface. Further, the numerical data registered in the electronic medical chart system may be acquired by cooperating with the electronic medical chart system via a network. Alternatively, the information may be acquired from an external reader such as a barcode reader or an RFID reader.

  In this embodiment, the contents of the input / output parameter vectors (c, h, x) shown in FIGS. 5 and 6 of the first embodiment are different. That is, the data input / output to / from the arithmetic processing unit 932 is different. In the first embodiment, a tomographic image or an analytical value obtained from the tomographic image is used as Params in the contents of the vector used during learning and generation of the analytical value. On the other hand, in the present embodiment, not only the tomographic image or the analysis value obtained from the tomographic image but also the evaluation value regarding the state of the eye to be inspected defined by the doctor and the treatment method should be added to the input / output parameter vector. Then, the analysis value is generated.

  For example, in Example 1, an example in which N analysis values are used as Params1 to ParamsN is shown. On the other hand, in the present embodiment, if the information acquired by the treatment information acquisition unit 935 is the information S related to the content of the treatment such as the surgical technique and the type of medication, the analysis values of Params1 to ParamsN and ParamsS are used. You will be learning. The arithmetic processing unit 932 acquires learning data from these N analysis values and the information S regarding treatment, and uses a learned model obtained by using these learning data.

  Next, with reference to FIG. 10, an example of a screen displayed on the display unit 600 according to the present embodiment is shown. In this embodiment, the display screen 1010 displays the second selector 1021 together with the selector 721. Here, in the time-series graph 1020 for Params1, an example is shown in which the treatment 1 selected by the second selector 1021 at the present time is performed on the eye to be inspected. The graph 720 shows that when the treatment 1 is applied to the subject's eye at the time indicated by the white circle 1023, the predicted value of the retinal thickness in the future becomes an asterisk 1024.

  In the present embodiment, for example, when performing a certain treatment (for example, medication) on the eye to be examined, it is possible to obtain a future analysis value of the selected measurement parameter based on the analysis value obtained by imaging. it can. Further, the type of treatment can be selected by switching the second selector 1021, and the analysis value obtained as an asterisk 1024 and the probability 1025 which is a predicted progress value change depending on the selected treatment.

  In the present embodiment, the arithmetic processing unit uses the learned model to obtain the tomographic data obtained by imaging the eye to be inspected at the first time and the eye to be inspected at the second time after the first time. A tomographic image captured at a third time after the second time is generated from the tomographic data obtained by the imaging. At that time, when the subject's eye is treated at at least one of the first time and the second time, this treatment is also input to the learned model together with the tomographic data. The image processing unit 331 performs analysis processing on the generated tomographic image to calculate a plurality of types of analysis values in the tomographic image. The calculation of the plurality of types of analysis values may be performed by the arithmetic processing unit 332, and may be generated using a learned model instead of calculation.

  Furthermore, the evaluation value of the subject's eye at at least one of the first time and the second time may be input to the learned model. With this, it is possible to numerically grasp what effect the treatment applied to the subject's eye has on the generated evaluation value. The data indicating the presence or absence of the treatment may be input to the learned model as 0 when the treatment is not performed and 1 when the treatment is performed. Furthermore, in this case, the input value may be set to 2 or 3 depending on the type of treatment (for example, medication treatment, surgery, etc.) as well as whether or not the treatment is performed.

  Here, the learning data used for training when obtaining the learned model in the present embodiment will be described. In the first embodiment, as input data, two tomographic images obtained by photographing the same eye under the same photographing condition at least two times of the first time and the second time after the first time. And its evaluation information can be used. Alternatively, instead of two tomographic images, it is also possible to use two inspection results consisting of a plurality of types of analysis values obtained from the tomographic images and their evaluation information. In the present embodiment, the content of the treatment performed on the subject's eye at least at either the first time or the second time is also used as input data. Further, as teacher data, it is composed of a tomographic image obtained by photographing the same eye under the same photographing condition at a third time after the second time or a plurality of types of analysis values obtained from the tomographic image. The inspection result and the evaluation information of the eye to be examined at the third time are used.

  When obtaining the learned model, as described here, learning is performed by using the inspection result and the evaluation information thereof obtained in advance at least at three points in the past as learning data. When using this learned model, two inspection results obtained at the past first time and second time corresponding to the input data used when the learned model is generated, and either Enter the evaluation information of the eye to be examined at the time and information on the treatment. The teacher data corresponds to analysis values and the like generated from the learned model and evaluation information. That is, by using the learned model, in the several months to several years ahead corresponding to the time interval between the second time and the third time described above, with an analysis value that is likely to be obtained from this eye, The evaluation information of the eye to be inspected at that time can be obtained.

  More specifically, for example, when a specific drug is given to the eye to be inspected having glaucoma at the above-mentioned first time, one of the input data is from the tomographic image of the eye to be inspected at that time. The obtained analysis value, evaluation information of the eye to be inspected at that time, and a flag indicating medication are provided. Further, when the medication is not administered in the second time, another input data is an analysis value obtained from the tomographic image of the eye to be inspected at that time and evaluation information of the eye to be inspected at that time. On the other hand, as a result of using the learned model, the analysis value of the tomographic image captured at the third time after the second time and the evaluation information of the eye at that time are generated in the same eye. To be done.

  In this case, as the learning data used to generate the learned model, the data of the eye to be inspected, to which the above-mentioned specific medicine has been given and whose time series analysis values have been obtained in advance, is used. As actual input data, the analysis value at the time of medication and the evaluation information are one of the input data. In addition, the other input data is an analysis value obtained by photographing the eye to be examined at a time interval substantially equal to the time interval between the first time and the second time from the administration, and at that time. It becomes evaluation information. Further, the teacher data is an analysis value obtained by photographing the eye to be examined at a time interval substantially equal to the time interval between the second time and the third time, and evaluation information at that time. Note that this time interval does not have to be exactly the same, and there may be an error of several days to several weeks depending on, for example, the rate of progress of the disease.

  Note that, here, with respect to the learning data used when obtaining the learned model according to the present example and the data to be input to the learned model and the generated result, the case where the medication is performed in the first time is taken as an example. Stated. However, the timing of treatment such as medication is not limited to the example described here, and may be performed at the second time or both the first time and the second time. In this case, by obtaining the corresponding learned model in advance, it is possible to obtain the analysis value and the evaluation information that are likely to be obtained as a result of the treatment.

  As described above, the treatment information acquisition unit 935 according to the present embodiment acquires information regarding the treatment performed on at least one of the first time and the second time for the eye to be inspected. The treatment in this case is associated with the data regarding the first slice and / or the data regarding the second slice. In the present embodiment, the arithmetic processing unit 932 determines a plurality of types at the third time from the data regarding the first slice, the data regarding the second slice, and the information regarding the treatment associated with at least one of these data. Generate at least one of the parsed values. The data relating to the tomography described here may be the above-described plurality of analysis values obtained from the tomographic image, or may be tomographic data (tomographic image).

  The image processing apparatus 900 according to the present embodiment is not limited to the mode in which the tomographic image and the analysis value thereof described in the first embodiment and the measurement value described in the second embodiment are input to the arithmetic processing unit. For example, the aspect may be such that evaluation information about the eye to be examined, at least one piece of eye information indicating the state of the eye to be examined, and information regarding treatment of the eye to be examined are input to the arithmetic processing unit 932.

  In this case, the subject to be evaluated is the eye to be inspected, which has been treated at at least one of the first time and the second time after the first time. The treatment information acquisition part 935 acquires the information regarding this treatment. The arithmetic processing unit 932 calculates first measurement data of the eye to be examined associated with the first time and second measurement data of the eye to be examined associated with the second time from, for example, a tomographic image. It should be noted that here, the analysis value obtained by the image analysis of the tomographic image of the eye to be examined described in Example 1 and the measurement value obtained by measuring the eye to be examined described in Example 2 are included and actually measured. The data obtained from the optometry is used as the measurement data. The arithmetic processing unit 932 inputs the evaluation information associated with at least one of the first time and the second time into the learned model, with the first and second measurement data and the information regarding the treatment. . The learned model can also generate the evaluation information of the eye to be inspected at the third time after the second time by using the input measurement data and information.

  In this case, the display control unit 305, the arithmetic processing unit 932 is the measurement data predicted to be obtained at the third time of the subject's eye generated using the learned model, the third time Generate the third measurement data in. The display control unit 305 generates the graph 1020 using the generated third measurement data, the first measurement data, and the second measurement data. The display control unit 305 can display the graph 1020 on the display screen 1010 of the display unit 600. Further, at that time, the display control unit 305 also displays the evaluation information (probability 1025) of the third time generated on the display screen 1010 and the type of treatment (second selector 1021) together. You can At this time, the type of treatment may include surgery and / or medication. Further, the evaluation information regarding the state of the eye to be inspected is information presented by a user such as a doctor as a diagnosis result of the eye to be inspected, based on a tomographic image, its analysis value, the above-mentioned measured value, or other observation image. , Entered by the user.

  According to the present embodiment, not only the tomographic data and the analysis value thereof used in the first embodiment but also the presence / absence of the treatment applied to the eye to be examined are used, and there is a high possibility that the learned model can be obtained from the eye to be examined in the future. It is possible to obtain the tomographic data, the analysis value thereof, and the evaluation information indicating the state of the eye to be inspected. Therefore, when diagnosing the eye to be inspected, it is possible to present the possibility of how the state of the eye to be inspected by treatment to a doctor, etc., and more effective information for assisting diagnosis for follow-up observation, treatment policy, etc. Can be presented.

(Modification 1)
In the first to third embodiments, the acquisition unit 301 acquires the tomographic data obtained by imaging the eye to be inspected by the OCT apparatus 200, and the image processing apparatus uses the tomographic data to generate the above-described analysis values and the like. There is. However, the configuration in which the acquisition unit 301 acquires these tomographic data is not limited to this. For example, the acquisition unit 301 acquires the tomographic data that has already been photographed and has been stored in a server that is connected to the image processing device such as the external storage device 500 via a LAN, WAN, or the Internet, and the tomographic data is acquired. You may perform the series of processes mentioned above using data. In that case, the processing relating to imaging is omitted, and instead, a tomographic image that has already been captured is acquired. Then, in step S304, the retinal layer is detected and the retinal thickness is measured. Accordingly, the detection of the retinal layer can be performed when necessary without performing processing at the time of shooting, and it is possible to concentrate only on shooting at the time of shooting.

(Modification 2)
In the first embodiment, by inputting the tomographic data, the analysis value obtained from the tomographic data, and the evaluation information of the eye to be examined into the learned model, an image for obtaining the analysis value and the evaluation information that are likely to be obtained in the future is obtained. The processor is described. Further, in the second embodiment, by inputting the measured value of the eye to be examined into the learned model together with the tomographic data or the analytical value obtained from the tomographic data, the analytical value and the measured value that are highly likely to be obtained in the future can be obtained. The image processing device to be obtained is described. However, the aspect of the present invention is not limited to these examples, and for example, by inputting the measured value of the eye to be inspected and the evaluation information obtained by referring to the measured value to the learned model, the possibility of being obtained in the future may be increased. It may be possible to obtain high measurement values and evaluation information.

  Further, in the third embodiment, only the flag relating to the treatment and the measured value of the eye to be examined may be input to the learned model to obtain a measured value that is likely to be obtained in the future. Further, by additionally inputting the evaluation information, the evaluation information that is highly likely to be obtained in the future may be obtained together. In the above-described embodiment, the input data is data obtained at two times, one associated with the first time and one associated with the second time, and obtained from these data at the third time. It is intended to generate data that is likely to be used. However, the number of input data or the number of input data used for generating the trained model is not limited to the illustrated two, and can be increased. By increasing the input data and the input data, the accuracy of the data (output data) predicted to be obtained can be increased. In addition, when the actual input data is less than the number of input data used in the generation of the learned model, for example, by complementing the missing data by an appropriate method, it is possible to obtain appropriate output data. For example, in Example 2, when a learned model obtained by using five analysis values and measurement values arranged in time series as input data is used, insufficient analysis is performed even when there are three or four analysis values. Data can be output by complementing the values. For example, it is conceivable to estimate the missing analysis value using at least two analysis values before and after the missing analysis value. At this time, the average value, the median value, etc. obtained from at least two analysis values may be estimated as the insufficient analysis value. Further, the insufficient analysis value may be estimated using an approximate straight line or an approximate curve obtained from at least two analysis values. Further, the plurality of types of analysis values associated with the first time and the plurality of types of analysis values associated with the second time do not have to be exactly the same type, and even if there are some different types. It's good, and there may be some missing types. Further, the above-described contents can be similarly applied to the range at the time of learning as long as there is no contradiction.

  The learned model can be provided in the arithmetic processing units 332 and 832 as described above. In this case, the learned model can be composed of, for example, a software module executed by a processor such as a CPU. The learned model may be provided in another server or the like connected to the image processing devices 300 and 800. In this case, by connecting to a server equipped with a learned model via an arbitrary network such as the Internet, which can be connected to the arithmetic processing units 332 and 832, the learned value is used to analyze, measure, and evaluate. It is possible to generate information and the like.

(Other embodiments)
Each of the above-described embodiments implements the present invention as an image processing apparatus. However, the embodiment of the present invention is not limited to the image processing apparatus or the image processing method described in the embodiment. The present invention can also be realized as software that operates on a computer. Further, the CPU of the image processing apparatus controls the entire computer by using the computer programs and data stored in the RAM and ROM. It is also possible to control the execution of software corresponding to each unit of the image processing apparatus, realize the functions of each unit, and execute each process performed during image processing. Further, the user interface such as buttons and the layout of display are not limited to the above-described modes.

  Furthermore, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Although the present invention has been described above with reference to the exemplary embodiments and modified examples, the present invention is not limited to the exemplary embodiments and modified examples. Inventions modified within a range not departing from the spirit of the present invention and inventions equivalent to the present invention are also included in the present invention. Further, the above-described respective embodiments and modified examples can be appropriately combined without departing from the spirit of the present invention.

1, 8: image processing system, 200: tomographic image capturing device, 300, 800: image processing device, 301: image acquisition unit, 303: processing unit, 305: display control unit, 311: tomographic image generation unit, 332, 832. : Arithmetic processing unit, 333: shooting condition acquisition unit, 334: selection unit, 835: measurement value acquisition unit

Claims (26)

Data relating to a slice including multiple types of analysis values of the eye to be inspected, data relating to the first slice of the eye to be examined associated with a first time, and a second time after the first time. Data relating to the second cross-section of the associated eye to be inspected, and acquisition means,
Using a learned model, from the acquired data concerning the first tom and the acquired data concerning the second tom, a plurality of types of the eye to be examined at a third time after the second time. Generating means for generating at least one of the analyzed values of
An image processing apparatus comprising: The data related to the first tomography is a plurality of types of analysis values obtained by analyzing a tomographic image obtained by photographing at the first time,
The image processing apparatus according to claim 1, wherein the data relating to the second tomographic image is a plurality of types of analysis values obtained by analyzing a tomographic image obtained by photographing at the second time.   The learned model is a plurality of types of analysis values obtained by analyzing a tomographic image obtained by imaging the eye to be examined at a fourth time before the first time, and before the first time. And and the plurality of types of analysis values obtained by analyzing the tomographic image obtained by photographing the eye to be examined at the fifth time after the fourth time and the input data, and before the first time. And, it can be obtained by using learning data in which a plurality of kinds of analysis values obtained by analyzing a tomographic image obtained by photographing the eye to be examined at a sixth time after the fifth time are used as teacher data. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Data relating to a tomographic image including a tomographic image obtained by photographing the eye to be inspected, data relating to a first tomographic region of the eye to be inspected associated with a first time, and second data after the first time. Acquisition means for acquiring data relating to a second cross section of the eye to be inspected, which is associated with time,
Using a learned model, from the acquired data concerning the first tom and the acquired data concerning the second tom, a plurality of types of the eye to be examined at a third time after the second time. Generating means for generating at least one of the analyzed values of
An image processing apparatus comprising:   The generation means generates data regarding a tomographic image including a tomographic image obtained by photographing the eye to be inspected, and data regarding a third tomographic image of the eye to be inspected, using the learned model, and is generated. The image processing apparatus according to claim 4, wherein at least one of the plurality of types of analysis values is generated using the data regarding the third tomographic image.   The learned model is a tomographic image obtained by photographing the eye to be inspected at a fourth time before the first time, and a tomographic image before the first time and after the fourth time. The tomographic image obtained by photographing the eye to be examined at five times is used as input data, and the eye to be examined is photographed at a sixth time before the first time and after the fifth time. The image processing device according to claim 4, wherein the image processing device is obtained by using learning data in which the obtained tomographic image is teacher data.   7. The elapsed time between the second time and the third time and the elapsed time between the fifth time and the sixth time are substantially equal. The image processing device according to item 1. The acquisition unit further acquires a first measurement value of the eye to be examined associated with the first time, and a second measurement value of the eye to be examined associated with the second time,
The generation means, based on the data regarding the first slice, the first measurement value, the data regarding the second slice, and the second measurement value, the plurality of types of the plurality of types at the third time. The image processing device according to claim 1, wherein at least one of the analysis values is generated. The acquisition means is a treatment associated with at least one of the data regarding the first tomography and the data regarding the second tomography, at the first time or the second time for the eye to be examined. Get information about the treatments given,
The generating means may generate at least one of the plurality of types of analysis values at the third time from the data on the first slice, the data on the second slice, and the information on the treatment. The image processing apparatus according to any one of claims 1 to 8, characterized in that. The acquisition means acquires evaluation information regarding the state of the eye to be examined at the first time and the second time,
10. The generating unit generates the evaluation information at the third time in combination with at least one of the plurality of types of analysis values at the third time. The image processing device according to item.   The image processing apparatus according to claim 10, further comprising a display control unit that displays the generated evaluation information on a display unit.   A first analysis value and a second analysis value obtained from the data on the first fault and the data on the second fault, which are the same kind as the at least one analysis value at the third time. The display control means for displaying a graph obtained by using the one analysis value, the second analysis value, and the at least one analysis value at the third time on a display unit. The image processing device according to any one of 1 to 11.   The selection means for selecting the learned model based on an acquisition condition for acquiring the data regarding the first slice and the data regarding the second slice from the eye to be inspected. The image processing apparatus according to any one of items.   The image processing apparatus according to claim 13, wherein the acquisition condition includes at least one of a shooting date and time, a shooting region name, a shooting region, a shooting angle of view, and a shooting method.   The plurality of types of analysis values include the thickness of the entire retina, the thickness between arbitrary layers, the thickness of the choroid, and the shape characteristics of the retina in the posterior segment of the eye to be inspected, the retina thickness in the optic papilla, and the Cup. And Disc, and the corneal thickness, the corneal curvature, the anterior chamber depth, the entire lens curvature, and the posterior lens curvature in the anterior segment of the eye, and the image according to any one of claims 1 to 14. Processing equipment. A subject eye that has been treated at at least one of a first time and a second time later than the first time, the first eye of the eye being associated with the first time. Measurement data, second measurement data of the eye to be examined associated with the second time, and an evaluation regarding the state of the eye to be examined, which is associated with at least one of the first time and the second time. Acquisition means for acquiring information and
Using a learned model, from the information about the treatment, the first measurement data, the second measurement data, and the evaluation information, the state of the eye to be examined at a third time after the second time Generating means for generating evaluation information regarding
An image processing apparatus comprising: The generation unit generates a third measurement data of the eye to be examined at the third time using a learned model,
The generated third measurement data, the graph obtained using the first measurement data and the second measurement data, the evaluation information at the generated third time, and the type of treatment 17. The image processing apparatus according to claim 16, further comprising a display control unit that displays, on the display unit.   The image processing device according to claim 16, wherein the evaluation information regarding the state of the eye to be inspected is information input by a user as a diagnosis result of the eye to be inspected.   19. The image processing apparatus according to claim 16, wherein the treatment includes at least one of surgery and medication. The first measurement data is obtained from a tomographic image obtained by imaging the eye to be examined at the first time,
20. The image processing apparatus according to claim 16, wherein the second measurement data is obtained from a tomographic image obtained by photographing the eye to be inspected at the second time.   Multiple types of analysis values obtained by analyzing a tomographic image obtained by photographing the eye to be examined at the first time, and obtained by photographing the eye to be examined at the second time after the first time Multiple types of analysis values obtained by analyzing the tomographic image are used as input data, and a plurality of images obtained by analyzing the tomographic image obtained by photographing the eye to be examined at a third time after the second time. A learned model that is obtained by using learning data that uses analysis values of various types as teacher data.   A tomographic image obtained by photographing the eye to be inspected at a first time and a tomographic image obtained by photographing the eye to be inspected at a second time after the first time are input data, and the second A learned model obtained by using learning data in which a tomographic image obtained by photographing the eye to be inspected is used as teacher data in a third time after the time. Data relating to a slice including multiple types of analysis values of the eye to be inspected, data relating to the first slice of the eye to be examined associated with a first time, and a second time after the first time. Acquiring data related to the second tomographic image of the eye to be inspected,
Using a learned model, from the acquired data concerning the first tom and the acquired data concerning the second tom, a plurality of types of the eye to be examined at a third time after the second time. Generating at least one of the parsed values of
An image processing method comprising: Data relating to a tomographic image including a tomographic image obtained by photographing the eye to be inspected, data relating to a first tomographic region of the eye to be inspected associated with a first time, and second data after the first time. Obtaining data relating to a second slice of the eye to be examined in relation to time, and
Using a learned model, from the acquired data concerning the first tom and the acquired data concerning the second tom, a plurality of types of the eye to be examined at a third time after the second time. Generating at least one of the parsed values of
An image processing method comprising: A subject eye that has been treated at at least one of a first time and a second time later than the first time, the first eye of the eye being associated with the first time. Measurement data, second measurement data of the eye to be examined associated with the second time, and an evaluation regarding the state of the eye to be examined, which is associated with at least one of the first time and the second time. The step of obtaining the information and
Using a learned model, from the information about the treatment, the first measurement data, the second measurement data, and the evaluation information, the state of the eye to be examined at a third time after the second time Generating evaluation information about
An image processing method comprising:   A program that, when executed by a processor, causes the processor to execute each step of the image processing method according to claim 23.

Images