JP7269413B2 - MEDICAL IMAGE PROCESSING APPARATUS, MEDICAL IMAGE PROCESSING SYSTEM, MEDICAL IMAGE PROCESSING METHOD AND PROGRAM - Google Patents

Description

The present invention relates to a medical image processing apparatus, a medical image processing system , a medical image processing method, and a program.

An eye tomography apparatus such as an apparatus using optical coherence tomography (OCT) (OCT apparatus) is capable of three-dimensionally observing the state inside the retinal layers. In recent years, this tomographic imaging apparatus has been attracting attention because it is useful for more accurately diagnosing diseases.

As a form of OCT, for example, there is TD-OCT (Time domain OCT) combining a broadband light source and a Michelson interferometer. This is configured to obtain depth-resolved information by scanning the delay of the reference arm to measure interference light with the backscattered light of the signal arm. However, high-speed image acquisition is difficult with such TD-OCT.

Therefore, SD-OCT (Spectral domain OCT), which uses a broadband light source and acquires an interferogram with a spectroscope, is known as a method for acquiring images at a higher speed. Also known is SS-OCT (Swept Source OCT), which uses a fast wavelength swept light source as a light source and measures spectral interference with a single-channel photodetector.

When a tomographic image obtained by OCT is acquired, if the thickness of the nerve fiber layer can be measured, the degree of progression of diseases such as glaucoma and the degree of recovery after treatment can be quantitatively diagnosed. In order to quantitatively measure the thickness of these layers, Patent Literature 1 discloses a technique of detecting boundaries between layers of the retina from a tomographic image using a computer and measuring the thickness of each layer.

JP-A-2008-73099

However, the conventional technique has the following problems. In diseased eyes, the shape of the retina becomes irregular due to layer loss, hemorrhage, vitiligo, and neovascularization. For this reason, conventional image processing methods, in which the results of image feature extraction are judged using the regularity of the shape of the retina and the boundaries of the retinal layers are detected, may cause errors such as false detections when automatically detecting the boundaries of the retinal layers. There was a limit to the occurrence of

Accordingly, it is an object of the present invention to provide a medical image processing apparatus, a medical image processing system , a medical image processing method, and a program capable of detecting retinal layer boundaries irrespective of disease, site, or the like.

A medical image processing apparatus according to an embodiment of the present invention includes an acquisition unit that acquires a tomographic image of an eye to be inspected, and learning data including a tomographic image of the eye to be inspected and an image containing position information about at least one layer in the tomographic image. wherein at least a detection unit that detects position information about at least one layer in the acquired tomographic image by outputting an image containing position information about one layer; generating a front image corresponding to a depth range determined using position information about the detected at least one layer , and using the front image as input data for a trained model different from the trained model; and a display control unit configured to display at least one of the front image and the high-quality front image on a display unit .

In addition, a medical image processing method according to another embodiment of the present invention includes the steps of obtaining a tomographic image of an eye to be inspected, and a tomographic image of the eye to be inspected and an image containing position information about at least one layer in the tomographic image. a learned model obtained by learning using learning data including a step of detecting position information about at least one layer in the acquired tomographic image by outputting an image containing position information about at least one layer in the eye; generating a front image corresponding to a depth range determined using position information about the detected at least one layer; The method includes a step of generating a front image with higher image quality than the front image by using an image , and a step of displaying at least one of the front image and the front image with high image quality on a display unit .

According to one embodiment of the present invention, boundary detection of retinal layers can be performed regardless of disease, site, or the like.

Further features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings.

1 shows an example of a schematic configuration of an image processing system according to a first embodiment; FIG. 2 is a diagram for explaining the structure of an eye, a tomographic image, and a fundus image; 4 is a flowchart of a series of processes according to the first embodiment; FIG. 4 is a diagram for explaining an example of a learning image; FIG. FIG. 4 is a diagram for explaining an example of the size of a learning image; FIG. FIG. 4 is a diagram for explaining an example of a machine learning model according to the first embodiment; FIG. An example of a display screen is shown. FIG. 11 shows an example of a schematic configuration of an image processing system according to a second embodiment; FIG. 9 is a flowchart of a series of processes according to Example 2; FIG. 4 is a diagram for explaining detection of a retinal region; FIG. 4 is a diagram for explaining an example of the size of a learning image; FIG. FIG. 11 is a diagram for explaining an example of a machine learning model according to Example 2; FIG. 11 is a diagram for explaining retinal layer detection according to the second embodiment; FIG. 4 is a diagram for explaining an example of input and output images in a trained model; FIG. 11 shows an example of a schematic configuration of an image processing system according to a fourth embodiment; FIG. 10 is a flowchart of a series of processes according to Example 4; FIG. 11 shows an example of a schematic configuration of an image processing system according to a fifth embodiment; FIG. 10 is a flowchart of a series of processes according to Example 5; FIG. 10 is a diagram for explaining correction processing of a retinal area; FIG. 14 is a diagram for explaining an example of a learning image according to Example 6; An example of a plurality of OCTA En-Face images and luminance tomographic images is shown. 11 shows an example of a user interface according to Example 7. FIG. 11 shows an example of a user interface according to Example 7. FIG. An example of an area label image related to explanation of terms is shown. An example of the configuration of a neural network related to the explanation of terms is shown. An example of the configuration of a neural network related to the explanation of terms is shown. An example of an area label image related to explanation of terms is shown. FIG. 11 shows an example of the configuration of an image processing apparatus according to an eighth embodiment; FIG. FIG. 12 is a flow chart showing an example of the flow of processing of the image processing apparatus according to the eighth embodiment; FIG. FIG. 12 is a flow chart showing an example of the flow of processing of the image processing apparatus according to the eighth embodiment; FIG. FIG. 21 is a diagram showing an example of a user interface provided in an imaging device according to Example 8; FIG. 21 is a diagram showing an example of a user interface provided in an imaging device according to Example 8; FIG. 16 is a flow chart showing an example of the flow of processing of the image processing apparatus according to the ninth embodiment; FIG. 11 shows image processing according to the eleventh embodiment. FIG. 22 is a flow chart showing an example of the flow of processing of the image processing apparatus according to the eleventh embodiment; FIG. 12 shows image processing according to the twelfth embodiment. FIG. 22 is a flow chart showing an example of the flow of processing of the image processing apparatus according to the thirteenth embodiment; FIG. 13 shows image processing according to the thirteenth embodiment. FIG. 22 is a flow chart showing an example of the flow of processing of the image processing apparatus according to the thirteenth embodiment; FIG. 13 shows image processing according to the thirteenth embodiment. FIG. 16 is a flow chart showing an example of the flow of processing of the image processing apparatus according to the fourteenth embodiment; FIG. FIG. 20 is a diagram showing an example of a user interface provided in an imaging device according to Example 15; FIG. 12 illustrates an example of the configuration of an image processing apparatus according to an eighteenth embodiment; FIG. FIG. 11 shows an example of the configuration of an image processing apparatus according to a nineteenth embodiment; FIG. 19 is a flow chart showing an example of the flow of processing of the image processing apparatus according to the nineteenth embodiment; 11 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. FIG. 11 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. FIG.

Exemplary embodiments for carrying out the invention will now be described in detail with reference to the drawings.

However, the dimensions, materials, shapes, relative positions of components, etc. described in the following examples are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, the same reference numbers are used in the drawings to indicate identical or functionally similar elements.

(Example 1)
An image processing system including an image processing apparatus using a tomographic image of an eye according to a first embodiment of the present invention will be described below with reference to FIGS. 1 to 7. FIG. In this embodiment, all target retinal layers are detected using a trained model related to the machine learning model. In the following description, a machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. A trained model is a machine learning model based on an arbitrary machine learning algorithm that has been trained (learned) in advance using appropriate teacher data. However, it is assumed that the trained model is not one that does not perform further learning, and that additional learning can be performed. Note that, hereinafter, teacher data means learning data, and is composed of a pair of input data and output data. Further, correct data means output data of learning data (teaching data).

FIG. 1 shows an example of a schematic configuration of an image processing system 1 including an image processing apparatus 20 (medical image processing apparatus) according to this embodiment. As shown in FIG. 1, the image processing system 1 includes an OCT apparatus 10 as an example of a tomography apparatus, an image processing apparatus 20, a fundus imaging apparatus 30, an external storage device 40, a display unit 50, and an input unit 60. is provided.

The OCT apparatus 10 is an example of a tomographic image capturing apparatus for capturing a tomographic image of an eye to be examined. Any type of OCT apparatus can be used as the OCT apparatus, such as SD-OCT and SS-OCT.

The image processing device 20 is connected to the OCT device 10, the fundus imaging device 30, the external storage device 40, the display section 50, and the input section 60 via interfaces, and can control these. The image processing device 20 generates various images such as a tomographic image and an En-Face image (frontal image) of the subject's eye based on various signals acquired from the OCT device 10, the fundus imaging device 30, and the external storage device 40. can do. Further, the image processing device 20 can perform image processing on these images. Note that the image processing apparatus 20 may be configured by a general-purpose computer, or may be configured by a computer dedicated to the image processing system 1 .

The fundus image capturing device 30 is a device for capturing a fundus image of the subject's eye, and for example, a fundus camera, an SLO (Scanning Laser Ophthalmoscope), or the like can be used as the device. The device configuration of the OCT device 10 and the fundus imaging device 30 may be an integrated type or a separate type.

The external storage device 40 holds information about the eye to be examined (patient's name, age, sex, etc.), various captured image data, imaging parameters, image analysis parameters, and parameters set by the operator in association with each other. there is The external storage device 40 may be configured by any storage device, and may be configured by, for example, a storage medium such as an optical disk or memory.

The display unit 50 is configured by an arbitrary display, and can display information and various images related to the subject's eye under the control of the image processing device 20 .

The input unit 60 is, for example, a mouse, a keyboard, or a touch operation screen. can be input to device 20; When the input unit 60 is a touch operation screen, the input unit 60 can be integrated with the display unit 50 .

In addition, although these components are shown as separate bodies in FIG. 1, some or all of these components may be integrated.

Next, the OCT apparatus 10 will be explained. The OCT apparatus 10 is provided with a light source 11 , a galvanomirror 12 , a focus lens stage 13 , a coherence gate stage 14 , a detector 15 and an internal fixation lamp 16 . Since the OCT apparatus 10 is a well-known apparatus, a detailed description thereof is omitted. Here, the imaging of a tomographic image performed according to an instruction from the image processing apparatus 20 will be described.

When the image processing device 20 gives an instruction for photographing, the light source 11 emits light. Light from the light source 11 is split into measurement light and reference light using a splitter (not shown). The OCT apparatus 10 irradiates a subject (eye to be examined) with measurement light and detects interference light between the return light from the subject and the reference light to generate an interference signal including tomographic information of the subject. be able to.

The galvanometer mirror 12 is used to scan the fundus of the subject's eye with measurement light, and the scanning range of the measurement light by the galvanometer mirror 12 can define the imaging range of the fundus in OCT imaging. By controlling the drive range and speed of the galvanomirror 12, the image processing apparatus 20 can define the planar imaging range and the number of scanning lines (scanning speed in the planar direction) on the fundus. In FIG. 1, the galvanomirror 12 is shown as one unit for the sake of simplification of explanation, but the galvanomirror 12 is actually composed of two mirrors for X scanning and Y scanning. A desired range on the above can be scanned with the measuring light. Note that the configuration of the scanning unit for scanning the measurement light is not limited to the galvanomirror, and any other deflection mirror can be used. Further, as the scanning unit, for example, a single deflecting mirror such as a MEMS mirror that can scan the measurement light in two-dimensional directions may be used.

A focus lens (not shown) is provided on the focus lens stage 13 . By moving the focus lens stage 13, the focus lens can be moved along the optical axis of the measurement light. Therefore, the focus lens can focus the measurement light on the retinal layer of the fundus via the anterior segment of the eye to be examined. The measurement light that irradiates the fundus is reflected and scattered by each retinal layer and returns along the optical path as return light.

The coherence gate stage 14 is used to adjust the length of the optical path of the reference light or measurement light in order to cope with the difference in axial length of the eye to be examined. In this embodiment, the coherence gate stage 14 is composed of a stage provided with a mirror, and by moving along the optical path of the reference light in the optical axis direction, the optical path length of the reference light can correspond to the optical path length of the measurement light. can. Here, the coherence gate represents a position in OCT where the optical distances of the measurement light and the reference light are equal. Coherence gate stage 14 may be controlled by image processor 20 . The image processing apparatus 20 can control the imaging range in the depth direction of the subject's eye by controlling the position of the coherence gate by the coherence gate stage 14, and can perform imaging on the retinal layer side or on the deeper side than the retinal layer. Shooting and the like can be controlled.

The detector 15 detects interference light between the return light of the measurement light from the eye to be inspected and the reference light, which is generated in an interference portion (not shown), and generates an interference signal. The image processing device 20 can generate a tomographic image of the subject's eye by acquiring an interference signal from the detector 15 and performing Fourier transform or the like on the interference signal.

The internal fixation lamp 16 is provided with a display section 161 and a lens 162 . In this embodiment, as an example of the display unit 161, a display unit in which a plurality of light emitting diodes (LD) are arranged in a matrix is used. The lighting position of the light-emitting diode is changed according to the part to be photographed under the control of the image processing device 20 . Light from the display unit 161 is guided to the subject's eye via the lens 162 . The light emitted from the display unit 161 has a wavelength of 520 nm, for example, and is displayed in a desired pattern under the control of the image processing device 20 .

Note that the OCT apparatus 10 may be provided with a drive control unit for the OCT apparatus 10 that controls driving of each component based on control by the image processing apparatus 20 .

Next, the structure and images of the eye acquired by the image processing system 1 will be described with reference to FIGS. 2(a) to 2(c). FIG. 2(a) is a schematic diagram of an eyeball. FIG. 2(a) shows the cornea C, lens CL, vitreous body V, macula M (the center of the macula represents the fovea), and optic papilla D. FIG. In this embodiment, a case of photographing the posterior pole of the retina including the vitreous body V, the macula M, and the optic papilla D will be mainly described. Although not described below, the OCT apparatus 10 can also photograph the anterior segment of the eye such as the cornea and the lens.

FIG. 2B shows an example of a tomographic image obtained by imaging the retina using the OCT apparatus 10. As shown in FIG. In FIG. 2B, AS indicates an image unit obtained by one A-scan. Here, the A-scan means acquisition of tomographic information in the depth direction at one point of the subject's eye through the series of operations of the OCT apparatus 10 . Acquiring two-dimensional tomographic information of the subject's eye in the transverse direction (main scanning direction) by performing A-scans multiple times in the transverse direction (main scanning direction) is called B-scanning. A single B-scan image can be configured by collecting a plurality of A-scan images acquired by A-scan. This B-scan image is hereinafter referred to as a tomographic image.

In FIG. 2(b), a blood vessel Ve, a vitreous body V, a macula M, and an optic papilla D are shown. In addition, the boundary line L1 is the boundary between the inner limiting membrane (ILM) and the nerve fiber layer (NFL), the boundary line L2 is the boundary between the nerve fiber layer and the ganglion cell layer (GCL), and the boundary line L3 is the photoreceptor inner segment. Represents outer segment junction (ISOS). Furthermore, the boundary line L4 represents the retinal pigment epithelium layer (RPE), the boundary line L5 represents the Bruch's membrane (BM), and the boundary line L6 represents the choroid. In a tomographic image, the horizontal axis (main scanning direction of OCT) is the x-axis, and the vertical axis (depth direction) is the z-axis.

FIG. 2C shows an example of a fundus image obtained by photographing the fundus of the subject's eye using the fundus imaging device 30 . FIG. 2(c) shows the macula M and the optic papilla D, and the blood vessels of the retina are indicated by thick curves. In the fundus image, the horizontal axis (main scanning direction of OCT) is the x-axis, and the vertical axis (sub-scanning direction of OCT) is the y-axis.

Next, the image processing device 20 will be described. The image processing device 20 is provided with an acquisition section 21 , an image processing section 22 , a drive control section 23 , a storage section 24 and a display control section 25 .

The acquisition unit 21 can acquire the data of the interference signal of the eye to be examined from the OCT apparatus 10 . Note that the interference signal data acquired by the acquisition unit 21 may be an analog signal or a digital signal. When the acquisition unit 21 acquires an analog signal, the image processing device 20 can convert the analog signal into a digital signal. Further, the acquisition unit 21 can acquire various images such as tomographic data, tomographic images, and En-Face images generated by the image processing unit 22 . Here, the tomographic data is data including information about the tomography of the subject, and data based on interference signals by OCT, and data obtained by subjecting this to fast Fourier transform (FFT) and arbitrary signal processing. includes

Furthermore, the acquisition unit 21 acquires a group of imaging conditions for a tomographic image to be image-processed (for example, imaging date and time, imaging site name, imaging region, imaging angle of view, imaging method, image resolution and gradation, image pixel size, image filter, and information about the data format of the image). Note that the imaging condition group is not limited to the examples. Also, the imaging condition group does not need to include all of the exemplified ones, and may include some of them.

The acquisition unit 21 can also acquire data including fundus information acquired by the fundus imaging device 30 . Furthermore, the acquisition unit 21 can acquire information for identifying the eye to be examined, such as the subject's identification number, from the input unit 60 or the like. The acquisition unit 21 can store the acquired various data and images in the storage unit 24 .

The image processing unit 22 can generate a tomographic image, an En-Face image, or the like from the data acquired by the acquisition unit 21 or the data stored in the storage unit 24, and can apply image processing to the generated or acquired image. . Therefore, the image processing unit 22 can function as an example of a generating unit that generates an En-Face image or a motion contrast front image described later. The image processing unit 22 is provided with a tomographic image generating unit 221 and a processing unit 222 (first processing unit).

The tomographic image generation unit 221 can generate tomographic data by performing processing such as Fourier transform on the interference signal acquired by the acquisition unit 21, and can generate a tomographic image based on the tomographic data. Note that any known method may be employed as a method for generating a tomographic image, and detailed description thereof will be omitted.

The processing unit 222 can include trained models for machine learning models by machine learning algorithms such as deep learning. A specific machine learning model will be described later. The processing unit 222 uses the learned model to execute detection processing for detecting the retinal layers of the subject's eye in the tomographic image, and detects each retinal layer.

The drive control unit 23 can control driving of each component of the OCT device 10 and the fundus imaging device 30 connected to the image processing device 20 . The storage unit 24 can store tomographic data acquired by the acquiring unit 21 and various images such as tomographic images generated and processed by the image processing unit 22, data, and the like. The storage unit 24 can also store a program or the like for performing the function of each component of the image processing apparatus 20 by being executed by the processor.

The display control unit 25 can control display on the display unit 50 of various information acquired by the acquisition unit 21, tomographic images generated and processed by the image processing unit 22, and information input by the operator. .

Each component of the image processing apparatus 20 other than the storage unit 24 may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit). Note that the processor may be, for example, a GPU (Graphical Processing Unit), an FPGA (Field-Programmable Gate Array), or the like. Also, each component may be configured by a circuit or the like that performs a specific function, such as an ASIC. The storage unit 24 may be composed of, for example, an arbitrary storage medium such as an optical disk or memory.

Next, a series of processes according to this embodiment will be described with reference to FIG. FIG. 3 is a flowchart of a series of processes according to this embodiment. When a series of processes according to this embodiment is started, the process proceeds to step S301.

In step S<b>301 , the acquisition unit 21 acquires a subject identification number, which is an example of information for identifying an eye to be examined, from outside the image processing apparatus 20 such as the input unit 60 . The acquisition unit 21 acquires information about the subject's eye held in the external storage device 40 based on the subject identification number, and stores the information in the storage unit 24 .

In step S<b>302 , the drive control unit 23 controls the OCT apparatus 10 to scan the subject's eye for imaging, and the acquisition unit 21 acquires an interference signal including tomographic information of the subject's eye from the OCT apparatus 10 . The eye to be inspected is scanned by the drive control unit 23 controlling the OCT apparatus 10 and operating the light source 11, the galvanomirror 12, etc. in response to an operator's instruction to start scanning.

The galvanomirror 12 includes a horizontal X scanner and a vertical Y scanner. Therefore, the drive control unit 23 can scan the measurement light in each of the horizontal (X) and vertical (Y) directions in the device coordinate system by changing the directions of these scanners. By simultaneously changing the directions of these scanners, the drive control unit 23 can also scan the measurement light in a direction that combines the horizontal direction and the vertical direction. Therefore, the drive control unit 23 can scan the measurement light in any direction on the fundus plane.

The drive control unit 23 adjusts various photographing parameters when photographing. Specifically, the drive control unit 23 sets at least the position of the pattern displayed by the internal fixation lamp 16, the scan range and scan pattern by the galvanomirror 12, the coherence gate position, and the focus.

The drive control unit 23 controls the light emitting diodes of the display unit 161 to control the position of the pattern displayed by the internal fixation lamp 16 so as to image the center of the macular region and the optic papilla of the subject's eye. Further, the drive control unit 23 sets, as the scan pattern of the galvanomirror 12, a scan pattern such as a raster scan, a radial scan, or a cross scan for imaging a three-dimensional volume. Note that regardless of which scan pattern is selected, a plurality of images (the number of repetitions is two or more) are repeatedly taken on one line. In this embodiment, a cross scan is used as the scan pattern, and 150 images of the same location are repeatedly photographed. After the adjustment of these imaging parameters is completed, the drive control unit 23 controls the OCT apparatus 10 to perform imaging of the subject's eye in response to an operator's instruction to start imaging. Note that the number of repetitions according to this embodiment is an example, and may be set to an arbitrary number according to a desired configuration.

Although detailed description is omitted in the present disclosure, the OCT apparatus 10 can perform tracking of the subject's eye in order to photograph the same location for averaging. As a result, the OCT apparatus 10 can scan the subject's eye with less influence of involuntary eye movement.

In step S<b>303 , the tomographic image generation unit 221 generates a tomographic image based on the interference signal acquired by the acquisition unit 21 . The tomographic image generator 221 can generate a tomographic image by performing general reconstruction processing on each interference signal.

First, the tomographic image generator 221 removes fixed pattern noise from the interference signal. Fixed pattern noise removal is performed by averaging a plurality of acquired A-scan signals to extract fixed pattern noise and subtracting it from the input interference signal. After that, the tomographic image generator 221 performs desired window function processing in order to optimize the depth resolution and the dynamic range, which are in a trade-off relationship when the interference signal is Fourier transformed in a finite interval. The tomographic image generating unit 221 generates tomographic data by performing fast Fourier transform (FFT) processing on the window function-processed interference signal.

The tomographic image generation unit 221 obtains each pixel value of the tomographic image based on the generated tomographic data, and generates a tomographic image. Note that the method for generating the tomographic image is not limited to this, and any known method may be used.

In step S304, the processing unit 222 of the image processing unit 22 performs retinal layer detection processing. Processing of the processing unit 222 will be described with reference to FIGS.

The processing unit 222 detects boundaries between retinal layers in a plurality of tomographic images acquired using the OCT apparatus 10 . The processing unit 222 detects each retinal layer using a learned model related to a machine learning model in which machine learning has been performed in advance.

Here, the machine learning algorithm according to this embodiment will be described with reference to FIGS. The learning data (teacher data) of the machine learning model according to this embodiment is composed of one or more pairs of input data and output data. Specifically, the input data includes a tomographic image 401 acquired by OCT, and the output data includes a boundary image 402 in which the boundaries of the retinal layers are specified for the tomographic image. In this embodiment, as the boundary image 402, an image showing a boundary 403 between the ILM and the NFL, a boundary 404 between the NFL and the GCL, the ISOS 405, the RPE 406, and the BM 407 is used. Although not shown, other boundaries include the boundary between the outer plexiform layer (OPL) and the outer nuclear layer (ONL), the boundary between the inner plexiform layer (IPL) and the inner nuclear layer (INL), and the boundary between the INL and the OPL. , a boundary between GCL and IPL, and the like may be used.

Note that the boundary image 402 used as the output data may be an image in which the boundary is indicated in the tomographic image by a doctor or the like, or may be an image in which the boundary is detected by rule-based boundary detection processing. However, if machine learning is performed using boundary images for which boundary detection has not been properly performed as output data for training data, images obtained using a trained model that has been trained using the training data will also detect boundaries appropriately. There is a possibility that it will become a boundary image that has not been processed. Therefore, by removing pairs including such boundary images from the training data, it is possible to reduce the possibility of generating inappropriate boundary images using the trained model. Here, rule-based processing refers to processing that uses known regularity, and rule-based boundary detection refers to boundary detection processing that uses known regularity such as the regularity of the shape of the retina.

Also, FIGS. 4A and 4B show an example of one XZ cross section in the XY plane of the retina, but the cross section is not limited to this. Although not shown, tomographic images and boundary images for arbitrary multiple XZ cross-sections in the XY plane are learned in advance, and cross-sections taken with various different scan patterns such as raster scan and radial scan can be handled. You can make it possible. For example, when using data such as tomographic images obtained by three-dimensionally photographing the retina by raster scanning, volume data obtained by aligning a plurality of adjacent tomographic images can be used as teacher data. In this case, from one volume data (three-dimensional tomographic image) and one corresponding three-dimensional boundary data (three-dimensional boundary image), it is possible to generate a group of paired images at arbitrary angles. It is possible. In addition, the machine learning model may be learned using images actually captured with various scan patterns as teacher data.

Next, images during learning will be described. An image group forming a pair group of a tomographic image 401 and a boundary image 402, which constitutes teacher data of a machine learning model, is created by rectangular area images of a fixed image size corresponding to a positional relationship. Creation of the image will be described with reference to FIGS. 5(a) to 5(c).

First, a case will be described in which a tomographic image 401 and a boundary image 402 are used as one of pairs constituting teacher data. In this case, as shown in FIG. 5A, a rectangular area image 501 that is the entire tomographic image 401 is input data, and a rectangular area image 502 that is the entire boundary image 402 is output data to form a pair. . In the example shown in FIG. 5A, the entire image constitutes a pair of input data and output data, but the pair is not limited to this.

For example, as shown in FIG. 5B, a rectangular area image 511 in the tomographic image 401 is used as input data, and a rectangular area image 513 corresponding to the imaging area in the boundary image 402 is used as output data to form a pair. good too. The rectangular areas of the rectangular area images 511 and 513 are based on the A-scan unit. The A-scan unit may be one A-scan unit or several A-scan units.

In FIG. 5B, the A-scan unit is used as a basis, but instead of defining the entire area in the depth direction of the image, portions outside the rectangular area may be provided above and below. That is, the horizontal size of the rectangular area may be set to several A scans, and the depthwise size of the rectangular area may be set smaller than the size of the image in the depthwise direction.

Also, as shown in FIG. 5C, a pair is formed by using a rectangular area image 521 in the tomographic image 401 as input data and a rectangular area image 523 corresponding to the imaging area in the boundary image 402 as output data. good too.

During learning, the scan range (shooting angle of view) and scan density (the number of A scans) can be normalized to uniform the image size, and the size of the rectangular area during learning can be uniformed. Also, the rectangular area images shown in FIGS. 5A to 5C are examples of rectangular area sizes when each is learned separately.

The number of rectangular areas can be set to one in the example shown in FIG. 5(a), and multiple in the examples shown in FIGS. 5(b) and 5(c). For example, in the example shown in FIG. 5B, a pair may be formed by using the rectangular area image 512 of the tomographic image 401 as input data and the rectangular area image 514, which is the corresponding imaging area in the boundary image 402, as output data. can. Further, for example, in the example shown in FIG. 5C, a pair is formed by using the rectangular area image 522 of the tomographic image 401 as input data and the rectangular area image 524 corresponding to the imaging area in the boundary image 402 as output data. can also In this manner, a pair of rectangular area images different from each other can be created from each pair of tomographic image and boundary image. By creating a large number of pairs of rectangular area images while changing the position of the area to different coordinates in the original tomographic image and boundary image, it is possible to enrich the pair group that constitutes the teacher data.

Although the examples shown in FIGS. 5B and 5C show discrete rectangular regions, in reality, the original tomographic image and the boundary image are continuously formed into rectangular regions of a constant image size without gaps. It can be divided into groups of images. Also, the original tomographic image and boundary image may be divided into rectangular area image groups at random positions corresponding to each other. In this way, by selecting an image of a smaller area as a rectangular area (or strip area) as a pair of input data and output data, many pairs can be obtained from the tomographic image 401 and the boundary image 402 that form the original pair. Can generate data. This reduces the time it takes to train machine learning models. On the other hand, the trained model of the completed machine learning model tends to take a long time to perform image segmentation processing. Here, image segmentation processing refers to processing for identifying and distinguishing regions and boundaries within an image.

Next, as an example of the machine learning model according to the present embodiment, a convolutional neural network (CNN) that performs image segmentation processing on input tomographic images will be described with reference to FIG. FIG. 6 shows an example of a machine learning model configuration 601 in the processing unit 222 . As the machine learning model according to the present embodiment, for example, FCN (Fully Convolutional Network), SegNet, or the like can also be used. Also, a machine learning model that performs object recognition on a region-by-region basis according to a desired configuration may be used. As a machine learning model for object recognition, RCNN (Region CNN), fastRCNN, or fasterRCNN, for example, can be used. Furthermore, YOLO (You Look Only Once) or SSD (Single Shot MultiBox Detector) can also be used as a machine learning model that performs object recognition in units of regions.

The machine learning model shown in FIG. 6 is composed of a plurality of layer groups that process and output a group of input values. The types of layers included in the configuration 601 of the machine learning model include a convolution layer, a downsampling layer, an upsampling layer, and a merger layer.

The convolution layer is a layer that performs convolution processing on an input value group according to set parameters such as the kernel size of filters, the number of filters, the stride value, and the dilation value. Note that the number of dimensions of the kernel size of the filter may also be changed according to the number of dimensions of the input image.

The down-sampling layer is a layer that performs processing to make the number of output value groups smaller than the number of input value groups by thinning out or synthesizing input value groups. Specifically, such processing includes, for example, Max Pooling processing.

The upsampling layer is a layer that performs processing to make the number of output value groups larger than the number of input value groups by duplicating the input value group or adding values interpolated from the input value group. Specifically, such processing includes, for example, linear interpolation processing.

The synthesizing layer is a layer that performs a process of synthesizing a group of values such as a group of output values of a certain layer and a group of pixel values forming an image from a plurality of sources and connecting or adding them.

As parameters to be set in the convolutional layer group included in the configuration 601 shown in FIG. image segmentation processing is possible. However, if the parameter settings for the layers and nodes that make up the neural network are different, the degree to which the tendencies trained from the teacher data can be reproduced in the output data may differ. In other words, in many cases, appropriate parameters differ depending on the mode of implementation, and can be changed to preferred values as necessary.

In addition to the method of changing the parameters as described above, there are cases where the CNN can obtain better characteristics by changing the configuration of the CNN. Better characteristics are, for example, higher accuracy of image segmentation processing, shorter time for image segmentation processing, shorter time for machine learning model training, and the like.

The configuration 601 of the CNN used in this embodiment has the function of an encoder consisting of a plurality of layers including a plurality of downsampling layers and the function of a decoder consisting of a plurality of layers including a plurality of upsampling layers. It is a net-type machine learning model. In the U-net type machine learning model, position information (spatial information) obscured in multiple layers configured as encoders is converted to the same dimensional layers (mutually corresponding layers) in multiple layers configured as decoders. ) (eg, using a skip connection).

Although not shown, as an example of changing the configuration of the CNN, for example, a batch normalization layer after the convolution layer or an activation layer using a rectifier linear unit may be incorporated. good.

When data is input to a trained model of such a machine learning model, data according to the design of the machine learning model is output. For example, output data that is highly likely to correspond to input data is output according to a tendency trained using teacher data.

In the trained model of the processing unit 222 according to this embodiment, when the tomographic image 401 is input, the boundary image 402 is output according to the tendency trained using the teacher data. The processing unit 222 can detect the retinal layers and their boundaries in the tomographic image 401 based on the boundary image 402 .

As shown in FIGS. 5(b) and 5(c), when the image area is divided and learned, the processing unit 222 uses the learned model to obtain a boundary image corresponding to each rectangular area. Get a rectangular area image. Therefore, the processing unit 222 can detect the retinal layer in each rectangular area. In this case, the processing unit 222 arranges each of the rectangular area image groups, which are boundary images obtained using the trained model, in the same positional relationship as each of the rectangular area image groups and combines them. , a boundary image 402 corresponding to the input tomographic image 401 can be generated. Also in this case, the processing unit 222 can detect the retinal layers and their boundaries in the tomographic image 401 based on the generated boundary image 402 .

In step S304, when the processing unit 222 performs the retinal layer detection process, the process proceeds to step S305. In step S<b>305 , the display control unit 25 displays the boundary detected by the processing unit 222 and the tomographic image on the display unit 50 . Here, an example of a screen displayed on the display unit 50 is shown in FIG.

FIG. 7 shows a display screen 700, which includes an SLO image 701, a thickness map 702 superimposed on the SLO image 701, an En-Face image 703, a tomographic image 711, and a retinal thickness graph. 712 is shown. Boundaries 715 and 716 of the retina are superimposed on the tomographic image 711 .

In this embodiment, the range of the retina is from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the retinal pigment epithelium layer L4, and the boundaries 715 and 716 correspond to the boundary L1 and the retinal pigment epithelium layer L4, respectively. The range of the retina is not limited to this, and may be, for example, the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the choroid L6. In this case, the boundaries 715 and 716 may correspond to the boundary L1 and the choroid L6, respectively. can.

A retinal thickness graph 712 is a graph showing the retinal thickness obtained from the boundaries 715 and 716 . A thickness map 702 is a color map representing the thickness of the retina obtained from the boundaries 715 and 716 . Note that although FIG. 7 does not show color information corresponding to the thickness map 702 for the sake of explanation, the thickness map 702 actually indicates the thickness of the retina corresponding to each coordinate in the SLO image 701 in a corresponding color. Can be displayed according to the map.

The En-Face image 703 is a front image generated by projecting the data of the range between the boundaries 715 and 716 in the XY directions. The front image is at least a partial depth range of volume data (three-dimensional tomographic image) obtained using optical interference, and is two-dimensional data corresponding to a depth range determined based on two reference planes. It is generated by projecting or integrating onto a plane. The En-Face image 703 according to the present embodiment is a two-dimensional plane image of data corresponding to the depth range (the depth range between the boundaries 715 and 716) determined based on the detected retinal layers in the volume data. It is a front image generated by projecting to . As a method of projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a representative value of data within the depth range is used as a pixel value on the two-dimensional plane. method can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within the range in the depth direction of the area surrounded by the two reference planes.

Also, the depth range related to the En-Face image 703 displayed on the display screen 700 is not limited to the depth range between the boundaries 715 and 716 . The depth range of the En-Face image 703 is, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries 715 and 716 regarding the detected retinal layers. There may be. In addition, the depth range of the En-Face image 703 is changed (offset) from the range between the two layer boundaries 715 and 716 of the detected retinal layers according to the operator's instruction. may be

Note that the front image displayed on the display screen 700 is not limited to the En-Face image based on the luminance value (En-Face image of luminance). The front image shown on the display screen 700 is, for example, a motion contrast front image generated by projecting or accumulating data corresponding to the above depth range on a two-dimensional plane for motion contrast data among a plurality of volume data. good too. Here, the motion contrast data is data indicating changes between a plurality of volume data obtained by controlling the measurement light to scan the same region (same position) of the subject's eye a plurality of times. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data indicating changes between a plurality of tomographic images obtained at approximately the same position at each different position. Note that the motion contrast frontal image is also called an OCTA frontal image (OCTA En-Face image) related to OCT angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also called OCTA data. Motion contrast data can be obtained, for example, as a decorrelation value between two tomographic images or their corresponding interference signals, a variance value, or a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value). , may be determined by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to scan the same region (same position) of the subject's eye a plurality of times.

In addition, the three-dimensional OCTA data (OCT volume data) used to generate the OCTA front image is generated using at least part of the volume data including the tomographic image for detecting the retinal layer and the common interference signal. may be In this case, volume data (three-dimensional tomographic image) and three-dimensional OCTA data can correspond to each other. As such, the three-dimensional motion contrast data corresponding to the volume data can be used to generate, for example, a motion contrast frontal image corresponding to a depth range determined based on the detected retinal layers.

The display of the thickness map 702, the En-Face image 703, the thickness graph 712, and the boundaries 715 and 716 may be generated by the image processing device 20 based on the boundaries and retinal layers detected by the processing unit 222. This is an example of what can be done. Any known method may be adopted as a method for generating these.

In addition to these, the display screen 700 of the display unit 50 may be provided with a patient tab, an imaging tab, a report tab, a setting tab, and the like. In this case, the contents shown on the display screen 700 of FIG. 7 are displayed on the report tab. The display screen 700 can also display a patient information display section, examination sort tabs, an examination list, and the like. Thumbnails of fundus images, tomographic images, and OCTA images may be displayed in the examination list.

Next, in step S<b>306 , the acquisition unit 21 acquires from the outside an instruction as to whether or not to end the series of processes related to the imaging of the tomographic image by the image processing system 1 . This instruction can be input by the operator using the input unit 60 . When the acquisition unit 21 acquires the instruction to end the process, the image processing system 1 ends the series of processes according to this embodiment. On the other hand, when the acquiring unit 21 acquires an instruction not to end the process, the process returns to step S302 to continue shooting.

As described above, the image processing apparatus 20 according to this embodiment includes the acquisition unit 21 and the processing unit 222 (first processing unit). Acquisition unit 21 acquires a tomographic image of an eye to be examined. The processing unit 222 uses the learned model to perform a first detection process for detecting at least one retinal layer among the plurality of retinal layers of the subject's eye in the tomographic image.

When image segmentation processing is performed using a trained model, boundary detection can be performed appropriately according to the learned tendency, for example, even with respect to changes in layer structure due to lesions in diseased eyes. Therefore, in the image processing apparatus 20 according to the present embodiment, by performing image segmentation processing using a trained model, boundary detection can be performed regardless of disease, site, etc., and the accuracy of boundary detection can be improved. be able to.

In addition, the image processing device 20 generates a front image corresponding to at least a partial depth range in the three-dimensional tomographic image of the subject's eye, which is determined based on the detected at least one retinal layer. The image processing unit 22 is further provided. The image processing unit 22 can generate a motion contrast front image corresponding to the determined depth range using the three-dimensional motion contrast data corresponding to the three-dimensional tomographic image.

In this embodiment, the configuration for performing image segmentation processing using one trained model has been described, but image segmentation processing may be performed using a plurality of trained models.

Since the trained model generates output data according to the tendency of learning using teacher data as described above, learning is performed using teacher data with similar tendencies in features, so that the tendency of learning for output data is obtained. Reproducibility can be improved. Therefore, for example, by performing image segmentation processing on tomographic images of corresponding imaging regions using a plurality of trained models that have been trained for each imaging region, it is possible to generate more accurate boundary images. . In this case, the image processing system can detect the retinal layers more accurately. In this case, it is also possible to increase the number of additional learning models, so it is possible to expect a version upgrade that gradually improves the performance.

Furthermore, the processing unit 222 uses a plurality of learning models that have been trained for each region such as the vitreous side region, the retinal region, and the sclera side region in the tomographic image, and processes the outputs of the respective learning models together. A final output of section 222 may be generated. In this case, a more accurate boundary image can be generated for each area, so that the retinal layer can be detected more accurately.

Also, in the present embodiment, a machine learning model that performs image segmentation processing has been described.

In general, the configuration of a machine learning model is not limited to the configuration that outputs an image corresponding to an image that is input data. For example, for input data, a machine learning model may be configured to output the types of output data trained using teacher data, or to output the probabilities for each of the types as numerical values. . For this reason, the format and combination of the input data and the output data of the pair group that constitutes the training data may be one of which is an image and the other of which is a numerical value, or one which is composed of a plurality of image groups and which is a character string, Both of them can be images, and can be made suitable for the form of use.

A specific example of teacher data for a machine learning model that estimates an imaging region is teacher data composed of pairs of a tomographic image acquired by OCT and an imaging region label corresponding to the tomographic image. . Here, the imaging region label is a unique numerical value or character string representing the region. When a tomographic image acquired using OCT is input to a trained model trained using such teacher data, the imaging region label of the region imaged in the image is output. A probability for each part label is output.

The processing unit 222 further uses such a trained model for estimating the imaging region to estimate the imaging region of the tomographic image, and uses the learned model corresponding to the estimated imaging region or the most probable imaging region. An image segmentation process may be performed. With such a configuration, even if the acquisition unit 21 cannot acquire the imaging conditions related to the imaging region of the tomographic image, the imaging region is estimated from the tomographic image, and the image segmentation process corresponding to the imaging region is performed. The retinal layer can be detected with high accuracy.

(Example 2)
In Example 1, an image segmentation process for detecting all target retinal layers from a tomographic image was performed using a trained model. On the other hand, in the second embodiment, boundary detection is performed using rule-based image features based on the detection result of the retinal area by the trained model.

Conventionally, in the optic papilla, it is customary to detect the open end of Bruch's membrane when detecting Cup (optic disc recession) and Disc (optic disc) using OCT images. In some cases, such as retinochoroidal atrophy, its detection was difficult.

In addition, in the conventional rule-based image segmentation processing, the robustness against individual differences and lesions of the eye to be examined is low, and the retinal region may be erroneously detected at first. In this case, subsequent detection of the inner retinal layer boundary could not be performed appropriately.

On the other hand, the accuracy of boundary detection can be improved by performing image segmentation processing using a machine learning model. However, when using a machine learning model based on a machine learning algorithm such as deep learning to recognize the imaged part and detect the boundary of the retinal layer, since this is a medical imaging field, the number of cases of normal and lesion images with correct answers is collected. It is generally very difficult to Furthermore, it takes time to create correct answer data for learning.

Therefore, in this embodiment, a trained model is used to detect a retinal region, and boundary detection based on image features is also used for the detected retinal region. As a result, erroneous detection of the retinal region is suppressed, and the detection accuracy of the inner retinal layer boundary is improved. can be done systematically.

The image processing system 8 according to the present embodiment will be described below with reference to FIGS. 8 to 13(d). Image processing by the image processing system according to the present embodiment will be described below, focusing on differences from the image processing according to the first embodiment. The configuration and processing of the image processing system according to the present embodiment, which are the same as those of the image processing system 1 according to the first embodiment, are denoted by the same reference numerals, and description thereof will be omitted.

FIG. 8 shows an example of a schematic configuration of the image processing system 8 according to this embodiment. In the image processing system 8 , a first processing section 822 and a second processing section 823 are provided instead of the processing section 222 in the image processing section 82 of the image processing device 80 .

The first processing unit 822 has a learned model related to a machine learning model based on a machine learning algorithm such as deep learning, and uses the learned model to detect the retinal region in the tomographic image. The second processing unit 823 performs rule-based determination of the result of image feature extraction for the retinal region detected by the first processing unit 822 to detect the boundary of the retinal layer.

Next, a series of processes according to this embodiment will be described with reference to FIGS. FIG. 9A is a flow chart of a series of processes according to this embodiment, and FIG. 9B is a flow chart of boundary detection processing in this embodiment. Note that the processing other than the boundary detection processing is the same as the processing in the first embodiment, so the description is omitted. After the tomographic image is generated in step S303, the process proceeds to step S904.

When the boundary detection process in step S904 is started, the process proceeds to step S941. In step S941, the first processing unit 822 detects the retinal region in the tomographic image using the learned model as first boundary detection processing.

Here, a machine learning model according to this embodiment will be described with reference to FIGS. The learning data (teacher data) of the machine learning model according to this embodiment is composed of one or more pairs of input data and output data. As an example of training data, a pair of a tomographic image 1001 obtained by OCT imaging shown in FIG. 10A and a label image 1002 obtained by labeling an arbitrary layer from the tomographic image 1001 shown in FIG. 10B. Teacher data or the like composed of groups can be mentioned.

Here, the labeled image is an image (image obtained by annotation) in which each pixel is labeled. is an image given In the label image 1002, a label 1003 on the superficial layer side (vitreous body side) of the retina, a label 1004 on the inner layer of the retina, and a label 1005 on the deep layer side (choroidal side) of the retina are provided. The first processing unit 822 in this embodiment detects the inner retina based on such label images. In this embodiment, the range of the retina (the range of the inner layer of the retina) is defined as the boundary L1 between the inner limiting membrane and the nerve fiber layer to the retinal pigment epithelium layer L4, but it is not limited to this. For example, the range of the retina is the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the junction of the inner segment and outer segment of the photoreceptor L3, the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the Bruch's membrane L5, or the inner It may be defined as the range from the boundary L1 to the choroid L6 between the limiting membrane and the nerve fiber layer.

Furthermore, although FIGS. 10A and 10B show an example of one XZ section within the XY plane of the retina, the section is not limited to this. Although not shown, arbitrary multiple XZ cross sections in the XY plane can be learned in advance so that cross sections photographed with various different scan patterns such as raster scan and radial scan can be handled. can. For example, when using data such as tomographic images obtained by three-dimensionally photographing the retina by raster scanning, volume data obtained by aligning a plurality of adjacent tomographic images can be used as teacher data. In this case, it is possible to generate a group of paired images at arbitrary angles from one volume data and one corresponding three-dimensional label data (three-dimensional label image). In addition, the machine learning model may be learned using images actually captured with various scan patterns as teacher images.

Next, images during learning will be described. A group of images forming a group of pairs of a tomographic image 1001 and a label image 1002, which constitute teacher data of a machine learning model, are created from rectangular area images of a fixed pixel size corresponding in positional relationship. Creation of the image will be described with reference to FIGS.

First, a case will be described in which a tomographic image 1001 and a label image 1002 are used as one of pairs constituting teacher data. In this case, as shown in FIG. 11A, a rectangular area image 1101 that is the entire tomographic image 1001 is used as input data, and a rectangular area image 1102 that is the entire label image 1002 is used as output data to form a pair. . In the example shown in FIG. 11(a), the entire image constitutes a pair of input data and output data, but the pair is not limited to this.

For example, as shown in FIG. 11B, a rectangular area image 1111 in the tomographic image 1001 is used as input data, and a rectangular area image 1113 corresponding to the imaging area in the label image 1002 is used as output data to form a pair. good too. Rectangular area images 1111 and 1113 are based on A-scan units. The A-scan unit may be one A-scan unit or several A-scan units.

Note that although FIG. 11B is based on the A-scan unit, it is also possible to provide portions outside the rectangular area above and below the image instead of making the entire area in the depth direction of the image. That is, the horizontal size of the rectangular area may be set to several A scans, and the depthwise size of the rectangular area may be set smaller than the size of the image in the depthwise direction.

Also, as shown in FIG. 11C, a rectangular area image 1121 in the tomographic image 1001 is used as input data, and a rectangular area image 1123 corresponding to the imaging area in the label image 1002 is used as output data to form a pair. good too. In this case, the size of the rectangular area is set to include a plurality of labels within one rectangular area.

During learning, the scan range (shooting angle of view) and scan density (the number of A scans) can be normalized to uniform the image size, and the size of the rectangular area during learning can be uniformed. Also, the rectangular area images shown in FIGS. 11A to 11C are examples of rectangular area sizes when each is learned separately.

The number of rectangular regions can be set to one in the example shown in FIG. 11(a), and multiple in the examples shown in FIGS. 11(b) and 11(c). For example, in the example shown in FIG. 11B, a pair may be formed by using the rectangular area image 1112 of the tomographic image 1001 as input data and the rectangular area image 1114, which is the corresponding imaging area in the label image 1002, as output data. can. Further, in the example shown in FIG. 11C, for example, a rectangular area image 1122 in the tomographic image 1001 is used as input data, and a rectangular area image 1124 corresponding to the imaging area in the label image 1002 is used as output data to form a pair. can also In this way, a pair of rectangular area images different from each other can be created from each pair of tomographic image and label image. Note that by creating a large number of pairs of rectangular area images while changing the position of the area to different coordinates in the original tomographic image and label image, it is possible to enrich the pair group that constitutes the teacher data.

Although the examples shown in FIGS. 11B and 11C show discrete rectangular regions, in reality, the original tomographic images and label images are continuously formed into rectangular regions of a constant image size without gaps. It can be divided into groups of images. Also, the original tomographic image and label image may be divided into rectangular area image groups at random positions corresponding to each other. In this way, by selecting an image of a smaller area as a rectangular area (or strip area) as a pair of input data and output data, many pairs can be obtained from the tomographic image 1001 and the label image 1002 that constitute the original pair. Can generate data. This reduces the time it takes to train machine learning models. On the other hand, the trained model of the completed machine learning model tends to take a long time to perform image segmentation processing.

Next, as an example of the machine learning model according to the present embodiment, the configuration of a convolutional neural network (CNN) that performs segmentation processing on an input tomographic image will be described with reference to FIG. FIG. 12 shows an example of a machine learning model configuration 1201 in the first processing unit 822 . As the machine learning model according to the present embodiment, for example, FCN, SegNet, or the like can be used as in the first embodiment. Also, depending on the desired configuration, a machine learning model that performs object recognition on a region-by-region basis as described in the first embodiment may be used.

The machine learning model shown in FIG. 12 is composed of a plurality of layer groups that process and output a group of input values, similarly to the example of the machine learning model according to the first embodiment shown in FIG. The types of layers included in the configuration 1201 of the machine learning model include convolutional layers, downsampling layers, upsampling layers, and synthetic layers. Note that the configuration of these layers and the modification of the configuration of the CNN are the same as those of the machine learning model according to the first embodiment, so detailed descriptions thereof will be omitted. The CNN configuration 1201 used in this embodiment is a U-net machine learning model, like the CNN configuration 601 described in the first embodiment.

In the trained model of the first processing unit 822 according to this embodiment, when the tomographic image 1001 is input, the label image 1002 is output according to the tendency trained using the teacher data. The first processing unit 822 can detect the retinal region in the tomographic image 1001 based on the label image 1002 .

Note that, as shown in FIGS. 11B and 11C, when an image region is divided for learning, the first processing unit 822 uses a trained model to correspond to each rectangular region. Obtain a rectangular area image that is a label image. Therefore, the first processing unit 822 can detect the retinal layer in each rectangular area. In this case, the first processing unit 822 arranges and combines each of the rectangular area image groups, which are label images obtained using the trained model, in the same positional relationship as each of the rectangular area image groups. do. Thereby, the first processing unit 822 can generate the label image 1002 corresponding to the input tomographic image 1001 . Also in this case, the first processing unit 822 can detect the retinal region in the tomographic image 1001 based on the generated label image 1002 .

In step S941, when the retinal area is detected by the first processing unit 822, the process proceeds to step S942. In step S942, the second processing unit 823 performs rule-based image segmentation based on the retinal region detected by the first processing unit 822 in the tomographic image 1001 shown in FIG. 10A as second detection processing. The processing detects remaining boundaries in the inner retina.

Second boundary detection processing by the second processing unit 823 will be described with reference to FIGS. FIG. 13A shows a tomographic image 1001, which is an example of a tomographic image to be input. FIG. 13B shows a label image 1002 output by the first processing unit 822, which is an image to which a label 1004 for the retina area and labels 1003 and 1005 corresponding to other areas are added. The second processing unit 823 according to the present embodiment sets the range of the retina area indicated by the label 1004 in the label image 1002 as the target area for layer detection.

The second processing unit 823 can detect the boundary of interest by detecting the contour within the retinal region indicated by the label 1004 in the label image 1002 . FIG. 13C shows an edge-enhanced image 1303 processed by the second processing unit 823 . Processing by the second processing unit 823 will be described below. As shown in FIGS. 13(c) and 13(d), the second processing unit 823 does not perform boundary detection for the optic papilla because the retinal layer is interrupted.

The second processing unit 823 performs noise removal and edge enhancement processing on the region corresponding to the label 1004 in the tomographic image 1001 to be processed. The second processing unit 823 applies, for example, a median filter or a Gaussian filter as noise removal processing. Also, the second processing unit 823 applies a Sobel filter or a Hessian filter as edge enhancement processing.

Edge enhancement processing for a two-dimensional tomographic image using a two-dimensional Hessian filter will now be described. The Hessian filter can emphasize the second-order local structure of the two-dimensional gray distribution based on the relationship between the two eigenvalues (λ ₁ , λ ₂ ) of the Hessian matrix. Therefore, in this embodiment, the relationship between the eigenvalues of the Hessian matrix and the eigenvectors (e ₁ , e ₂ ) is used to emphasize the two-dimensional line structure. Since the line structure in the two-dimensional tomographic image of the eye to be examined corresponds to the structure of the retinal layers, the application of the Hessian filter can emphasize the structure of the retinal layers.

In order to detect retinal layers with different thicknesses, it is sufficient to change the resolution of smoothing by the Gaussian function performed when calculating the Hessian matrix. When applying the two-dimensional Hessian filter, it can be applied after transforming the data so as to match the XZ physical size of the image. In general OCT, the physical sizes in the XY direction and the Z direction are different. Therefore, the filter is applied according to the physical size of the retinal layer for each pixel. Since the physical sizes in the XY and Z directions can be grasped from the design/configuration of the OCT apparatus 10, it is possible to transform the tomographic image data based on the physical sizes. Also, if the physical size is not normalized, it can be approximated by changing the smoothing resolution using the Gaussian function.

Although processing with a two-dimensional tomographic image has been described above, the object to which the Hessian filter is applied is not limited to this. If the data structure for capturing a tomographic image is a three-dimensional tomographic image obtained by raster scanning, it is possible to apply a three-dimensional Hessian filter. In this case, after the image processing unit 82 performs registration processing in the XZ direction between adjacent tomographic images, the second processing unit 823 calculates three eigenvalues (λ ₁ , λ ₂ , λ ₃ ) of the Hessian matrix. Based on the relationship, the secondary local structure of the three-dimensional gray distribution can be emphasized. Therefore, by emphasizing the three-dimensional layered structure using the relationship between the eigenvalues of the Hessian matrix and the eigenvectors (e ₁ , e ₂ , e ₃ ), it is also possible to enhance the edge three-dimensionally.

In the edge-enhanced image 1303 , the edge-enhanced portion appears as a white line 1304 . Note that an area in the tomographic image 1001 that does not correspond to the label 1004 can be treated as an area in which edge detection is not performed. Also, here, a configuration for performing edge enhancement processing using a Hessian filter has been described, but the processing method for edge enhancement processing is not limited to this, and any existing method may be used.

FIG. 13D shows a boundary image 1305 indicating the boundary of the retinal layers detected by the second processing unit 823 using the label image 1002 and the edge-enhanced image 1303. FIG. In the boundary image 1305, a black line 1306 indicates an example of the boundary. Processing for detecting the boundary of the retinal layers from the label image 1002 and the edge-enhanced image 1303 by the second processing unit 823 will be described below.

A second processing unit 823 detects an edge-enhanced boundary from the edge-enhanced image 1303 . In this embodiment, since the first processing unit 822 has already detected the boundary between the ILM and NFL and the RPE, the second processing unit 823 continues to detect the ISOS, NFL and GCL boundaries. Although not shown, other boundaries include the boundary between the outer plexiform layer (OPL) and the outer nuclear layer (ONL), the boundary between the inner plexiform layer (IPL) and the inner nuclear layer (INL), and the boundary between the INL and the OPL. A boundary, a boundary between GCL and IPL, and the like may be detected.

As a boundary detection method, a plurality of locations with strong edge strength are detected as boundary candidates in each A-scan, and points (locations with strong edge strength) are drawn as lines based on the continuity between boundary candidates in adjacent A-scans. Perform the process of connecting. Further, the second processing unit 823 can remove outliers by evaluating the smoothness of the line when the points are connected as a line. More specifically, for example, the Z-direction positions of the connected points are compared, and if the difference in Z-direction position is greater than a predetermined threshold value, the newly connected point is determined as an outlier, It can be excluded from the connecting process. Further, when outliers are removed, boundary candidates in A-scans adjacent to the A-scan position of the excluded point may be connected as a line. Note that the method of removing outliers is not limited to this, and any existing method may be used.

The second processing unit 823 determines a corresponding boundary for each line formed by connecting points based on the vertical distance in the Z direction and the positional relationship of the boundary of the retinal layers. If there is no boundary detected as a result of removing outliers in each A-scan, interpolation may be performed from surrounding boundaries. Alternatively, a boundary candidate may be searched for in the horizontal direction (X or Y direction) by relying on edges from the surrounding boundary, and the boundary may be determined again based on the boundary candidate searched for from the surrounding boundary. .

After that, the second processing unit 823 executes a process of smoothly correcting the shape of the detected boundary. For example, an active contour model such as Snakes or Level Set method may be used to smooth the shape of the boundary using image features and shape features. Alternatively, the coordinate values of the boundary shape may be regarded as time-series data by a signal, and the shape may be smoothed by smoothing processing such as a Savitzky-Golay filter, simple moving average, weighted moving average, or exponential moving average.

Through such processing, the second processing unit 823 can detect the retinal layers within the retinal region detected by the first processing unit 822 . Note that the detection processing of the retinal layers by the second processing unit 823 described above is an example, and the retinal layers can also be detected using any existing segmentation processing. When the second processing unit 823 detects the retinal layer, the process proceeds to step S305. Since subsequent processing is the same as that of the first embodiment, description thereof is omitted.

As described above, the image processing apparatus 80 according to this embodiment includes the acquisition unit 21 , the first processing unit 822 and the second processing unit 823 . Acquisition unit 21 acquires a tomographic image of an eye to be examined. The first processing unit 822 uses the learned model to perform a first detection process for detecting at least one retinal layer among the plurality of retinal layers of the subject's eye in the tomographic image. The second processing unit 823 executes second detection processing for detecting at least one retinal layer among the plurality of retinal layers in the tomographic image without using the trained model.

Specifically, the second processing unit 823 uses the second detection process to detect at least one retinal layer other than the at least one retinal layer detected by the first detection process. In particular, in this embodiment, the first detection processing is the layer from the boundary between the inner limiting membrane and the nerve fiber layer of the eye to be examined to the junction of the inner segment and outer segment of the photoreceptor, the retinal pigment epithelium layer, and any one of Bruch's membrane. This is a process for detecting Also, the second detection process is performed after the first detection process and is included in the layers detected by the first detection process, that is, the process of detecting at least one retinal layer between the detected layers. .

The image processing apparatus 80 according to the present embodiment can perform boundary detection regardless of disease, body part, or the like. In addition, the accuracy of boundary detection can be improved by also using boundary detection based on image features for the area output by the machine learning model. Furthermore, in the process of machine learning, it is only necessary to create correct data for the retinal layer or other layers at the time of learning, so learning can also be performed efficiently.

In addition, as the number of boundaries detected by the machine learning model increases, the possibility of false detection occurring in the output label image and boundary image may increase. On the other hand, in the detection of the retinal region using the machine learning model according to the present embodiment, since the number of boundaries to be detected is small, erroneous detection in the output label image and boundary image can be suppressed.

Also in this embodiment, as in the first embodiment, the first processing unit 822 may be configured to detect the retinal region using a plurality of machine learning models. In this case, the accuracy of detection of the retinal area can be improved. In addition, since it is also possible to increase the number of additional learning models, it is also possible to expect version upgrades that gradually improve performance.

(Modification of Example 2)
In the second embodiment described above, prior to detection of the retinal boundary by the second processing unit 823 based on the rule base, the first processing unit 822 detects the retinal area using the learned model as a preliminary step. showed that. However, the order of the processing by the first processing unit 822 and the processing by the second processing unit 823 is not limited to this. For example, if the first processing unit 822 takes a long time to detect the retinal area, the second processing unit 823 can first execute the retinal area detection processing based on a rule.

In such a case, the second processing unit 823 first detects the boundary between the ILM and the NFL and the RPE or ISOS using a method similar to the method by the second processing unit 823 shown in the second embodiment. do. This is because these boundaries are located in the superficial and deep layers of the retina where the luminance values are high. When detecting the boundary between the ILM and NFL and the RPE or ISOS, features are more likely to appear than other boundaries, so these boundaries may be detected based on an image with large blur that has been subjected to noise processing several times. . In this case, since only global features can be detected, erroneous detection of other boundaries can be prevented. Alternatively, the tomographic image may be binarized using a dynamic threshold to limit the retinal region, and the boundary between the ILM and NFL and the RPE or ISOS may be detected from the region. Note that the second processing unit 823 may detect the boundary between the ILM and the NFL and the BM.

However, when detecting the retinal region, which is the primary detection target, using such a rule base, as described above, the robustness against individual differences in the eye to be examined and lesions is low, and the retinal region is initially misdetected. There is In this case, subsequent detection of the inner retinal layer boundary may not be performed appropriately.

As a countermeasure, in this modified example, the image processing unit 82 sets parameters for checking retinal region erroneous detection, such as the discontinuity of the retinal region boundary, the local curvature, or the variance of the boundary coordinates in the local region, as a predetermined threshold value. compare. If these parameters exceed predetermined thresholds, the image processing unit 82 determines that the detection of the retina area in the second processing unit 823 is an erroneous detection. Then, when the image processing unit 82 determines that the detection of the retina area by the second processing unit 823 is an erroneous detection, the first processing unit 822 is configured to detect the retina area.

According to this modification, even if it takes a long time to detect the retinal region by the first processing unit 822, the substantial processing waiting time of the examiner (operator) who performs many examinations can be reduced. Robustness against individual differences and lesions of the eye to be examined can be ensured.

Also, in this modified example, the configuration in which the processing of the second processing unit 823 is performed prior to the processing of the first processing unit 822 has been described, but these processes may be started at the same time. In this case, when the image processing unit 82 determines that the detection of the retina area by the second processing unit 823 is an erroneous detection, the second processing unit waits for the detection of the retina area by the first processing unit 822 and 823 performs inner retinal boundary detection. Note that when the second processing unit 823 properly detects the retinal area, the processing by the first processing unit 822 is interrupted or the processing result by the first processing unit 822 is discarded. be able to.

Further, when the first processing unit 822 and the second processing unit 823 detect the retinal region (the same retinal layer), the display control unit 25 detects the detection by the first processing unit 822 and the second processing unit 823. A processing result of the processing may be displayed on the display unit 50 . In this case, according to the operator's instruction for the processing result displayed on the display unit 50, the second processing is performed on either the processing result of the detection processing by the first processing unit 822 or the second processing unit 823. A unit 823 may perform boundary detection of the inner retina. In this case, the process of detecting the retinal area by the second processing unit 823 may be defined as the second detection process, and the subsequent process of detecting the boundary of the inner retina by the second processing unit 823 may be defined as the third detection process. .

(Example 3)
In Example 2, an example of detecting a retinal region using a trained model and detecting a boundary of the inner layer of the retina with respect to the detected retinal region was shown. On the other hand, in the present embodiment, the area to be detected using the trained model is not limited to the retinal area, and a characteristic area of the photographed part of the image that is the input data is detected.

Image processing by the image processing system according to the present embodiment will be described below, focusing on differences from the image processing according to the second embodiment. Since the configuration and processing procedure of the image processing system according to the present embodiment are the same as those of the image processing system 8 according to the second embodiment, the same reference numerals are used to denote them, and the description thereof is omitted. .

A region detected by the first processing unit 822 according to the present embodiment using the learned model will be described with reference to FIGS. FIGS. 14A to 14D show an example of an image obtained by photographing each part of the subject's eye and a label image as a result of processing by the first processing unit 822. FIG.

FIG. 14(a) shows a tomographic image 1401 when the macula is imaged and a label image 1402 in the macula obtained using the learned model. Label image 1402 shows vitreous label 1403 , ILM to ISOS range label 1404 , ISOS to RPE range label 1405 , choroid label 1406 , and sclera label 1407 .

Regarding the macula, for example, in order to capture the thickness of the entire retina due to hemorrhage or neovascularization, the loss of photoreceptor cells related to vision, or the thinning of the choroid due to pathological myopia, each area where morphological changes are likely to appear is labeled. Configure and pre-train a machine learning model. As training data, tomographic images of the macular region are used as input data, and for example, labeled images with a vitreous body label, a label ranging from ILM to ISOS, a label ranging from ISOS to RPE, a choroid label, and a sclera label. is the output data. As a result, the first processing unit 822 inputs the tomographic image of the macular region to the trained model, acquires the label image indicating the label for each region where the above-described morphological change is likely to appear, and A region can be detected.

FIG. 14(b) shows a tomographic image 1411 obtained by imaging the optic papilla and a labeled image 1412 in the optic papilla obtained using the learned model. Label image 1412 shows vitreous label 1413, label 1414 ranging from the ILM to the NFL and GCL boundary, and label 1415 ranging from the NFL and GCL boundary to the GCL and IPL boundary. Further, the label image 1412 shows a label 1416 ranging from the boundary between GCL and IPL to ISOS and a label 1417 ranging from ISOS to RPE. Also shown in the label image 1412 is a label 1418 for the RPE to deep area and a label 1419 for the lamina cribrosa area. In this case, tomographic images of the optic papilla are used as input data. Labels, labels from the boundary of the GCL and IPL to the ISOS range, labels from the ISOS to the RPE range, labels from the RPE to the deep layer range, and labeled images of the lamina cribrosa range are output data.

A front image 1421 in FIG. 14C is an image viewed from the front in the XY plane during imaging of the optic papilla, and is an image captured using the fundus image capturing device 30 . A labeled image 1422 in FIG. 14C is a labeled image obtained by using a trained model for the front image 1421 of the optic papilla. A label image 1422 shows a label 1423, a Disc label 1424, and a Cup label 1425 for the periphery of the optic nerve papilla.

In the optic papilla, loss of ganglion cells and morphological changes such as RPE tip (RPE-tip) or Bruch's membrane opening (BMO), cribriform plate, and Cup and Disc are likely to appear due to glaucoma. Therefore, a label is set for each of these regions and the machine learning model is trained in advance. As the learning data, the front image of the optic papilla is used as input data, and the labeled image with, for example, the peripheral portion of the optic papilla, the Disc label, and the Cup label is used as output data. As a result, the first processing unit 822 inputs the image of the optic papilla to the trained model, acquires the label image indicating the label for each region where the above-described morphological change is likely to appear, and obtains the label image for each label unit. A region can be detected.

FIG. 14D shows a tomographic image 1431 when the anterior segment is captured and a label image 1432 in the anterior segment captured using the learned model. Label image 1432 shows cornea label 1433 , anterior chamber label 1434 , iris label 1435 , and lens label 1436 . Regarding the tomographic image 1431 of the anterior segment, unlike the posterior segment image, the machine learning model is trained in advance on the main regions as described above. As learning data, a tomographic image of the anterior segment is used as input data, and labeled images with, for example, a cornea label, an anterior chamber label, an iris label, and a lens label are used as output data. As a result, the first processing unit 822 inputs an image of the anterior segment to the trained model, acquires a label image indicating a label for each region where the above-described morphological change is likely to occur, and obtains a label image for each label unit. A region can be detected.

The second processing unit 823 according to the present embodiment, in the tomographic images 1401, 1411, and 1431 shown in FIGS. to detect the remaining boundaries. Also, the second processing unit 823 may measure the thickness of the detected boundary, the layer region sandwiched between the boundaries, or the region detected by the first processing unit 822 .

In addition, the second processing unit 823 measures each area classified by the label for the front image of FIG. can be calculated. Any existing method may be used for these measurements and ratio calculations.

As described above, the second processing unit 823 according to the present embodiment executes the image processing algorithm corresponding to the area detected by the first processing unit 822, and sets rules for applying the image processing algorithm to each area. can be changed. The rule here includes, for example, the type of boundary to be detected. For example, the second processing unit 823 performs additional boundary detection in the range from ILM to ISOS indicated by the label 1404 in the tomographic image 1401 of the macular portion shown in FIG. can be Note that the boundary detection method may be the same as the boundary detection method in the first embodiment. Further, the second processing unit 823 detects the boundary between ILM and NFL, the boundary between OPL and ONL, the boundary between IPL and INL, the boundary between INL and OPL, and the boundary between GCL and IPL in the tomographic image 1401 of the macular region. Image processing algorithms and rules may be applied as follows. Thus, in the second processing unit 823, arbitrary image processing algorithms and rules may be set for each region according to a desired configuration.

As described above, in the image processing apparatus 80 according to the present embodiment, the first processing unit 822 detects a predetermined boundary for each imaging region in the input image. For this reason, in the present embodiment, the region detected using the trained model is not limited to the retinal region, but is a region characteristic of the imaging site, so variations such as diseases can be dealt with. Note that, when the input image is a tomographic image of the retina, the first processing unit 822 performs the first detection processing for detecting at least one of the retinal layers, which is a predetermined boundary for each imaging region. can be detected. In addition, when the input image is an image other than the tomographic image of the retina, the first processing unit 822 detects a predetermined boundary for each imaging part as a process different from the first detection process. can do. As in the first embodiment, the final output of the first processing unit 822 may be generated by combining the respective outputs of a plurality of learning models that have learned for each region of the tomographic image.

Further, in the image processing apparatus 80 according to the present embodiment, the height (thickness), width, area, Cup/Disc Certain shape features, such as ratios, can be measured.

(Example 4)
In the third embodiment, an example of detecting a characteristic area in a part to be imaged without being limited to the retinal area as the area to be detected using the trained model is shown. On the other hand, in the fourth embodiment, selection of execution of processing using the learned model and narrowing down of areas to be detected using the learned model are performed according to the shooting conditions under which the image was shot.

Image processing by the image processing system 150 according to the present embodiment will be described below with reference to FIGS. 15 to 16B, focusing on differences from the image processing according to the second embodiment. The configuration and processing of the image processing system according to the present embodiment, which are the same as those of the image processing system 8 according to the second embodiment, are denoted by the same reference numerals, and description thereof will be omitted.

FIG. 15 shows an example of a schematic configuration of an image processing system 150 according to this embodiment. In the image processing system 150 according to the present embodiment, in the image processing unit 1520 of the image processing device 152, in addition to the tomographic image generation unit 221, the first processing unit 822, and the second processing unit 823, the selection unit 1524 is provided.

The selection unit 1524 selects image processing to be performed on the tomographic image based on the imaging condition group acquired by the acquisition unit 21 and learning contents (teacher data) regarding the learned model of the first processing unit 822 . Specifically, based on the imaging condition group and learning of the learned model, the retinal layer is detected only by the first processing unit 822, or the retinal area is detected by the first processing unit 822, and the second processing is performed. It is selected whether the retinal layer is detected by the unit 823 or the retinal layer is detected only by the second processing unit 823 . Further, when the first processing unit 822 has a plurality of learned models, the selection unit 1524 selects the first processing unit A trained model to be used for the detection process by 822 can be selected.

Next, a series of processes according to this embodiment will be described with reference to FIGS. FIG. 16(a) is a flow chart of a series of processes according to this embodiment, and FIG. 16(b) is a flow chart of boundary detection processing according to this embodiment. Note that the processing other than the boundary detection processing is the same as the processing in the second embodiment, so description thereof will be omitted. After the tomographic image is generated in step S303, the process proceeds to step S1604. When the process proceeds to step S1604, boundary detection processing is started, and the process proceeds to step S1641.

In step S1641, the acquisition unit 21 acquires an imaging condition group for the generated tomographic image, and the image processing unit 1520 performs processing by the first processing unit 822 and the second processing unit 823 from the acquired imaging condition group. Get the information needed to perform the selection of . For example, these conditions can include the imaging region, imaging method, imaging region, imaging angle of view, image resolution, and the like.

In step S1642, the selection unit 1524 selects whether or not to execute processing by the first processing unit 822 based on the imaging conditions acquired in step S1641. Here, as an example, for the trained model of the first processing unit 822, a model for the optic papilla and a model for the macula, which are learned using teacher data for each imaging region of the optic papilla and the macula. Consider the case where only two are available. Also, in this case, the first processing unit 822 will be described as being incapable of handling wide-angle images (images in a range in which both the optic papilla and the macula are captured).

In this example, when the selection unit 1524 determines that the input image is an image obtained by imaging the optic nerve papilla or the macular region alone based on, for example, the imaging region name and imaging angle of view information among the imaging conditions, The selection unit 1524 selects execution of processing by the first processing unit 822 . As a result, the process moves to step S1643. On the other hand, if the selection unit 1524 determines that the input image is an image captured other than the optic disc region and the macular region, or an image with a wide angle of view including both the optic disc region and the macular region, the selection unit 1524 selects not to execute processing by the first processing unit 822 . As a result, the process moves to step S1645.

In step S1643, the selection unit 1524 selects an appropriate learned model to be used by the first processing unit 822 based on the imaging conditions acquired in step S1641. In the above example, if the selection unit 1524 determines that the input image is an image obtained by imaging the optic papilla based on the information of the imaging region name and the imaging angle of view, for example, the model for the optic papilla is selected. select. Similarly, when the selection unit 1524 determines that the input image is an image of the macular region, it selects a model for the macular region.

Here, an example in which the trained model has learned only images obtained by photographing the optic papilla and the macula has been shown, but the learning content of the trained model is not limited to this. For example, a trained model that has been trained on other parts, or a trained model that has been trained using a wide-angle image including the optic papilla and the macula may be used.

Further, when trained models are separately prepared according to the imaging method instead of the imaging region, selection of processing and selection of the learned model may be performed according to the imaging method. Examples of imaging methods include SD-OCT and SS-OCT imaging methods, and image quality, imaging range, depth of penetration in the depth direction, and the like differ depending on the difference between the two imaging methods. Therefore, appropriate processing or learned models may be selected for images captured by different imaging methods. It should be noted that if the learning is done together regardless of the shooting method, there is no need to change the processing according to the shooting method. Also, if there is only one trained model, there is no need to select a learned model in step S1643, so this process can be skipped.

In step S1644, the first processing unit 822 performs first boundary detection processing using the learned model selected in step S1643. Note that the processing described in the first to third embodiments can be used for this processing. For example, when the macular model has learned about image segmentation processing for each retinal layer in the macular, the first processing unit 822 performs detection as the first detection processing in the same manner as in the first embodiment. All boundaries of interest can be detected. Further, for example, when the optic nerve papilla model has learned processing for detecting the retinal region in the optic papilla, the first processing unit 822 performs the first detection processing in the same manner as in the second embodiment. In addition, the retinal area can be detected. Similarly, when the macular model has learned the process of detecting a characteristic region of the macular as in the third embodiment, the first processing unit 822 performs the following as the first detection process: As in the third embodiment, characteristic regions can be detected. Note that a specific detection method is the same as the detection method in Examples 1 to 3, and thus description thereof is omitted.

In step S1645, the selection unit 1524 selects whether or not to execute processing by the second processing unit 823 based on the imaging conditions acquired in step S1641. If the selection unit 1524 selects execution of the process by the second processing unit 823, the process moves to step S1646. On the other hand, if the selection unit 1524 selects not to execute the process by the second processing unit 823, the process of step S1604 ends and the process proceeds to step S305.

Here, an example of selection processing by the selection unit 1524 in step S1645 will be described. The case where the processing by the second processing unit 823 is executed is, for example, as described in the second and third embodiments, based on the region detected by the first processing unit 822, the second processing unit 823 detects the boundary is detected.

In addition, processing by the second processing unit 823 is also executed when the first processing unit 822 receives an unlearned image. In this case, since skipping the processing by the first processing unit 822 is selected in step S1642, the second processing unit 823 performs rule-based image segmentation processing without using a trained model. Perform boundary detection.

On the other hand, the case where the processing by the second processing unit 823 is not executed is, for example, the case where the first processing unit 822 can detect all target boundaries using the learned model. In this case, since the processing is completed only by the first processing unit 822, the processing of the second processing unit 823 can be skipped.

However, even if the first processing unit 822 can detect all target boundaries using the learned model, the second processing unit 823 measures the layer thickness based on the detected boundaries. , execution of the process by the second processing unit 823 may be selected. On the other hand, the layer thickness measurement and the like based on the detected boundary are not limited to being performed by the second processing unit 823 and may be performed by the image processing unit 1520 . Therefore, even when layer thickness measurement or the like is performed, execution of processing by the second processing unit 823 may not be selected.

In step S1646, the selection unit 1524 selects the image processing necessary for executing the processing by the second processing unit 823 and the rule for applying the image processing based on the imaging conditions acquired in step S1641. conduct. For example, when the input image is a tomographic image as shown in FIGS. Select the process and rules for detecting the boundaries of .

Specifically, for example, when the input image is a tomographic image 1401 obtained by imaging the macular portion shown in FIG. Further, for example, in the case of the tomographic image 1411 obtained by imaging the optic nerve papilla shown in FIG. Select image processing and rules that take into account the optic nerve head exception handling. Furthermore, for example, in the case of a tomographic image 1431 obtained by imaging the anterior segment shown in FIG.

Furthermore, in step S1647, in addition to detecting the layer boundary or independently, when measuring the thickness of the detected boundary or the thickness of the layer region sandwiched between the boundaries, the selection unit 1524 performs such an image measurement function. You can also select the image processing required for

In step S1647, the second processing unit 823 performs boundary detection and/or measurement of the detected boundary or area. As in the second and third embodiments, the description of the process of detecting the boundary in the area based on the area detected by the first processing unit 822 and the process of measuring the boundary will be omitted. .

Here, an example of processing performed by the second processing unit 823 when an image is input in which the trained model of the first processing unit 822 has not been trained will be described. In this case, since there is no retinal area candidate detected from the input image, the second processing unit 823 detects the boundary between the ILM and NFL, and the RPE or ISOS, as described in the modified example of the second embodiment. is detected first.

After detecting the boundary between the ILM and NFL and the RPE or ISOS, the second processing unit 823 detects the remaining boundary based on the area between these boundaries. Since the detection process is the same as the detection process in the second and third embodiments, description thereof is omitted. After the second processing unit 823 executes these processes, the process moves to step S305. As in the modification of the second embodiment, the second processing unit 823 may first detect the boundary between the ILM and NFL and the BM. Also, since the subsequent processing is the same as the processing in the first to third embodiments, the description is omitted.

As described above, in the image processing apparatus 152 according to the present embodiment, the acquisition unit 21 acquires the imaging conditions for the tomographic image of the subject's eye. The image processing device 152 further includes a selection unit 1524 that selects processing based on imaging conditions. A selection unit 1524 selects at least one of the first detection process and the second detection process based on the imaging conditions.

For this reason, in the image processing apparatus 152 according to the present embodiment, based on the shooting conditions, the possibility of boundary detection by the trained model executed by the first processing unit 822 and the image feature detection executed by the second processing unit 823 are determined. It is determined whether or not it is necessary to execute boundary detection processing. As a result, even when the boundary detection by the trained model corresponds only to a specific image, it is possible to appropriately perform processing according to the input image. Therefore, even if the learning model does not correspond to various image patterns, it is possible to reliably execute the boundary detection process. Therefore, by using at least one of the machine learning models in various maturity stages created by machine learning together with an image processing method that detects the boundaries of the retinal layers by judging the results of image feature extraction on a rule basis, The accuracy of boundary detection can be improved.

Also, the first processing unit 822 includes a plurality of trained models that have undergone machine learning using different teacher data. Furthermore, the first processing unit 822 executes the first detection process using a trained model that has been machine-learned using teacher data corresponding to the imaging conditions, among the plurality of trained models.

According to this embodiment, it is possible to detect the retinal layer using an appropriate learning model based on the imaging conditions, and perform more appropriate processing according to the input image. In addition, since it is also possible to increase the number of additional learning models, it is also possible to expect version upgrades that gradually improve performance. Furthermore, according to the selection unit 1524, image processing and rules to be used in the second processing unit 823 can be selected based on the shooting conditions, and more appropriate processing can be performed according to the input image.

In this embodiment, the selection of processing by the first processing unit 822 and the second processing unit 823 is performed separately in steps S1642 and S1645, but the procedure for selecting processing is not limited to this. For example, the selection unit 1524 selects, in one step, processing by only the first processing unit 822, processing by only the second processing unit 823, or processing by the first processing unit 822 and the second processing unit 823. It may be configured as

(Modification of Example 4)
In the fourth embodiment, appropriate processing can be performed in various cases, such as when the first processing unit 822 and the second processing unit 823 share the boundary detection or the like, or when only one of them can complete the task. showed that there is On the other hand, the first processing unit 822 and the second processing unit 823 may execute the same process, for example, the process of detecting the same boundary in parallel.

As such an example, for example, the first processing unit 822 can detect all target boundaries using the trained model, and the second processing unit 823 can detect the retinal region that is the first detection target. Consider the case where all the boundaries of interest can be detected. In this case, the first processing unit 822 and the second processing unit 823 output results of detecting their boundaries separately.

Since these detection results are the results of the processing by the trained model and the image processing by the rule base, respectively, there may be a difference between the two results. Therefore, in this modified example, the display control unit 25 can cause the display unit 50 to display these two results side by side, switch between them, or overlap them. Also, the image processing unit 1520 can determine whether the two results match or not, and the display control unit 25 can highlight the mismatched portion and display it on the display unit 50 . In this case, the reliability of layer detection can be indicated to the operator. Furthermore, the display control unit 25 may display the non-matching part on the display unit 50 so that a result with a higher degree of satisfaction can be selected according to the operator's instruction.

(Example 5)
In Examples 2 to 4, an example of detecting a retinal region using a trained model and detecting a boundary of the inner layer of the retina for the detected retinal region on a rule basis has been described. On the other hand, in Example 5, the region detected using the learned model is corrected based on the medical characteristics.

Image processing by the image processing system 170 according to the present embodiment will be described below with reference to FIGS. 17 to 19D, focusing on differences from the image processing according to the second embodiment. The configuration and processing of the image processing system according to the present embodiment, which are the same as those of the image processing system 8 according to the second embodiment, are denoted by the same reference numerals, and description thereof will be omitted.

FIG. 17 shows an example of a schematic configuration of an image processing system 170 according to this embodiment. The image processing unit 1720 of the image processing device 172 in the image processing system 170 according to the present embodiment includes a tomographic image generation unit 221, a first processing unit 822, and a second processing unit 823, and a correction unit 1724. is provided.

The correction unit 1724 corrects the labeled regions based on the medical characteristics of the eye in the labeled image obtained by the first processing unit 822 using the trained model. This allows the image processing unit 1720 to more appropriately detect the retinal area and the characteristic area.

Next, a series of processes according to this embodiment will be described with reference to FIGS. 18(a) and 18(b). FIG. 18(a) is a flow chart of a series of processes according to this embodiment, and FIG. 18(b) is a flow chart of boundary detection according to this embodiment. Note that the processing other than the boundary detection processing is the same as the processing in the second embodiment, so description thereof will be omitted. After the tomographic image is generated in step S303, the process proceeds to step S1804. When the process moves to step S1804 and the process by the first processing unit 822 is executed in step S941, the process moves to step S1841.

In step S1841, the correction unit 1724 corrects the retinal area detected by the first processing unit 822 in step S941. More specifically, the first processing unit 822 corrects the labeled region based on the medical characteristics of the eye in the labeled image obtained using the trained model.

Here, with reference to FIGS. 19A to 19D, correction processing by the correction unit 1724 according to this embodiment will be described. FIG. 19A shows an example of a tomographic image 1901 to be input to the first processing unit 822. FIG. FIG. 19B shows an example of a label image 1902 obtained by the first processing unit 822 using the learned model with the tomographic image 1901 as input. The label image 1902 shows a label 1904 for the inner layer of the retina, a label 1903 for the superficial layer side (vitreous body side) of the retina, and a label 1905 for the deeper layer side (choroidal side) of the retina.

Note that the labeling is based on the label setting at the time of learning of the trained model. Therefore, the type of label is not limited to this, and a plurality of labels may be set in the retinal layer as shown in the third embodiment. Even in that case, it is possible to apply the correction processing according to the present embodiment.

In this embodiment, the first processing unit 822 performs image segmentation processing on a pixel-by-pixel basis using a trained model. Therefore, as indicated by labels 1903' and 1904' in FIG. 19(b), erroneous detection may occur partially. Corrector 1724 corrects these false positives based on the medical characteristics of the eye.

The first processing unit 822 performs labeling processing for each detected label, and integrates adjacent pixels having the same label as one area. There are three types of labels given in this embodiment: a label 1904 for the inner layer of the retina, a label 1903 for the superficial layer side (vitreous body side) of the retina, and a label 1905 for the deeper layer side (choroidal side) of the retina. . Also, since the object is an image obtained by taking a tomographic image of the retina, the order in which these labels appear is the order of label 1903, label 1904, and label 1905 from the top of the image. In the case of an EDI (Enhanced Depth Imaging) mode in which the choroid side is emphasized and photographed, the retina is reversed and photographed. in order.

As described above, since the image input to the first processing unit 822 is a tomographic image of the retina, it is possible to estimate the positional relationship of the labels based on the medical characteristics from the imaging conditions and the imaging site. Therefore, the correction unit 1724 specifies the detection result for each labeled region, and corrects the region considered to be erroneously detected to a region estimated based on the medical characteristics.

Specifically, for example, the correcting unit 1724 identifies the labeled regions in order from the larger area, and determines whether the area of the labeled region is equal to or smaller than the threshold value or the spatially different area from the already identified region. If the distance is large, it is judged to be an erroneous detection. After that, the correction unit 1724 resets the label information determined to be erroneous detection. An example of this case is shown in FIG. An area 1910 shown in FIG. 19(c) indicates an area where label information has been reset for the areas indicated by labels 1903' and 1904', which are areas regarded as erroneous detection by the correction unit 1724. FIG.

The correction unit 1724 assigns label information inferred from surrounding label information to the area 1910 whose label information has been reset. In the example shown in FIG. 19C, the label 1903 is assigned to the area 1910 surrounded by the label 1903, and the label 1905 is assigned to the area 1910 surrounded by the label 1905. In FIG.

As a result of these processes by the correction unit 1724, a final label image 1920 is output as shown in FIG. 19(d). This allows the image processing section 1720 to more appropriately detect the retinal area.

After the correction processing by the correction unit 1724 is performed, the processing moves to step S942. In step S942, the second processing unit 823 performs second boundary detection processing based on the corrected retinal region, as in the second embodiment. Since subsequent processing is the same as the processing of the second embodiment, the description is omitted.

As described above, the image processing apparatus 172 according to this embodiment further includes the correction unit 1724 that corrects the structure of the retinal layers detected by the first processing unit 822 based on the medical characteristics of the retinal layers.

Therefore, the image processing apparatus 172 according to the present embodiment can correct the region detected using the learned model using the medical features. Therefore, erroneous detection can be reduced even when an image is detected on a pixel-by-pixel basis.

In this embodiment, the correction process by the correction unit 1724 is added to the process according to the second embodiment, but the correction process may be added to the process according to the third or fourth embodiment.

(Example 6)
In Examples 1 to 5, the learned model was used to detect the boundary of the inner layer of the retina and the retinal area for the captured tomographic images. On the other hand, in the present embodiment, another trained model is used to generate a high-quality image in which the image quality of a tomographic image is improved, and the trained model according to the first or second embodiment is applied to the high-quality image. Perform boundary detection and area detection using It should be noted that the improvement of image quality in this embodiment includes noise reduction, conversion to colors and gradations that are easy to observe, improvement in resolution and spatial resolution, and enlargement of image size while suppressing deterioration in resolution. including.

Image processing by the image processing system according to the present embodiment will be described below, focusing on differences from the image processing according to the second embodiment. Since the configuration and processing procedure of the image processing system according to the present embodiment are the same as those of the image processing system 8 according to the second embodiment, the same reference numerals are used to denote them, and the description thereof is omitted. .

In this embodiment, the first processing unit 822 improves the image quality of the input image using a trained model for an image quality improvement model, which is a machine learning model different from the machine learning model for detecting the retinal region. process. The high image quality model is a machine learning model based on an arbitrary machine learning algorithm, trained in advance using appropriate teacher data, and is a trained model that outputs an image in which the image quality of the input image is improved.

Here, FIG. 20 shows an example of training data of the image quality enhancement model according to the present embodiment. In FIG. 20, a tomographic image 2001 shows an example of a tomographic image acquired by OCT imaging, and a tomographic image 2002 shows a tomographic image obtained by processing the tomographic image 2001 to improve image quality. A tomographic image 2001 shows an example of input data, a tomographic image 2002 shows an example of output data, and a pair group constituted by these images constitutes teacher data.

Note that as the image quality improvement processing, tomographic images obtained by photographing the same position spatially a plurality of times are aligned, and the tomographic images that have been aligned are averaged. Note that the image quality improvement process is not limited to the averaging process, and may be, for example, a process using a smoothing filter, a maximum a posteriori probability estimation process (MAP estimation process), a gradation conversion process, or the like. Further, the image that has undergone image quality enhancement processing may be, for example, an image that has undergone filter processing such as noise removal and edge enhancement, or an image that has undergone contrast adjustment such that a low-brightness image is changed to a high-brightness image. may be used. Furthermore, since the output data of the training data related to the high-quality image model may be a high-quality image, it is captured using an OCT apparatus with higher performance than the OCT apparatus used when capturing the tomographic image as the input data. It may be a tomographic image captured with a high load or a tomographic image captured with a high-load setting.

The first processing unit 822 inputs a tomographic image obtained by OCT imaging to the high image quality model trained using such teacher data, and acquires a high quality tomographic image. . Note that the first processing unit 822 can acquire a high-quality volume tomographic image by inputting a volume tomographic image obtained by three-dimensionally scanning the retina by raster scanning into the high-quality model. .

The first processing unit 822 receives a high-quality image acquired using the high-quality image model as an input, and detects a retinal region or a feature region using the learned model in the same manner as in the second to fifth embodiments.

Also, the second processing unit 823 can detect the retinal layer based on the high-quality image acquired by the first processing unit 822 and the detected retinal area and characteristic area.

As described above, in the image processing apparatus 80 according to the present embodiment, the first processing unit 822 performs the first detection process on the tomographic image whose image quality has been enhanced using the learned model.

As a result, the image processing apparatus 80 according to the present embodiment can improve the image quality of the input image using the learned model of the machine learning model, and detect the retinal layers in the image whose image quality has been improved. Therefore, it is possible to detect the retinal layer using an image whose image quality has been improved by noise reduction or the like, and to reduce erroneous detection.

In the present embodiment, a process for improving the image quality of a tomographic image, which is an input image, is added to the process according to the second embodiment. processing may be added.

Further, in the present embodiment, a machine learning model different from a machine learning model for performing detection processing is used as an image quality enhancement model for performing image quality enhancement. However, the machine learning model that performs the detection process may be made to learn the image quality enhancement process, and the machine learning model may be configured to perform both the image quality enhancement process and the detection process.

In this embodiment, the first processing unit 822 generates a high-quality image by improving the image quality of the tomographic image using a learned model (image-quality enhancement model) regarding image-quality enhancement processing. However, the component that generates a high quality image using the high quality model is not limited to the first processing unit 822 . For example, a third processing unit (image quality enhancement unit) may be provided separately from the first processing unit 822, and the third processing unit may generate a high quality image using the image quality enhancement model. Therefore, the first processing unit 822 or the third processing unit generates a tomographic image having higher image quality than the tomographic image, using a trained model for improving image quality. It can function as an example of a generator. In addition, the third processing unit and the high image quality model may be configured by a software module etc. executed by a processor such as CPU, MPU, GPU, FPGA, etc., or by a circuit etc. performing a specific function such as ASIC may be configured.

(Example 7)
Next, an image processing apparatus 80 according to the seventh embodiment will be described with reference to FIGS. In Example 6, the first processing unit 822 performed the first detection processing on the tomographic image whose image quality was enhanced using the image quality enhancement model, and detected the retinal region or the characteristic region. In this regard, the first processing unit 822 may perform image quality enhancement processing on other images using the image quality enhancement model, and the display control unit 25 displays various images with enhanced image quality on the display unit 50. may be displayed. For example, the first processing unit 822 generates a luminance En-Face image or an OCTA front image generated based on information (for example, a boundary image) of a retinal layer detected by the first detection process and the second detection process. etc. may be processed to improve image quality. In addition, the display control unit 25 can cause the display unit 50 to display at least one of the tomographic image, the brightness En-Face image, and the OCTA front image, which have been subjected to image quality enhancement processing by the first processing unit 822 . The image to be displayed with high image quality may be an SLO image, a fundus image obtained by a fundus camera or the like, a fluorescence fundus image, or the like. Here, the SLO image is a front image of the fundus obtained by an SLO (Scanning Laser Ophthalmoscope) optical system (not shown).

Here, the learning data of the image quality enhancement model for performing image quality enhancement processing on various images is the image before the image quality enhancement processing for each image, similar to the learning data of the image quality enhancement model according to the sixth embodiment. This is used as input data, and the image after image quality enhancement processing is used as output data. As for the image quality improvement processing for the learning data, as in the sixth embodiment, for example, averaging processing, processing using a smoothing filter, maximum posterior probability estimation processing (MAP estimation processing), gradation conversion processing, etc. can be Further, the image after image quality enhancement processing may be, for example, an image subjected to filter processing such as noise removal and edge enhancement, or an image whose contrast has been adjusted such that a low brightness image is changed to a high brightness image. may be used. Furthermore, since the output data of the training data related to the high image quality model may be a high image quality image, it was captured using an OCT device with higher performance than the OCT device used to capture the image that is the input data. An image or an image captured with a high-load setting may be used.

Also, the image quality enhancement model may be prepared for each type of image to be subjected to image quality enhancement processing. For example, a high image quality model for tomographic images, a high image quality model for luminance En-Face images, and a high image quality model for OCTA frontal images may be prepared. Furthermore, the high image quality model for the luminance En-Face image and the high image quality model for the OCTA front image are learned by comprehensively learning images of different depth ranges regarding the depth range (generation range) related to image generation. model. Images of different depth ranges may include, for example, images of superficial layer (Im2110), deep layer (Im2120), outer layer (Im2130), and choroidal vascular network (Im1940), as shown in FIG. . Further, as the image quality improvement model for the brightness En-Face image and the image quality improvement model for the OCTA front image, a plurality of image quality improvement models obtained by learning images for different depth ranges may be prepared.

Also, when preparing a high-quality model for tomographic images, it may be a trained model that has been exhaustively trained on tomographic images obtained at different positions in the sub-scanning (Y) direction. Tomographic images Im2151 to Im2153 shown in FIG. 21B are examples of tomographic images obtained at different positions in the sub-scanning direction. However, in the case of images taken at different locations (e.g., the center of the macula and the center of the optic papilla), it may be possible to learn separately for each region, or the You may choose to study together. Note that the tomographic images whose image quality is to be improved may include a luminance tomographic image and a tomographic image of motion contrast data. However, the tomographic image of brightness and the tomographic image of motion contrast data have significantly different image feature amounts, and therefore, they may be trained separately as high-quality models.

Image processing by the image processing system according to the present embodiment will be described below, focusing on differences from the image processing according to the sixth embodiment. Since the configuration and processing procedure of the image processing system according to the present embodiment are the same as those of the image processing system 8 according to the sixth embodiment, the same reference numerals are used and the description thereof is omitted. .

In this embodiment, an example will be described in which the display control unit 25 displays on the display unit 50 an image that has undergone image quality enhancement processing by the first processing unit 822 . In addition, in the present embodiment, description will be made using FIGS. 22(a) and 22(b), but the display screen is not limited to this. The image quality improvement process (image quality improvement process) can be similarly applied to a display screen in which a plurality of images obtained on different dates and times are displayed side by side, such as follow-up observation. In addition, like the imaging confirmation screen, the image quality enhancement process can be similarly applied to a display screen on which the examiner confirms the success or failure of imaging immediately after imaging. The display control unit 25 can cause the display unit 50 to display a plurality of high-quality images generated by the first processing unit 822 and low-quality images that have not been enhanced in image quality. In addition, the display control unit 25 selects a low image quality image and a high image quality image according to an instruction from the examiner for a plurality of high image quality images and low image quality images that have not been enhanced in image quality displayed on the display unit 50. can be displayed on the display unit 50, respectively. The image processing device 80 can also output the low-quality image and the high-quality image selected according to the examiner's instruction to the outside.

An example of the display screen 2200 of the interface according to the present embodiment is shown below with reference to FIGS. A display screen 2200 shows the entire screen, and the display screen 2200 shows a patient tab 2201 , an imaging tab 2202 , a report tab 2203 and a setting tab 2204 . Also, diagonal lines in the report tab 2203 represent the active state of the report screen. In this embodiment, an example of displaying a report screen will be described.

The report screen shown in FIG. 22A shows an SLO image Im2205, OCTA front images Im2207 and 2208, brightness En-Face image Im2209, tomographic images Im2211 and 2212, and a button 2220. FIG. An OCTA front image Im2206 corresponding to the OCTA front image Im2207 is superimposed on the SLO image Im2205. Furthermore, boundary lines 2213 and 2214 of the depth ranges of the OCTA front images Im2207 and Im2208 are superimposed on the tomographic images Im2211 and 2212, respectively. A button 2220 is a button for designating execution of image quality improvement processing. Button 2220 may be a button for instructing display of a high-quality image, as will be described later.

In this embodiment, execution of image quality improvement processing is performed by designating a button 2220, or whether or not to execute processing is determined based on information saved (stored) in a database. First, an example will be described in which the display of a high-quality image and the display of a low-quality image are switched by specifying the button 2220 according to an instruction from the examiner. In the following description, it is assumed that an OCTA front image is a target image for image quality enhancement processing.

Note that the depth ranges of the OCTA front images Im2207 and Im2208 may be determined using information on the retinal layers detected by the first detection process and the second detection process. The depth range may be, for example, a range between two layer boundaries regarding the detected retinal layers, or may be a predetermined range in a deeper direction or a shallower direction with reference to one of the two layer boundaries regarding the detected retinal layers. may be a range including the number of pixels of . Also, the depth range may be, for example, a range that is changed (offset) according to an operator's instruction from the range between two layer boundaries for the detected retinal layers.

When the examiner designates the report tab 2203 and transitions to the report screen, the display control unit 25 displays low-quality OCTA front images Im2207 and Im2208. After that, when the examiner designates a button 2220, the first processing unit 822 executes image quality enhancement processing on the OCTA front images Im2207 and Im2208 displayed on the screen. After the image quality improvement process is completed, the display control unit 25 displays the high quality image generated by the first processing unit 822 on the report screen. Note that the OCTA front image Im2206 is displayed by superimposing the OCTA front image Im2207 on the SLO image Im2205, so the display control unit 25 can also display an image that has undergone image quality enhancement processing for the OCTA front image Im2206. . In addition, the display control unit 25 can change the display of the button 2220 to an active state so that the user can see that the image quality improvement process has been executed.

Here, the execution of processing in the first processing unit 822 need not be limited to the timing when the examiner designates the button 2220 . Since the types of the OCTA front images Im2207 and Im2208 to be displayed when opening the report screen are known in advance, the first processing unit 822 executes image quality improvement processing when the displayed screen transitions to the report screen. You may Then, at the timing when the button 2220 is pressed, the display control unit 25 may display the high-quality image on the report screen. Furthermore, it is not necessary that two types of images are subjected to image quality enhancement processing in response to an instruction from the examiner or when transitioning to the report screen. For images that are likely to be displayed, for example, multiple OCTA frontal images such as the superficial layer (Im2110), the deep layer (Im2120), the outer layer (Im2130), and the choroidal vascular network (Im2140) as shown in FIG. You may make it process it. In this case, the image that has undergone image quality enhancement processing may be temporarily stored in a memory or stored in a database.

Next, a case will be described where image quality enhancement processing is executed based on information stored (recorded) in the database. If the state of executing the image quality improvement process is saved in the database, the display control unit displays the high quality image obtained by the first processing unit 822 executing the image quality improvement process when transitioning to the report screen. 25 is displayed on the display unit 50 by default. Then, the display control unit 25 displays the button 2220 in an active state by default so that the examiner can understand that a high-quality image obtained by executing the high-quality image processing is displayed. Can be configured. When the examiner wants to display a low image quality image before image quality enhancement processing, the display control unit 25 causes the display unit 50 to display the low image quality image by specifying the button 2220 to cancel the active state. can be done. At this time, if the examiner wishes to return the displayed image to a high-quality image, the examiner designates the button 2220 to activate it, and the display control unit 25 displays the high-quality image on the display unit 50 again. Let

Whether or not the image quality enhancement process is to be performed on the database is specified in common for all the data stored in the database, and for each hierarchy, such as for each imaging data (each examination). For example, when a state in which image quality enhancement processing is performed for the entire database is saved, a state in which the examiner does not perform image quality enhancement processing for individual imaging data (individual examination) is saved. be able to. In this case, the individual shot data for which the state in which the image quality enhancement process is not performed can be displayed next time without the image quality enhancement process being performed. According to such a configuration, if execution or non-execution of image quality enhancement processing is not specified for each imaging data unit (inspection unit), processing can be executed based on information specified for the entire database. can. In addition, when it is specified in units of imaging data (units of inspection), it is possible to execute processing individually based on that information.

Note that a user interface (for example, a save button) (not shown) may be used to save the execution state of image quality improvement processing for each image data (each examination). Also, when transitioning to other imaging data (other examination) or other patient data (for example, changing to a display screen other than the report screen in response to an instruction from the examiner), the display state (for example, button 2220 state), the state of executing the image quality improvement process may be saved.

In this embodiment, OCTA front images Im2207 and Im2208 are displayed as OCTA front images, but the OCTA front images to be displayed can be changed by designation by the examiner. Therefore, the change of the image to be displayed when the execution of the image quality enhancement process is specified (the button 2220 is in the active state) will be described.

The image to be displayed can be changed using a user interface (for example, a combo box) (not shown). For example, when the examiner changes the type of image from the superficial layer to the choroidal vascular network, the first processing unit 822 performs high-quality processing on the choroidal vascular network image, and the display control unit 25 performs the first A high-quality image generated by the processing unit 822 is displayed on the report screen. That is, the display control unit 25 changes the display of the high-quality image of the first depth range to the high-quality image of the second depth range that is at least partially different from the first depth range, in accordance with the instruction from the examiner. You can change the display to an image. At this time, the display control unit 25 changes the display of the high-quality image of the first depth range to the second depth range by changing the first depth range to the second depth range in accordance with an instruction from the examiner. The display may be changed to a high-quality image with a depth range of . As described above, when a high-quality image has already been generated for an image that is likely to be displayed when the report screen transitions, the display control unit 25 may display the generated high-quality image. .

In addition, the method for changing the type of image is not limited to the one described above, and an OCTA front image is generated in which a different depth range is set by changing the reference layer and offset value, and the generated OCTA front image is subjected to high image quality processing. It is also possible to display a high-quality image obtained by executing In that case, when the reference layer or the offset value is changed, the first processing unit 822 performs high-quality processing on an arbitrary OCTA front image, and the display control unit 25 reports a high-quality image. display on the screen. It should be noted that the reference layer and the offset value can be changed using a user interface (for example, a combo box and a text box) (not shown). Further, by dragging (moving the layer boundary) either of the boundary lines 2213 and 2214 superimposed on the tomographic images Im2211 and Im2212, the depth range (generation range) of the OCTA front image can be changed. .

When the boundary line is changed by dragging, the instruction to execute the image quality enhancement process is continuously executed. Therefore, the first processing unit 822 may always process the execution command, or may execute it after the layer boundary is changed by dragging. Alternatively, the execution of the image quality improvement processing is instructed continuously, but when the next instruction arrives, the previous instruction may be canceled and the latest instruction may be executed.

It should be noted that the image quality enhancement process may take a relatively long time. For this reason, it may take a relatively long time to display a high-quality image regardless of the timing at which the command is executed. Therefore, after the depth range for generating the OCTA front image is set according to the instruction from the examiner, until the high-quality image is displayed, the low-quality OCTA corresponding to the set depth range A front image (low quality image) may be displayed. That is, when the depth range is set, a low-quality OCTA front image (low-quality image) corresponding to the set depth range is displayed. may be changed to display a high-quality image. Further, information indicating that image quality enhancement processing is being performed may be displayed after the depth range is set until the high quality image is displayed. It should be noted that these processes are not limited to the configuration applied when it is assumed that execution of the image quality improvement process has already been specified (the button 2220 is in an active state). For example, it is possible to apply these processes until a high-quality image is displayed when execution of the image quality improvement process is instructed in response to an instruction from the examiner.

In this embodiment, OCTA front images Im2207 and Im2108 relating to different layers are displayed as OCTA front images, and an example is shown in which low image quality and high image quality images are switched and displayed, but the displayed image is not limited to this. . For example, a low-quality OCTA front image as the OCTA front image Im2207 and a high-quality OCTA front image as the OCTA front image Im2208 may be displayed side by side. When the images are switched and displayed, the images are switched at the same place, so that parts with changes are easily compared, and when the images are displayed side by side, the images can be displayed at the same time, so it is easy to compare the entire images.

Next, the execution of image quality improvement processing in screen transition will be described with reference to FIGS. FIG. 22(b) is a screen example showing an enlarged display of the OCTA front image Im2207 in FIG. 22(a). Also in FIG. 22(b), a button 2220 is displayed in the same manner as in FIG. 22(a). Screen transition from FIG. 22(a) to FIG. 22(b) is made by, for example, double-clicking the OCTA front image Im2207, and transition from FIG. 22(b) to FIG. do. Note that screen transition is not limited to the method shown here, and a user interface (not shown) may be used.

If the execution of image quality enhancement processing is specified at the time of screen transition (the button 2220 is active), that state is maintained even at the time of screen transition. That is, when a high-quality image is displayed on the screen of FIG. 22(a) and the screen is changed to the screen of FIG. 22(b), the high-quality image is also displayed on the screen of FIG. 22(b). Button 2220 is then activated. The same applies when transitioning from FIG. 22(b) to FIG. 22(b). In FIG. 22(b), a button 2220 can be designated to switch the display to a low image quality image.

Regarding screen transitions, the display state of high-quality images is maintained not only for the screens shown here, but also for transitions to screens that display the same imaging data, such as the display screen for follow-up observation or the display screen for panorama. You can still make transitions. That is, on the display screen after transition, an image corresponding to the state of button 2220 on the display screen before transition can be displayed. For example, if the button 2220 on the display screen before transition is active, a high-quality image is displayed on the display screen after transition. Further, for example, if the active state of the button 2220 on the display screen before transition is canceled, a low image quality image is displayed on the display screen after transition. When the button 2220 on the display screen for follow-up observation becomes active, the multiple images obtained on different dates (different examination dates) displayed side by side on the display screen for follow-up observation are switched to high-quality images. may In other words, when the button 2220 on the display screen for follow-up observation becomes active, it may be configured such that a plurality of images obtained on different dates and times are collectively reflected.

An example of the display screen for follow-up observation is shown in FIG. When the tab 2301 is selected according to an instruction from the examiner, a display screen for follow-up observation is displayed as shown in FIG. At this time, the depth range of the OCTA front image can be changed by selecting a set desired by the examiner from predetermined depth range sets displayed in list boxes 2302 and 2303 . For example, the superficial layer of the retina is selected in the list box 2302 and the deep layer of the retina is selected in the list box 2303 . The upper display area displays the analysis result of the OCTA frontal image of the superficial layer of the retina, and the lower display area displays the analysis result of the OCTA frontal image of the deep layer of the retina. When the depth range is selected, the analysis results of the multiple OCTA enface images of the selected depth range are collectively displayed in parallel for multiple images of different dates and times.

At this time, if the display of the analysis result is set to a non-selected state, the parallel display of a plurality of OCTA front images of different dates and times may be collectively changed. When the examiner designates the button 2220, the display of the multiple OCTA front images is collectively changed to the display of the multiple high-quality images.

Further, when the display of the analysis results is in a selected state, when the button 2220 is specified in response to an instruction from the examiner, the display of the analysis results of a plurality of OCTA front images is changed to the analysis results of a plurality of high-quality images. is changed collectively to the display of Here, the analysis result may be displayed by superimposing the analysis result on the image with arbitrary transparency. At this time, the display of the image may be changed to the display of the analysis result, for example, by superimposing the analysis result on the displayed image with an arbitrary degree of transparency. In addition, the change from the display of the image to the display of the analysis result, for example, even if it is a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency good.

Also, the type and offset position of the layer boundary used to specify the depth range can be collectively changed from the user interfaces 2305 and 2306, respectively. Note that the user interfaces 2305 and 2306 for changing the type of layer boundary and the offset position are examples, and any other form of interface may be used. By displaying the tomographic image together and moving the layer boundary data superimposed on the tomographic image in accordance with instructions from the examiner, the depth range of multiple OCTA front images of different dates and times can be changed collectively. You may At this time, a plurality of tomographic images of different dates and times may be displayed side by side, and when the movement is performed on one tomographic image, the layer boundary data may be similarly moved on the other tomographic images.

Also, the image projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a user interface such as a context menu.

Alternatively, a selection screen (not shown) may be displayed by selecting the selection button 2307, and an image selected from the image list displayed on the selection screen may be displayed. An arrow 2304 displayed at the top of FIG. 23 is a mark indicating the currently selected examination. image on the far left). Of course, a mark indicating the reference inspection may be displayed on the display unit.

Also, when the "Show Difference" check box 2308 is specified, the measurement value distribution (map or sector map) for the reference image is displayed on the reference image. Furthermore, in this case, a difference measurement value map between the measurement value distribution calculated for the reference image and the measurement value distribution calculated for the image displayed in the region corresponding to the inspection date other than that display. As the measurement result, a trend graph (a graph of measurement values for images on each inspection day obtained by measurement of changes over time) may be displayed on the report screen. That is, time-series data (for example, time-series graph) of a plurality of analysis results corresponding to a plurality of images of different dates and times may be displayed. At this time, the analysis results for dates and times other than the multiple dates and times corresponding to the multiple images being displayed are also kept in a state that can be distinguished from the multiple analysis results corresponding to the multiple images being displayed (for example, time series (The color of each point on the graph differs depending on whether or not the image is displayed.) It may be displayed as time-series data. Also, the regression line (curve) of the trend graph and the corresponding formula may be displayed on the report screen.

In this embodiment, the OCTA front image has been described, but the image to which the processing according to this embodiment is applied is not limited to this. An image related to processing such as display, image quality improvement, and image analysis according to the present embodiment may be a brightness En-Face image. In addition to En-Face images, different images such as B-scan tomographic images, SLO images, fundus images, and fluorescent fundus images may be used. In that case, the user interface for executing the image quality enhancement process may be one that instructs the execution of the image quality enhancement process for multiple images of different types, or one that selects an arbitrary image from multiple images of different types. There may be something that instructs execution of image quality improvement processing.

For example, when tomographic images obtained by B-scan are displayed with high image quality, the tomographic images Im2211 and Im2212 shown in FIG. 22A may be displayed with high image quality. Also, a tomographic image with high image quality may be displayed in the area where the OCTA front images Im2207 and Im2208 are displayed. Only one tomographic image may be displayed, or a plurality of tomographic images may be displayed. When a plurality of tomographic images are displayed, the tomographic images acquired at different positions in the sub-scanning direction may be displayed. When displayed, images in different scanning directions may be displayed. For example, when displaying a plurality of tomographic images obtained by radial scanning or the like with high image quality, a plurality of partially selected tomographic images (for example, two tomographic images at mutually symmetrical positions with respect to the reference line) ) may be displayed respectively. Furthermore, a plurality of tomographic images are displayed on a display screen for follow-up observation as shown in FIG. ) may be displayed. Further, image quality enhancement processing may be performed on the tomographic image based on the information stored in the database by a method similar to the method described above.

Similarly, when the SLO image is displayed with high image quality, for example, the SLO image Im2205 may be displayed with high image quality. Furthermore, when the luminance En-Face image is to be displayed with high image quality, for example, the luminance En-Face image 2209 may be displayed with high image quality. Furthermore, a plurality of SLO images and brightness En-Face images are displayed on the display screen for follow-up observation as shown in FIG. specific layer thickness, etc.) may be provided. Further, image quality improvement processing may be performed on the SLO image or luminance En-Face image based on the information stored in the database by a method similar to the method described above. Note that the display of the tomographic image, the SLO image, and the brightness En-Face image is an example, and these images may be displayed in any manner according to the desired configuration. Also, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the brightness En-Face image may be displayed with high image quality by a single instruction.

With such a configuration, the display control unit 25 can display on the display unit 50 an image that has undergone image quality enhancement processing by the first processing unit 822 according to this embodiment. At this time, as described above, when at least one of a plurality of conditions relating to the display of high-quality images, the display of analysis results, and the depth range of the front image to be displayed is selected, the display screen is transitioned, the selected state may be maintained.

Further, as described above, when at least one of a plurality of conditions is in a selected state, the state in which at least one is selected is maintained even if another condition is changed to a selected state. may For example, when the display of the analysis result is selected, the display control unit 25 changes the display of the analysis result of the low-quality image to high in response to an instruction from the examiner (for example, when the button 2220 is specified). The display may be changed to the analysis result of the image quality image. In addition, when the display of the analysis result is selected, the display control unit 25 displays the analysis result of the high-quality image according to the instruction from the examiner (for example, when the designation of the button 2220 is canceled). may be changed to display the analysis results of low-quality images.

In addition, when the display of the high-quality image is not selected, the display control unit 25 displays the low-quality image according to the instruction from the examiner (for example, when the display of the analysis result is canceled). The display of the analysis result may be changed to the display of the low image quality image. In addition, when the display of high-quality images is in a non-selected state, the display control unit 25 displays low-quality images in response to instructions from the examiner (for example, when display of analysis results is specified). You may change to the display of the analysis result of a low-quality image. In addition, when the display of high-quality images is selected, the display control unit 25 analyzes the high-quality images according to instructions from the examiner (for example, when the display of the analysis results is canceled). The display of the results may be changed to display a high quality image. In addition, when the display of high-quality images is selected, the display control unit 25 increases the display of high-quality images according to instructions from the examiner (for example, when display of analysis results is specified). The display may be changed to the analysis result of the image quality image.

Also, consider a case where the display of the high-quality image is in a non-selected state and the display of the first type of analysis result is in a selected state. In this case, the display control unit 25 displays the first type of analysis result of the low image quality image according to the instruction from the examiner (for example, when display of the second type of analysis result is specified). The display may be changed to display the second type of analysis result of the low quality image. Also, consider a case where the display of the high-quality image is selected and the display of the first type of analysis result is selected. In this case, the display control unit 25 displays the first type of analysis result of the high-quality image in response to an instruction from the examiner (for example, when display of the second type of analysis result is specified). The display may be changed to display the second type of analysis results of the high quality image.

As described above, the display screen for follow-up observation may be configured so that these display changes are collectively reflected in a plurality of images obtained at different dates and times. Here, the analysis result may be displayed by superimposing the analysis result on the image with arbitrary transparency. At this time, the display of the analysis result may be changed to a state in which the analysis result is superimposed on the displayed image with arbitrary transparency, for example. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

In this embodiment, the first processing unit 822 generates a high-quality image by improving the image quality of the tomographic image using a learned model (image-quality enhancement model) regarding image-quality enhancement processing. However, the component that generates a high quality image using the high quality model is not limited to the first processing unit 822 . For example, a third processing unit (image quality enhancement unit) may be provided separately from the first processing unit 822, and the third processing unit may generate a high quality image using the image quality enhancement model. In this case, the third processing unit and the high image quality model may be composed of a software module or the like executed by a processor such as a CPU, MPU, GPU, or FPGA, or a circuit or the like that performs a specific function such as an ASIC. may be configured by

(Modifications of Examples 6 and 7)
In Examples 6 and 7, the display control unit 25 causes the display unit 50 to display an image selected according to an instruction from the examiner, from among the high-quality image generated by the first processing unit 822 and the input image. can be made In addition, the display control unit 25 may switch the display on the display unit 50 from a captured image (input image) to a high-quality image in accordance with an instruction from the examiner. That is, the display control unit 25 may change the display of the low image quality image to the display of the high image quality image according to the instruction from the examiner. Further, the display control unit 25 may change the display of the high-quality image to the display of the low-quality image in accordance with an instruction from the examiner.

Furthermore, the first processing unit 822 executes the start of image quality improvement processing by the image quality improvement engine (image quality improvement model) (input of image to the image quality improvement engine) in accordance with an instruction from the examiner, The display control unit 25 may cause the display unit 50 to display the generated high-quality image. On the other hand, when the input image is captured by the imaging device (OCT device 10), the first processing unit 822 automatically generates a high quality image based on the input image using the image quality enhancement engine, and displays it. The control unit 25 may cause the display unit 50 to display a high-quality image in accordance with an instruction from the examiner. Here, the image quality improvement engine includes a trained model that performs the image quality improvement process (image quality improvement process) described above.

Note that these processes can be similarly performed for the output of analysis results. That is, the display control unit 25 may change the display of the analysis result of the low image quality image to the display of the analysis result of the high image quality image in accordance with the instruction from the examiner. Further, the display control unit 25 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image in accordance with an instruction from the examiner. Furthermore, the display control unit 25 may change the display of the analysis result of the low image quality image to the display of the low image quality image according to the instruction from the examiner. Further, the display control unit 25 may change the display of the low image quality image to the display of the analysis result of the low image quality image according to the instruction from the examiner. Furthermore, the display control unit 25 may change the display of the analysis result of the high-quality image to the display of the high-quality image in accordance with an instruction from the examiner. In addition, the display control unit 25 may change the display of the high-quality image to the display of the analysis result of the high-quality image in accordance with an instruction from the examiner.

Furthermore, the display control unit 25 may change the display of the analysis result of the low image quality image to display of another type of analysis result of the low image quality image according to the instruction from the examiner. In addition, the display control unit 25 may change the display of the analysis result of the high-quality image to display of another type of analysis result of the high-quality image in accordance with an instruction from the examiner.

Here, the analysis result of the high-quality image may be displayed by superimposing the analysis result of the high-quality image on the high-quality image with an arbitrary degree of transparency. Further, the analysis result of the low image quality image may be displayed by superimposing the analysis result of the low image quality image on the low image quality image with arbitrary transparency. At this time, the display of the analysis result may be changed to a state in which the analysis result is superimposed on the displayed image with arbitrary transparency, for example. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

Note that in this modification, the first processing unit 822 generates a high-quality image by improving the image quality of the tomographic image using a learned model (image-quality enhancement model) regarding image-quality enhancement processing. However, the component that generates a high quality image using the high quality model is not limited to the first processing unit 822 . For example, a third processing unit that is separate from the first processing unit 822 may be provided, and the third processing unit may generate a high quality image using a high quality image model. In this case, the third processing unit and the high image quality model may be composed of a software module or the like executed by a processor such as a CPU, MPU, GPU, or FPGA, or a circuit or the like that performs a specific function such as an ASIC. may be configured by

Further, in Example 7, an image that has undergone image quality enhancement processing using the image quality enhancement model is displayed according to the active state of the button 2220 on the display screen. On the other hand, according to the active state of the button 2220, the analysis value using the result of the image segmentation processing using the trained model may be displayed. In this case, for example, when the button 2220 is in an inactive state (an image segmentation process using a learned model is not selected), the display control unit 25 performs the image segmentation process performed by the second processing unit 823. The analysis result using the result is displayed on the display unit 50 . On the other hand, when the button 2220 is activated, the display control unit 25 controls the image segmentation processing performed by the first processing unit 822 alone or by the first processing unit 822 and the second processing unit 823. The analysis result using the result is displayed on the display unit 50 .

With such a configuration, the analysis result using the result of image segmentation processing without using the trained model and the analysis result using the result of image segmentation processing using the trained model are switched according to the active state of the button. displayed. Since these analysis results are based on the results of processing by a trained model and image processing by a rule base, respectively, there may be differences between the two results. Therefore, by switching and displaying these analysis results, the examiner can compare the two and use a more convincing analysis result for diagnosis.

When the image segmentation process is switched, for example, when the image to be displayed is a tomographic image, the numerical value of the layer thickness analyzed for each layer may be switched and displayed. Further, for example, when tomographic images are displayed in which layers are divided by color, hatching pattern, etc., the tomographic images in which the shape of the layer is changed according to the result of the image segmentation process may be switched and displayed. . Furthermore, when the thickness map is displayed as the analysis result, the thickness map may be displayed with the color indicating the thickness changed according to the result of the image segmentation processing. Also, the button for specifying the image quality enhancement process and the button for specifying the image segmentation process using the trained model may be provided separately, either one may be provided, or both buttons may be provided. You may provide as one button.

Switching of image segmentation processing may be performed based on information stored (recorded) in a database, similarly to switching of image quality enhancement processing described above. As for the processing at the time of screen transition, the switching of the image segmentation processing may be performed in the same manner as the switching of the image quality improvement processing described above.

In Examples 1 to 7, the acquisition unit 21 acquired the interference signal acquired by the OCT apparatus 10 and the three-dimensional tomographic data generated by the tomographic image generation unit 221 . However, the configuration in which the acquisition unit 21 acquires these signals and data is not limited to this. For example, the acquisition unit 21 may acquire these signals from a server or an imaging device connected to the image processing device 20 via a LAN, WAN, Internet, or the like. In this case, it is possible to omit the processing related to imaging and acquire the three-dimensional tomographic data that has been captured. Then, boundary detection processing can be performed in steps S304, S904, and the like. Therefore, it is possible to shorten a series of processing time from acquisition of tomographic information to display of a front image, a thickness map, or the like.

Note that the trained model for image segmentation and the trained model for improving image quality used by the processing unit 222 and the first processing unit 822 can be provided in the image processing apparatuses 20 , 80 , and 152 . A trained model may be configured by, for example, a software module or the like executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured by a circuit or the like that performs a specific function such as an ASIC. Also, these learned models may be provided in another server device or the like connected to the image processing devices 20 , 80 , 152 . In this case, the image processing apparatuses 20, 80, and 152 can use the trained model by connecting to a server or the like having the trained model via an arbitrary network such as the Internet. Here, the server provided with the learned model may be, for example, a cloud server, a fog server, an edge server, or the like.

In addition, in the second to seventh embodiments, label images labeled for each pixel have been described as label images, but label images labeled for each region may be used as label images.

The configuration of the OCT apparatus 10 is not limited to the configuration described above, and part of the configuration included in the OCT apparatus 10 may be configured separately from the OCT apparatus 10 . Also, the user interface such as buttons and the display layout are not limited to those shown above.

According to the above-described first to seventh embodiments and their modifications, it is possible to detect the boundaries of the retinal layers regardless of disease, site, or the like.

(Example 8)
2. Description of the Related Art In the medical field, image diagnosis using images acquired by various imaging devices is performed in order to identify a disease of a subject and to observe the degree of the disease. For example, in the field of radiology, types of imaging apparatuses include X-ray imaging apparatus, X-ray computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET). In the field of ophthalmology, for example, there are fundus cameras, scanning laser ophthalmoscopes (SLO), optical coherence tomography (OCT) devices, and OCT angiography (OCTA) devices.

Imaging diagnosis is basically carried out by medical professionals observing the lesions, etc. depicted on the images, but in recent years improvements in image analysis technology have made it possible to obtain a variety of useful information for diagnosis. . For example, image analysis can assist healthcare professionals in detecting small lesions that may be overlooked, provide quantitative measurements of lesion shape and volume, or even be unobserved by healthcare professionals. It has become possible to identify diseases in

There are various image analysis methods, but the process for identifying regions such as organs and lesions depicted in images, which is called image segmentation processing, is one of the most important factors in implementing many image analysis methods. This is the necessary processing. In the following, the image segmentation process is also simply referred to as the segmentation process for the sake of simplification.

Conventional image segmentation processing is performed by image processing algorithms based on medical knowledge and image characteristics of target organs and lesions, as in Patent Document 1. However, in the actual medical field, the images obtained from the imaging device may not be captured clearly due to various factors such as the condition of the subject, the imaging environment of the imaging device, and the lack of skill of the photographer. . For this reason, in conventional image segmentation processing, target organs and lesions are not rendered as expected, and specific regions cannot be extracted with high accuracy.

Specifically, for example, in an image obtained by photographing a diseased eye with retinal layer loss, bleeding, vitiligo, and neovascularization with an OCT apparatus, the depiction of the shape of the retinal layer is irregular. something happened. In such cases, erroneous detection may occur in retinal layer area detection processing, which is a type of image segmentation processing.

One of the objects of the eighth through nineteenth embodiments below is to provide a medical image processing apparatus, a medical image processing method, and a program capable of performing image segmentation processing with higher precision than conventional image segmentation processing.

<Description of terms>
Here, terms used in the present disclosure will be explained.

In the network herein, each device may be connected by wired or wireless links. Here, the lines connecting each device in the network are, for example, dedicated lines, local area network (hereinafter referred to as LAN) lines, wireless LAN lines, Internet lines, Wi-Fi (registered trademark), and Bluetooth (registered trademark). ) etc.

A medical image processing apparatus may be composed of two or more devices capable of communicating with each other, or may be composed of a single device. Also, each component of the medical image processing apparatus may be configured by a software module executed by a processor such as a CPU, MPU, GPU, or FPGA. Also, each component may be configured by a circuit or the like that performs a specific function, such as an ASIC. Also, it may be configured by a combination of any other hardware and any software.

A medical image processed by a medical image processing apparatus or a medical image processing method is a tomographic image of a subject acquired using an OCT apparatus. Here, the OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Fourier-domain OCT devices may also include spectral-domain OCT (SD-OCT) devices and wavelength-swept OCT (SS-OCT) devices. In addition, as the OCT apparatus, a wavefront compensation OCT (AO-OCT) apparatus using a wavefront compensation optical system, a line OCT apparatus in which the measurement light irradiated to the subject is formed in a line, Any OCT device, such as a formed full-field OCT device, may be included.

Medical images include tomographic images of the subject's eye (subject's eye). The tomographic image of the subject's eye is not limited to a tomographic image of the retina or the like in the posterior segment of the subject's eye, but includes a tomographic image of the anterior segment of the subject's eye, the chamber of the eye, or the like. Further, when the OCT apparatus is used for an endoscope or the like, a tomographic image of the subject's skin or organs may be used as a medical image to be processed by the medical image processing apparatus or the medical image processing method according to the following embodiments. .

An image management system is a device and system that receives and stores images captured by an imaging device such as an OCT device or images that have undergone image processing. In addition, the image management system can transmit images in response to requests from connected devices, perform image processing on stored images, and request other devices to perform image processing. can. The image management system can include, for example, a picture archival communication system (PACS). In particular, the image management system according to the embodiment described below includes a database capable of storing various types of information such as subject information and imaging time associated with received images. Also, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with saved images in response to requests from other devices. .

The shooting conditions are various information at the time of shooting the image acquired by the shooting device. The imaging conditions include, for example, information about the imaging device, information about the facility where the imaging was performed, information about the examination related to imaging, information about the photographer, and information about the subject. The imaging conditions include, for example, the date and time of imaging, the name of the imaging part, the imaging area, the imaging angle of view, the imaging method, the resolution and gradation of the image, the image size, the applied image filter, and information on the data format of the image. include. Note that the imaging region can include a peripheral region shifted from a specific imaging region, a region including a plurality of imaging regions, and the like. Also, the imaging method may include any OCT imaging method such as a spectral domain method or a wavelength sweep method.

The imaging conditions can be stored in the data structure that constitutes the image, stored as imaging condition data separate from the image, or stored in a database or image management system associated with the imaging device. Therefore, the imaging conditions can be acquired by a procedure corresponding to the storage means of the imaging conditions of the imaging device. Specifically, the imaging conditions are used, for example, to analyze the data structure of an image output by the imaging device, to acquire imaging condition data corresponding to the image, or to acquire the imaging conditions from a database related to the imaging device. It is obtained by accessing the interface of

Depending on the imaging device, there are imaging conditions that cannot be obtained because they are not saved. For example, if the imaging device does not have the ability to acquire or store specific imaging conditions, or if such a function is disabled. Further, for example, there are cases in which the photographing conditions are not stored because they are unrelated to the photographing device or photographing. Furthermore, for example, there are cases where the shooting conditions are hidden, encrypted, or cannot be obtained without the right. However, even shooting conditions that are not saved may be acquired. For example, by performing image analysis, it is possible to specify the imaging site name and imaging region.

A region label image is a label image in which a region label is assigned to each pixel. Specifically, as shown in FIG. 24, arbitrary regions are divided by a group of identifiable pixel values (hereinafter referred to as region label values) from among the region groups depicted in the image Im2410 acquired by the imaging device. image Im2420. Here, the specified arbitrary region includes a region of interest (ROI), a volume of interest (VOI), and the like.

By specifying a coordinate group of pixels having an arbitrary region label value from the image Im2420, it is possible to specify a coordinate group of pixels that render a corresponding region such as a retinal layer in the image Im2410. Specifically, for example, when the region label value indicating the ganglion cell layer that constitutes the retina is 1, a coordinate group having a pixel value of 1 among the pixel groups of the image Im2420 is specified, and the coordinates are extracted from the image Im2410. Extract the pixel group corresponding to the group. As a result, the region of the ganglion cell layer in image Im2410 can be identified.

Note that some embodiments include a process of performing a reduction or enlargement process on the area label image. At this time, the image interpolation processing method used to reduce or enlarge the region label image uses the nearest neighbor method or the like so as not to erroneously generate an undefined region label value or a region label value that should not exist at the corresponding coordinates. shall be

Image segmentation processing is processing for identifying regions called ROIs and VOIs, such as organs and lesions depicted in images, for use in image diagnosis and image analysis. For example, according to the image segmentation process, it is possible to specify a region group of layers forming the retina from an image obtained by OCT imaging of the posterior segment of the eye. Note that the number of specified regions is 0 if the region to be specified is not rendered in the image. In addition, if a plurality of region groups to be specified are drawn in the image, the number of specified regions may be plural, or even one region surrounding the region group may be included. good.

The identified region group is output as information that can be used in other processing. Specifically, for example, it is possible to output a group of coordinates of a group of pixels forming each of the identified region groups as a group of numerical data. Further, for example, a group of coordinates indicating a rectangular area, an elliptical area, a rectangular parallelepiped area, an ellipsoidal area, etc. containing each of the identified area groups can be output as a numerical data group. Further, for example, it is possible to output a group of coordinates indicating a straight line, a curve, a plane, a curved surface, or the like, which is the boundary of the identified region group, as a numerical data group. Also, for example, an area label image indicating the specified area group can be output.

In the following description, when the image segmentation processing is expressed as having high accuracy or as an area labeled image with high accuracy, it means that the area is correctly identified at a high rate. Conversely, when the accuracy of image segmentation processing is expressed as low, it means that the rate of erroneously specifying regions is high.

An image without a region label is a type of region labeled image, and is a region labeled image that does not contain information corresponding to ROIs and VOIs for use in image diagnosis and image analysis. Specifically, as an example, a case where it is desired to know the region of the ganglion cell layer forming the retina depicted in the medical image for use in image analysis will be described.

Here, it is assumed that the region label value indicating the region of the ganglion cell layer is 1, and the region label value indicating the region of the other portion is 0. When an area label image corresponding to a certain medical image is generated by image segmentation processing or the like, all pixel values of the area label image are 0 if the ganglion cell layer is not rendered in the medical image. In this case, the region-labeled image is an unlabeled image because the region-labeled image does not have a region with a region label value of 1 corresponding to the ROI of the ganglion cell layer to be used for image analysis. It should be noted that, depending on the setting and implementation mode, the image without area label may not be an image, but may be a numerical data group or the like indicating a coordinate group having information similar to that of the image.

Here, the machine learning model refers to a learning model based on a machine learning algorithm. Specific algorithms of machine learning include nearest neighbor method, naive Bayes method, decision tree, support vector machine, and the like. Another example is deep learning in which a neural network is used to generate feature values and connection weighting coefficients for learning. As appropriate, any of the above algorithms can be used and applied to the learning model according to the embodiment. A trained model is a model that has been trained (learned) in advance using appropriate teacher data (learning data) for a machine learning model according to an arbitrary machine learning algorithm. However, it is assumed that the trained model is not one that does not perform further learning, and that additional learning can be performed. The teacher data consists of one or more pairs of input data and output data. The format and combination of the input data and the output data of the paired group that constitutes the training data may be one of which is an image and the other is a numerical value, or one is composed of a plurality of image groups and the other is a character string, or both. may be suitable for a desired configuration, such as an image.

Specifically, for example, teacher data (hereinafter referred to as first teacher data) configured by a pair group of an image obtained by OCT and an imaging region label corresponding to the image can be used. Note that the imaging region label is a unique numerical value or character string representing the region. Further, as another example of training data, training data composed of a pair group of an image acquired by OCT imaging targeting the posterior segment of the eye and an area label image of the retinal layer corresponding to the image. (hereinafter referred to as second teacher data). Furthermore, another example of training data is a pair group of a low-quality image with a lot of noise obtained by normal OCT imaging and a high-quality image that has been imaged multiple times by OCT and processed to improve the image quality. Teacher data (hereinafter referred to as third teacher data) can be mentioned.

When input data is input to the trained model, output data according to the design of the trained model is output. A trained model outputs output data that is highly likely to correspond to input data according to a tendency trained using teacher data, for example. In addition, the learned model can, for example, output the possibility of corresponding to the input data as a numerical value for each type of output data according to the tendency trained using the teacher data.

Specifically, for example, when an image acquired by OCT is input to a trained model trained with the first teacher data, the trained model outputs the imaging site label of the imaging site imaged in the image. or output the probability for each imaging site label. Further, for example, when an image depicting the retinal layer obtained by OCT imaging targeting the posterior segment of the eye is input to the trained model trained by the second teacher data, the trained model is rendered in the image. output a region-labeled image for the retinal layer. Furthermore, for example, when a low-quality image with a lot of noise obtained by normal OCT imaging is input to the trained model trained with the third teacher data, the trained model is imaged multiple times by OCT to improve the image quality. A high-quality image equivalent to the processed image is output.

Machine learning algorithms include deep learning techniques such as convolutional neural networks (CNN). In methods related to deep learning, different parameter settings for layers and nodes that make up a neural network may result in different degrees of reproducibility in output data of tendencies trained using teacher data.

For example, in a trained model of deep learning using the first teacher data, if more appropriate parameters are set, the probability of outputting a correct imaging region label may increase. Also, for example, in a trained model using the second teacher data, if more appropriate parameters are set, it may be possible to output a region label image with higher accuracy. Furthermore, for example, in the trained model of deep learning using the third training data, if more appropriate parameters are set, it may be possible to output an image with higher image quality.

Specifically, the parameters in the CNN are, for example, the kernel size of the filter, the number of filters, the value of the stride, the value of the dilation, and the number of nodes output by the fully connected layer, which are set for the convolution layer etc. Note that the parameter group and the number of training epochs can be set to values that are preferable for the mode of use of the learned model, based on teacher data. For example, based on training data, set parameters and the number of epochs that can output correct imaging site labels with a higher probability, output region label images with higher accuracy, and output images with higher image quality. can do.

One method for determining such a parameter group and the number of epochs will be exemplified. First, 70% of the pairs constituting the teacher data are randomly set for training and the remaining 30% for evaluation. Next, the trained model is trained using the training pairs, and at the end of each epoch of training, the evaluation pairs are used to calculate a training evaluation value. The training evaluation value is, for example, the average value of a group of values obtained by evaluating the output when the input data constituting each pair is input to the learned model during training and the output data corresponding to the input data using a loss function. is. Finally, the parameter group and the number of epochs when the training evaluation value becomes the smallest are determined as the parameter group and the number of epochs of the learned model. It should be noted that, by dividing the pair groups that make up the teacher data into training pair groups and evaluation pair groups and determining the number of epochs in this way, the learned model may overfit the training pair groups. can be prevented.

An image segmentation engine is a module that performs image segmentation processing and outputs a region label image corresponding to an input image that has been input. Examples of the input image include an OCT B-scan image and a three-dimensional tomographic image (three-dimensional OCT volume image). Examples of the region label image include a region label image indicating each layer of the retinal layer when the input image is an OCT B-scan image, and each layer of the retinal layer when the input image is an OCT three-dimensional tomographic image. There is a region label image showing a 3D region showing .

An image processing method that constitutes an image segmentation processing method in the following embodiments performs processing using a trained model according to various machine learning algorithms such as deep learning. Note that the image processing method may be performed not only with a machine learning algorithm but also with any other existing processing. The image processing includes, for example, various image filter processing, matching processing using a database of region label images corresponding to similar images, image registration processing of reference region label images, and knowledge-based image processing. .

In particular, there is a configuration 2500 shown in FIG. 25 as an example of a convolutional neural network (CNN) that performs image segmentation processing on a two-dimensional image Im2510 input as an input image to generate an area label image Im2520. The configuration 2500 of the CNN includes multiple layers responsible for processing and outputting input values. As shown in FIG. 25, the types of layers included in the configuration 2500 include a convolution layer, a downsampling layer, an upsampling layer, and a merger layer. Note that the CNN configuration 2500 used in this embodiment is a U-net type machine learning model, like the CNN configuration 601 described in the first embodiment.

The convolution layer is a layer that performs convolution processing on an input value group according to set parameters such as the kernel size of filters, the number of filters, the stride value, and the dilation value. The down-sampling layer is a layer that performs processing to make the number of output value groups smaller than the number of input value groups by thinning out or synthesizing input value groups. A specific example of processing performed in the downsampling layer is Max Pooling processing.

The upsampling layer is a layer that performs processing to make the number of output value groups larger than the number of input value groups by duplicating the input value group or adding values interpolated from the input value group. A specific example of processing performed in the upsampling layer is linear interpolation processing. The synthesizing layer is a layer that performs a process of synthesizing a group of values such as a group of output values of a certain layer and a group of pixel values forming an image from a plurality of sources and connecting or adding them.

In addition, as parameters set in the convolutional layer group included in the configuration of the CNN, for example, the kernel size of the filter is 3 pixels in width, 3 pixels in height, and the number of filters is 64, so that an image with a certain accuracy Segmentation processing is possible. However, if the parameter settings for the layers and nodes that make up the neural network are different, the degree to which the tendencies trained from the teacher data can be reproduced in the output data may differ. In other words, in many cases, appropriate parameters for each layer group and each node group differ depending on the embodiment, and may be changed as necessary.

Also, depending on the embodiment, the CNN may obtain better characteristics not only by changing the parameters as described above, but also by changing the configuration of the CNN. Better characteristics include, for example, high accuracy of image segmentation processing, short time for image segmentation processing, short time for training a learned model, and the like. Examples of modifications to the configuration of the CNN include, for example, incorporating a batch normalization layer after the convolutional layer and an activation layer using a rectifier linear unit.

As a machine learning model used by the image segmentation engine, for example, FCN, SegNet, or the like can be used as in the first embodiment. Also, depending on the desired configuration, a machine learning model that performs object recognition on a region-by-region basis as described in the first embodiment may be used.

When it is necessary to process a one-dimensional image, a three-dimensional image, or a four-dimensional image, the kernel size of the filter may be one-dimensional, three-dimensional, or four-dimensional. Here, a four-dimensional image includes, for example, a three-dimensional moving image or an image in which parameters at each pixel position of a three-dimensional image are indicated by different hues.

Also, the image segmentation process may be performed using only one image processing technique, or may be performed using a combination of two or more image processing techniques. Additionally, multiple image segmentation processing techniques may be implemented to generate multiple region label images.

Also, in some embodiments, the input image is divided into groups of small regions, image segmentation processing is performed on each of them to obtain a group of region label images of the small regions, and the group of region label images of the small regions is combined. Thus, there is a way to generate region label images. If the input image is a three-dimensional image, the small area may be a three-dimensional image smaller than the input image, a two-dimensional image, or a one-dimensional image. Also, if the input image is a two-dimensional image, the small area may be a two-dimensional image smaller than the input image, or may be a one-dimensional image. Also, depending on the embodiment, a plurality of region label image groups may be output.

Parameters may also be input to the image segmentation engine along with the input image. The parameters to be input in this case may include, for example, parameters that specify the extent of the range in which image segmentation processing is performed, such as the upper limit of the lesion size, and parameters that specify the image filter size used in the image processing method. can. It should be noted that the image segmentation engine may output other images or coordinate data groups capable of identifying regions, instead of region label images, depending on the embodiment.

Note that when performing a plurality of image segmentation processing methods or performing image segmentation processing on a plurality of small region groups, processing time can be reduced by performing image segmentation processing in parallel.

Note that when using some image processing methods such as image processing using CNN, it is necessary to pay attention to the image size. Specifically, it should be noted that different image sizes may be required for the input image and the output region label image in order to deal with problems such as the peripheral portion of the region label image not being segmented with sufficient accuracy. is.

For the sake of clarity, it will not be specified in the embodiments described later, but if an image segmentation engine that requires different image sizes for an image input to the image segmentation engine and an output image is used, the image size may be changed as appropriate. Assume it is adjusted. Specifically, input images, such as images used as teacher data for training a trained model and images input to an image segmentation engine, are padded, and the surrounding shooting areas of the input image are combined. or to adjust the image size. The area to be padded is filled with a constant pixel value, filled with neighboring pixel values, or mirror-padded according to the characteristics of the image segmentation processing method so that the image segmentation processing can be performed effectively.

The imaging location estimation engine is a module for estimating the imaging part and imaging area of the input image. The imaging location estimation engine determines where the imaging location or imaging region drawn in the input image is, or determines the probability of being the imaging location or imaging region for each imaging location label or imaging region label with a required level of detail. can be output.

Depending on the imaging device, the imaging site and imaging region may not be stored as imaging conditions, or may not be stored because the imaging device cannot acquire them. Further, even if the imaging parts and imaging regions are stored, there are cases where the imaging parts and imaging regions with the required level of detail are not stored. For example, only "posterior segment of the eye" is stored as an imaging region, but in detail, is it "macula", "optical papilla", or "macula and optic papilla"? Sometimes I don't know if it's "other". In another example, only "breast" is stored as the imaged region, but in detail, it is not known whether it is "right breast", "left breast", or "both". be. Therefore, by using the imaging location estimation engine, it is possible to estimate the imaging part and imaging area of the input image in these cases.

The image and data processing method that constitutes the estimation method of the shooting location estimation engine performs processing using a trained model according to various machine learning algorithms such as deep learning. In the image and data processing method, in addition to or instead of processing using machine learning algorithms, any known arbitrary processing such as natural language processing, matching processing using a database of similar images and similar data, knowledge base processing, etc. An estimation process may be performed. Note that the training data for training the learned model constructed using the machine learning algorithm can be images labeled with the imaging site and imaging region. In this case, with respect to teacher data, an image for which an imaging part or an imaging region is to be estimated is input data, and a label of an imaging part or imaging region is output data.

In particular, there is a configuration 2600 shown in FIG. 26 as a configuration example of the CNN for estimating the shooting location of the two-dimensional input image Im2610. The structure 2600 of the CNN includes a group of convolution processing blocks 2620 composed of a convolution layer 2621, a batch normalization layer 2622, and an activation layer 2623 using a normalization linear function. The CNN structure 2600 also includes a final convolutional layer 2630 , a Full Connection layer 2640 and an output layer 2650 . Fully connected layer 2640 fully connects the output values of convolution processing block 2620 . In addition, the output layer 2650 uses the Softmax function to output the probability for each assumed imaging site label for the input image Im2610 as an estimation result 2660 (Result). In such a configuration 2600, for example, if the input image Im2610 is an image obtained by imaging the "macula", the highest probability is output for the radiographed region label corresponding to the "macula".

Note that, for example, the number of convolution processing blocks 2620 is 16, the parameters of the group of convolution layers 2621 and 2630 are set to a filter kernel size of 3 pixels wide, 3 pixels high, and 64 filters. can be used to estimate the imaging site. However, in practice, as described in the above explanation of the trained model, it is possible to set a better parameter group using teacher data according to the usage pattern of the trained model. If it is necessary to process a one-dimensional image, a three-dimensional image, or a four-dimensional image, the kernel size of the filter may be expanded to one-dimensional, three-dimensional, or four-dimensional. Note that the estimation technique may be implemented using only one image and data processing technique, or may be implemented using a combination of two or more image and data processing techniques.

The region label image evaluation engine is a module that evaluates whether an input region label image is plausible and image segmentation processing can be performed. Specifically, the area label image evaluation engine outputs a true value as an image evaluation index if the input area label image is plausible, and a false value otherwise. Examples of methods for performing the evaluation include processing using trained models according to various machine learning algorithms such as deep learning, knowledge base processing, and the like. One method of knowledge-based processing, for example, is based on anatomical knowledge, where known anatomical knowledge, such as the regularity of the shape of the retina, is used to evaluate region-labeled images. I do.

Specifically, as an example, a case of evaluating a region label image corresponding to an OCT tomographic image of the posterior segment as an imaging target by knowledge base processing will be described. In the posterior segment of the eye, anatomically, the tissue groups have fixed locations. Therefore, there is a method of checking the pixel value group in the area label image, that is, the coordinates of the area label value group, and evaluating whether or not the position is output correctly. In this evaluation method, for example, if there is a region label value corresponding to the crystalline lens at coordinates close to the anterior segment and a region label value corresponding to the retinal layer group at coordinates far from the anterior segment in a certain range, it is likely that the image segmentation process can be performed. Evaluate that there is. On the other hand, if these region label values are not at such expected locations, then we assess that the image segmentation process has not been performed properly.

The method for evaluating the knowledge base will be described more specifically using the region label image Im2710 of the layer group constituting the retinal layers, which corresponds to the OCT tomographic image of the posterior segment of the eye shown in FIG. do. In the posterior segment of the eye, tissue groups have anatomically fixed positions. Therefore, by checking the pixel value group in the region label image, that is, the coordinates of the region label value group, it is possible to determine whether the region label image is a plausible image. You can decide whether or not

The area label image Im2710 includes an area Seg2711, an area Seg2712, an area Seg2713, and an area Seg2714 in which groups of pixels having the same area label value are continuously formed. Although the region Seg2711 and the region Seg2714 have the same region label value, the layer group that anatomically constitutes the retinal layers has a layered structure. It is evaluated as having undergone image segmentation processing. In this case, the region label image rating engine will output a false value for the image rating index.

Further, among the knowledge-based evaluation processing methods, there is also a method of evaluating whether or not the area label image includes a pixel having an area label value corresponding to an area that must exist in the object to be photographed. Furthermore, for example, there is also a method of evaluating whether or not the area label image includes a certain number or more of pixels having area label values corresponding to areas that must always exist in the imaging target.

Also, if the image segmentation engine performs multiple image segmentation processing techniques to generate multiple region label images, the region label image evaluation engine selects one of the most likely region label images from among the multiple label images. You can also output

Note that if each of a plurality of area label image groups is a plausible area label image, it may not be possible to select a single area label image to be output. In such a case where only one region label image cannot be selected, the region label evaluation engine can select one region label image with a predetermined priority, for example. The region label evaluation engine may also, for example, weight and merge multiple region label images into one region label image. Further, for example, the region label evaluation engine may display a plurality of region label image groups on a user interface provided in an arbitrary display unit or the like, and select one according to an instruction from the examiner (user). good. Note that the region label image evaluation engine may output all of the plausible region images.

A region label image correction engine is a module that corrects incorrectly image segmented regions in an input region label image. Examples of techniques for performing the correction include knowledge base processing and the like. One method of knowledge-based processing, for example, uses anatomical knowledge.

The region label image Im2710 of the layer group constituting the retinal layer corresponding to the OCT tomographic image of the posterior segment of the eye shown in FIG. The method will be described more specifically. As described above, in the region labeled image Im2710, the layer group that anatomically configures the retinal layers has a layered structure. It can be seen that it is an area that has undergone image segmentation processing. The region label image correction engine detects incorrectly segmented regions and overwrites the detected regions with different region label values. For example, in FIG. 27, region Seg 2714 is overwritten with a region label value indicating that it is not one of the retinal layers.

Note that the region label image correction engine may detect or identify incorrectly segmented regions using the evaluation results of the region label evaluation engine. The region label image modification engine may also overwrite label values of detected incorrectly segmented regions with label information inferred from label information around the region. In the example of FIG. 27, when label information is attached to the area surrounding the area Seg2714, the label information of the area Seg2714 can be overwritten with the label information of the area. Note that the peripheral label information is not limited to the label information of the area that completely surrounds the area to be corrected, and may be the most numerous label information among the label information of the areas adjacent to the area to be corrected. .

(Configuration of image processing apparatus according to embodiment 8)
A medical image processing apparatus according to an eighth embodiment will be described below with reference to FIGS. In the following, for the sake of simplification, the medical image processing apparatus will simply be referred to as an image processing apparatus. FIG. 28 shows an example of a schematic configuration of an image processing apparatus according to this embodiment.

The image processing device 2800 is connected to the imaging device 2810 and the display unit 2820 via circuits and networks. Alternatively, the imaging device 2810 and the display unit 2820 may be directly connected. Although these devices are separate devices in this embodiment, some or all of these devices may be integrated. Also, these devices may be connected to any other device via a circuit or network, or may be integrally configured with any other device.

The image processing apparatus 2800 includes an acquisition unit 2801, an imaging condition acquisition unit 2802, a processing propriety determination unit 2803, a segmentation processing unit 2804, an evaluation unit 2805, an analysis unit 2806, and an output unit 2807 (display control unit). and are provided. Note that the image processing device 2800 may be composed of a plurality of devices provided with some of these components.

The acquisition unit 2801 can acquire various data and images from the imaging device 2810 and other devices, and can acquire input from the examiner via an input device (not shown). As the input device, a mouse, keyboard, touch panel, and other arbitrary input devices may be adopted. Moreover, the display unit 2820 may be configured as a touch panel display.

An imaging condition acquisition unit 2802 acquires imaging conditions for the medical image (input image) acquired by the acquisition unit 2801 . Specifically, the imaging condition group stored in the data structure forming the medical image is obtained according to the data format of the medical image. Note that if imaging conditions are not saved in the medical image, the imaging information group can be acquired from the imaging device 2810 or the image management system via the acquisition unit 2801 .

A process availability determination unit (determination unit) 2803 determines whether or not the segmentation processing unit 2804 can process the medical image using the imaging condition group acquired by the imaging condition acquisition unit 2802 . A segmentation processing unit 2804 uses an image segmentation engine (segmentation engine) including a trained model to perform image segmentation processing on medical images that can be handled, and generates region label images (region information).

The evaluation unit 2805 evaluates the region label image generated by the segmentation processing unit 2804 using a region label image evaluation engine (evaluation engine), and determines whether to output the region label image based on the evaluation result. . The region label image evaluation engine outputs a true value as an image evaluation index if the input region label image is plausible, and a false value otherwise. The evaluation unit 2805 determines to output the region label image when the image evaluation index obtained by evaluating the region label image is a true value.

The analysis unit 2806 performs image analysis processing of the input image using the input image and the region label image determined to be output by the evaluation unit 2805 . The analysis unit 2806 can calculate, for example, the shape change and layer thickness of tissue included in the retinal layer by image analysis processing. Any known image analysis process may be used as the image analysis process. The output unit 2807 causes the display unit 2820 to display the area label image and the analysis result by the analysis unit 2806 . Also, the output unit 2807 may store the region label image and the analysis result in a storage device connected to the image processing apparatus 2800, an external device, or the like.

Next, the segmentation processing unit 2804 will be described in detail. The segmentation processor 2804 uses an image segmentation engine to generate region label images corresponding to the input image (input). The image segmentation processing method provided in the image segmentation engine according to this embodiment performs processing using a trained model.

In this embodiment, training of a machine learning model consists of pairs of input data, which are images acquired under specific shooting conditions assumed to be processed, and output data, which are region labeled images corresponding to the input data. Use the training data that has been prepared. The specific imaging conditions specifically include a predetermined imaging region, imaging method, imaging angle of view, image size, and the like.

In this embodiment, the input data of the teacher data is an image acquired with the same model as the imaging device 2810 and the same settings as the imaging device 2810 . Note that the input data of the teacher data may be an image acquired from an imaging device having the same image quality tendency as the imaging device 2810 .

Also, the output data of the teacher data is the region label image corresponding to the input data. For example, referring to FIG. 24, input data is a tomographic image Im2410 of a retinal layer captured by OCT. The output data is a region label image Im2420 obtained by adding region label values representing the types of retinal layers to the corresponding coordinates according to the types of retinal layers depicted in the tomographic image Im2410, and dividing each region. . A region label image is created by a medical specialist referring to a tomographic image, by arbitrary image segmentation processing, or by a medical specialist correcting the region label image created by the image segmentation processing. I can prepare.

The input data group of the teacher data pair group exhaustively includes input images having various conditions assumed to be processed. The various conditions are, specifically, image conditions caused by different combinations of variations such as the condition of the subject, the imaging environment of the imaging device, and the technical level of the photographer. By making the conditions of the images included in the input data group exhaustive, it is possible to perform highly accurate image segmentation processing even for images with poor conditions for which the accuracy of conventional image segmentation processing is low. A machine learning model is trained. Therefore, the segmentation processing unit 2804 uses an image segmentation engine that includes a trained model that has undergone such training to stably generate highly accurate region label images for images under various conditions. be able to.

It should be noted that pairs that do not contribute to the image segmentation processing can be removed from the training data among the pair group that constitutes the training data. For example, if the region label value of the region label image, which is the output data constituting the teacher data pair, is incorrect, the region label of the region label image obtained using the trained model trained using the teacher data Values are also more likely to be incorrect. In other words, the accuracy of the image segmentation process is lowered. Therefore, it is possible to improve the accuracy of the trained model included in the image segmentation engine by removing pairs that have region label images with erroneous region label values in the output data from the training data.

By using an image segmentation engine including such a trained model, the segmentation processing unit 2804 can identify organs and lesions depicted in the medical image when a medical image obtained by imaging is input. A region label image can be output.

Next, a series of image processing according to this embodiment will be described with reference to the flowchart of FIG. FIG. 29 is a flow chart of a series of image processing according to this embodiment. First, when a series of image processing according to this embodiment is started, the process moves to step S2910.

In step S2910, the acquisition unit 2801 acquires an image captured by the imaging device 2810 as an input image from the imaging device 2810 connected via a circuit or network. Note that the acquisition unit 2801 may acquire an input image in response to a request from the imaging device 2810 . For example, when the imaging device 2810 generates an image, such a request is made before or after saving the image generated by the imaging device 2810 in the recording device of the imaging device 2810, and displaying the saved image on the display unit 2820. It may be issued when displaying, when using the region label image for image analysis processing, and the like.

Note that the acquisition unit 2801 may acquire data for generating an image from the imaging device 2810 and acquire an image generated based on the data by the image processing device 2800 as the input image. In this case, any existing image generation method may be employed as an image generation method for the image processing apparatus 2800 to generate various images.

In step S2920, the imaging condition acquisition unit 2802 acquires a group of imaging conditions for the input image. Specifically, according to the data format of the input image, the imaging condition group stored in the data structure forming the input image is obtained. As described above, if the input image does not store the imaging conditions, the imaging condition acquisition unit 2802 can acquire the imaging information group from the imaging device 2810 or an image management system (not shown).

In step S2930, the processability determination unit 2803 determines whether the image segmentation engine used by the segmentation processing unit 2804 can perform image segmentation processing on the input image using the acquired imaging condition group. Specifically, the processability determination unit 2803 determines whether or not the imaging region, imaging method, imaging angle of view, and image size of the input image match conditions that can be handled using the trained model of the image segmentation engine. judge.

If the process propriety determination unit 2803 determines all the imaging conditions and determines that the condition can be handled, the process proceeds to step S2940. On the other hand, if the processing enable/disable determining unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970.

Depending on the settings and implementation of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on some of the imaging region, imaging method, imaging angle of view, and image size, Step S2940 may be performed. For example, it is assumed that the image segmentation engine can comprehensively handle any imaging part of the subject, and even if the input data contains an unknown imaging part, it can be handled. Such processing may be performed if it is implemented. In addition, the processing enable/disable determining unit 2803 determines whether or not at least one of the imaging region, imaging method, imaging angle of view, and image size of the input image matches a condition that can be dealt with by the image segmentation engine according to the desired configuration. It may be determined whether

In step S2940, the segmentation processing unit 2804 uses the image segmentation engine to perform image segmentation processing on the input image and generate region label images from the input image. Specifically, the segmentation processing unit 2804 inputs the input image to the image segmentation engine. The image segmentation engine generates a region label image as region information that can identify organs and lesions depicted in the input image based on a trained model that has undergone machine learning using teacher data.

Note that depending on the settings and implementation of the image processing apparatus 2800, the segmentation processing unit 2804 inputs parameters together with the input image to the image segmentation engine according to the imaging condition group, and adjusts the degree of the range of image segmentation processing. You may In addition, the segmentation processing unit 2804 may input parameters according to the input of the examiner to the image segmentation engine together with the input image to adjust the degree of the range of image segmentation processing.

In step S2950, the evaluation unit 2805 uses the region label image evaluation engine to evaluate whether the region label image generated by the segmentation processing unit 2804 is a plausible image. In this embodiment, the evaluator 2805 uses a region label evaluation engine that employs a knowledge-based evaluation method to evaluate whether the region labeled image is a plausible image.

Specifically, the region label evaluation engine checks the pixel values in the region label image, ie, the coordinates of the region label value group, and evaluates whether the positions are output to anatomically correct positions. In this case, for example, if there is a region label value corresponding to the crystalline lens at coordinates close to the anterior segment and a region label value corresponding to the retinal layer group at coordinates far from the anterior segment in a certain range, it is plausible that the image segmentation process can be performed. evaluate. On the other hand, if these region label values are not at such expected locations, then we assess that the image segmentation process has not been performed properly. The region label evaluation engine outputs a true value as an image evaluation index when the region label is evaluated as plausibly processed for image segmentation, and outputs a false value when evaluated as plausibly processed for image segmentation. to output

The evaluation unit 2805 determines whether to output the region label image based on the image evaluation index output from the region label evaluation engine. Specifically, the evaluation unit 2805 determines that an area label image is to be output when the image evaluation index is a true value. On the other hand, if the image evaluation index is a false value, it is determined not to output the region label image generated by the segmentation processing unit 2804 . Note that the evaluation unit 2805 can generate an image without a region label when determining that the region label image generated by the segmentation processing unit 2804 is not to be output.

In step S2960, when the evaluation unit 2805 determines that the region label image is to be output, the analysis unit 2806 performs image analysis processing of the input image using the region label image and the input image. The analysis unit 2806 calculates, for example, changes in layer thickness and tissue shape depicted in the input image by image analysis processing. Any known processing may be adopted as the image analysis processing method. If the evaluation unit 2805 determines not to output an area label image or if an image without an area label is generated, the process proceeds without image analysis.

In step S2970, if the evaluation unit 2805 determines to output the region label image, the output unit 2807 outputs the region label image and the image analysis result and causes the display unit 2820 to display them. Note that the output unit 2807 may cause the imaging device 2810 or another device to display or store the region label image and the image analysis result instead of displaying the region label image and the image analysis result on the display unit 2820 . Depending on the settings and implementation of the image processing apparatus 2800, the output unit 2807 processes these so that they can be used by the imaging apparatus 2810 and other apparatuses, or converts them into data formats that can be transmitted to an image management system or the like. may be converted. Also, the output unit 2807 is not limited to outputting both the region label image and the image analysis result, and may output only one of them.

On the other hand, if it is determined in step S2930 that the image segmentation processing is impossible, the output unit 2807 outputs an image without region label, which is a type of region label image, and causes the display unit 2820 to display it. It should be noted that instead of outputting the region-unlabeled image, a signal indicating that the image segmentation process was not possible may be sent to the imaging device 2810 .

Moreover, even when it is determined in step S2950 that the image segmentation processing could not be performed appropriately, the output unit 2807 outputs the region-unlabeled image and causes the display unit 2820 to display it. In this case as well, the output unit 2807 may send a signal indicating that the image segmentation processing could not be performed properly to the imaging device 2810 instead of outputting the region-unlabeled image. When the output processing in step S2970 ends, the series of image processing ends.

As described above, the image processing apparatus 2800 according to this embodiment includes the acquisition unit 2801 and the segmentation processing unit 2804 . Acquisition unit 2801 acquires an input image that is a tomographic image of a predetermined region of the subject. A segmentation processing unit 2804 uses an image segmentation engine including a trained model to generate a region label image, which is region information capable of identifying anatomical regions, from the input image. The image segmentation engine includes a trained model using tomographic images under various conditions and region labeled images as learning data. The image segmentation engine takes the tomographic image as input and the region label image as output.

Also, the image processing apparatus 2800 further includes an evaluation unit 2805 . The evaluation unit 2805 evaluates the region label image using a knowledge-based evaluation engine using anatomical features, and determines whether to output the region label image according to the evaluation result.

With this configuration, the image processing apparatus 2800 according to the present embodiment uses a segmentation engine including a trained model to generate region label images as region information used to identify ROIs and VOIs that can be used for image diagnosis and image analysis. can be generated. Therefore, it is possible to output a region label image with high accuracy even for an input image with poor conditions for conventional segmentation processing, and to provide ROIs and VOIs that can be used for image diagnosis and image analysis.

Furthermore, the image processing apparatus 2800 can prevent inappropriate area label images from being used for image diagnosis and image analysis by evaluating whether or not the area label image is a plausible image by the evaluation unit 2805 .

The image processing apparatus 2800 further includes an imaging condition acquisition unit 2802 and a processing propriety determination unit 2803 . The imaging condition acquisition unit 2802 acquires imaging conditions including at least one of the imaging region of the input image, the imaging method, the imaging angle of view, and the image size. A process propriety determination unit 2803 determines whether or not an area label image can be generated from an input image using the image segmentation engine. The processing enable/disable determination unit 2803 performs the determination based on the shooting conditions of the input image.

With this configuration, the image processing apparatus 2800 according to the present embodiment can omit an input image that cannot be processed by the segmentation processing unit 2804 from the image segmentation process, thereby reducing the processing load of the image processing apparatus 2800 and the occurrence of errors. can.

Furthermore, the image processing apparatus 2800 further includes an analysis unit 2806 that performs image analysis on the input image using the region label image that is region information. With this configuration, the image processing apparatus 2800 can perform image analysis using highly accurate region label images generated by the segmentation processing unit 2804, and obtain highly accurate analysis results.

In this embodiment, the processability determination unit 2803 determines whether or not the input image can be subjected to image segmentation processing by the image segmentation engine. Thereafter, when the processability determination unit 2803 determines that the input image can be processed by the segmentation processing unit 2804, the segmentation processing unit 2804 performs image segmentation processing. On the other hand, if the imaging device 2810 performs imaging only under imaging conditions that allow image segmentation processing, the image acquired from the imaging device 2810 may be unconditionally subjected to image segmentation processing. In this case, as shown in FIG. 30, steps S2920 and S2930 can be omitted, and step S2940 can be performed after step S2910.

In this embodiment, the output unit 2807 (display control unit) is configured to display the generated region label image and the analysis result on the display unit 2820, but the operation of the output unit 2807 is not limited to this. . For example, the output unit 2807 can also output the region label image and the analysis result to other devices connected to the imaging device 2810 and the image processing device 2800 . As such, the region label images and analysis results may be displayed in the user interfaces of these devices, stored on any recording device, used for any image analysis, or sent to an image management system. can.

Also, in this embodiment, the output unit 2807 is configured to display the area label image and the image analysis result on the display unit 2820 . However, the output unit 2807 may display the region label image and the image analysis result on the display unit 2820 in accordance with instructions from the examiner. For example, the output unit 2807 may cause the display unit 2820 to display the region label image or the image analysis result in response to the examiner pressing any button on the user interface of the display unit 2820 . In this case, the output unit 2807 may display the region label image by switching from the input image. The output unit 2807 may display an area label image UI 3120 side by side with the input image UI 3110 as shown in FIG. An image may be superimposed and displayed. Any known translucency method may be used. For example, the area label image can be made translucent by setting the transparency of the area label image to a desired value.

In addition, the output unit 2807 outputs information indicating that the region label image has been generated using the trained model and that it is the result of image analysis performed based on the region label image generated using the trained model. may be displayed on the display unit 2820. Furthermore, the output unit 2807 may cause the display unit 2820 to display a display showing what kind of teacher data the trained model has learned. The display may include a description of the types of input data and output data of the teacher data, and any display related to the teacher data such as imaging regions included in the input data and output data.

In this embodiment, when the processability determination unit 2803 determines that the input image can be processed by the image segmentation engine, the process proceeds to step S2940, and the image segmentation process by the segmentation processing unit 2804 is started. On the other hand, the output unit 2807 may cause the display unit 2820 to display the judgment result by the processing feasibility judgment unit 2803, and the segmentation processing unit 2804 may start image segmentation processing according to an instruction from the examiner. At this time, the output unit 2807 can also cause the display unit 2820 to display the input image and the imaging conditions such as the imaging region acquired for the input image together with the determination result. In this case, image segmentation processing is performed after the examiner determines whether or not the determination result is correct, so image segmentation processing based on erroneous determination can be reduced.

In relation to this, the display unit 2820 may display the input image and imaging conditions such as the imaging region acquired for the input image by the output unit 2807 without performing the determination by the processing propriety determination unit 2803 . In this case, the segmentation processing unit 2804 can start image segmentation processing according to an instruction from the examiner.

Furthermore, in this example, the evaluation unit 2805 evaluated the region label images generated by the segmentation processing unit 2804 using a region label image evaluation engine that employs a knowledge-based evaluation method. On the other hand, the evaluation unit 2805 uses an area label image evaluation engine that includes a trained model that has been trained using an area label image and an image evaluation index according to a predetermined evaluation method as teacher data to determine whether the area label image is a plausible image. It may be evaluated whether

In this case, the training data of the machine learning model included in the region label image evaluation engine uses region label images and false images that look like region label images as input data, and image evaluation indices for each image as output data. The image evaluation index is a true value when the input data is a proper area label image, and a false value when it is a false image. In addition, as a method of generating a false image, there is a method of using an arbitrary generator of an area label image with inappropriate conditions, a method of intentionally overwriting an appropriate area label image with an inappropriate area label, etc. may be adopted.

Even if the evaluation unit 2805 evaluates whether the region labeled image is a plausible image by using the region labeled image evaluation engine including the trained model that has undergone such learning, it is not suitable for image diagnosis or image analysis. It is possible to prevent the use of an inappropriate area label image.

Note that when the processability determination unit 2803 determines that the image segmentation processing is not possible, the image without region label may be generated by the output unit 2807 or generated by the processability determination unit 2803. good. If the evaluation unit 2805 determines that the image segmentation has not been properly performed, the output unit 2807 may cause the display unit 2820 to display that the image segmentation has not been properly performed in step S2970. .

(Example 9)
Next, an image processing apparatus according to a ninth embodiment will be described with reference to FIGS. 28 and 33. FIG. In Example 8, the segmentation processor 2804 had one image segmentation engine. On the other hand, in the present embodiment, the segmentation processing unit uses a plurality of image segmentation engines, each of which includes a trained model that has been machine-learned using different teacher data, to generate a plurality of regions for an input image. Generate a label image.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the eighth embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the eighth embodiment, so the same reference numerals are used for the configuration shown in FIG. 28, and the description thereof is omitted.

The segmentation processing unit 2804 according to this embodiment performs image segmentation processing on an input image using two or more image segmentation engines each including a trained model machine-learned using different teacher data. conduct.

Here, a method of creating a teacher data group according to this embodiment will be described. Specifically, first, a group of pairs of an image as input data and a region label image as output data in which various imaging parts are imaged is prepared. Next, a teacher data group is created by grouping the pair groups for each imaging part. For example, first training data composed of a pair group obtained by imaging a first imaging part, and second training data composed of a pair group obtained by imaging a second imaging part Create a training data group as follows.

Each training data is then used to machine-learn a machine-learning model contained in a separate image segmentation engine. For example, providing a first image segmentation engine that includes a trained model trained on first training data. In addition, a group of image segmentation engines are provided, such as a second image segmentation engine including a trained model trained with a second teacher data.

Such image segmentation engines differ in the training data used to train the learned models they contain. Therefore, such an image segmentation engine differs in the extent to which an input image can be subjected to image segmentation processing depending on the shooting conditions of the image input to the image segmentation engine. Specifically, the first image segmentation engine has high image segmentation processing accuracy for the input image acquired by imaging the first imaging region, and the image segmentation engine acquired by imaging the second imaging region. The accuracy of the image segmentation process is low for images with Similarly, the second image segmentation engine has high image segmentation processing accuracy for the input image acquired by imaging the second imaging region, and the image acquired by imaging the first imaging region. The accuracy of the image segmentation process is low for

Since each of the teacher data is composed of a pair group grouped according to an imaging part, the image quality tendencies of the image groups forming the pair group are similar. For this reason, the image segmentation engine can perform image segmentation processing with higher accuracy than the image segmentation engine according to the eighth embodiment, as long as it is a corresponding imaging part. The imaging condition for grouping pairs of training data is not limited to the imaging site, but may be the imaging angle of view, the resolution of the image, or a combination of two or more of these. good.

A series of image processing according to this embodiment will be described with reference to FIG. FIG. 33 is a flowchart of a series of image processing according to this embodiment. It should be noted that the processing of steps S3310 and S3320 is the same as that of steps S2910 and S2920 according to the eighth embodiment, so the description thereof will be omitted. Note that if the input image is to be subjected to image segmentation processing unconditionally, after the processing of step S3320, the processing of step S3330 may be omitted and the processing may proceed to step S3340.

When the imaging conditions for the input image are acquired in step S3320, the process proceeds to step S3330. In step S3330, the process propriety determination unit 2803 uses the imaging condition group acquired in step S3320 to determine whether or not any of the above-described image segmentation engine groups can process the input image.

If the process propriety determination unit 2803 determines that none of the image segmentation engines can process the input image, the process proceeds to step S3380. On the other hand, if the process possibility determination unit 2803 determines that any one of the image segmentation engine group can process the input image, the process proceeds to step S3340. It should be noted that even if the image segmentation engine determines that part of the shooting conditions cannot be dealt with, as in the eighth embodiment, step S3340 may be executed depending on the settings and implementation of the image processing apparatus 2800. good.

In step S3340, the segmentation processing unit 2804 selects an image segmentation engine to be processed based on the shooting conditions of the input image acquired in step S3320 and the training data information of the image segmentation engine group. Specifically, for example, for an imaging region in the imaging condition group acquired in step S3320, image segmentation having teacher data information about the imaging region or surrounding imaging regions and having high accuracy in image segmentation processing. Choose your engine. In the above example, the segmentation processor 2804 selects the first image segmentation engine if the imaging region is the first imaging region.

In step S3350, the segmentation processing unit 2804 uses the image segmentation engine selected in step S3340 to generate region label images by subjecting the input image to image segmentation processing. Steps S3360 and S3370 are the same as steps S2950 and S2960 in the eighth embodiment, so description thereof will be omitted.

After image analysis is performed in step S3370, the process proceeds to step S3380. In step S3380, when the evaluation unit 2805 determines to output the region label image, the output unit 2807 outputs the region label image and the analysis result and causes the display unit 2820 to display them. When displaying the region label image on the display unit 2820, the output unit 2807 may display that the region label image is generated using the image segmentation engine selected by the segmentation processing unit 2804. FIG. Note that the output unit 2807 may output only one of the region label image and the analysis result.

On the other hand, if it is determined in step S3330 that the image segmentation processing is impossible, the output unit 2807 outputs an area labelless image, which is a type of area label image, and causes the display unit 2820 to display it. It should be noted that instead of generating an unlabeled region image, a signal may be sent to the imager 2810 indicating that the image segmentation process was not possible.

Moreover, even when it is determined in step S3360 that the image segmentation processing could not be performed appropriately, the output unit 2807 outputs an area labelless image, which is a kind of area label image, and causes the display unit 2820 to display it. In this case also, the output unit 2807 may send a signal to the imaging device 2810 indicating that the image segmentation process could not be performed properly, instead of generating the region-unlabeled image. When the output processing in step S3380 ends, the series of image processing ends.

As described above, the segmentation processing unit 2804 according to this embodiment uses at least one of a plurality of image segmentation engines including trained models trained using different training data to generate region label images. Generate. In this embodiment, each of the plurality of image segmentation engines includes a trained model trained using different learning data for at least one of the imaging region, imaging angle of view, and image resolution. The segmentation processing unit 2804 generates an area label image using an image segmentation engine that corresponds to at least one of the imaging site of the input image, the imaging angle of view, and the resolution of the image.

With such a configuration, the image processing apparatus 2800 according to this embodiment can perform image segmentation processing with higher accuracy according to the imaging conditions.

In this embodiment, the segmentation processing unit 2804 selects an image segmentation engine to be used for image segmentation processing based on the shooting conditions of the input image, but image segmentation engine selection processing is not limited to this. For example, the output unit 2807 may cause the user interface of the display unit 2820 to display the acquired imaging conditions of the input image and the group of image segmentation engines. Furthermore, an image segmentation engine used for image segmentation processing by the segmentation processing unit 2804 may be selected according to an instruction from the examiner.

Note that the output unit 2807 may cause the display unit 2820 to display the image segmentation engine group and the information of the teacher data used for learning of each image segmentation engine. Note that the information of the training data used for learning of the image segmentation engine may be displayed in any manner. For example, the image segmentation engine group may be displayed using a name associated with the training data used for learning.

In addition, the output unit 2807 may display the image segmentation engine selected by the segmentation processing unit 2804 on the user interface of the display unit 2820 and receive instructions from the examiner. In this case, the segmentation processing unit 2804 may determine whether or not to finally select the image segmentation engine as the image segmentation engine to be used for image segmentation processing according to an instruction from the examiner.

Note that the output unit 2807 may output the generated region label image and the evaluation result to another device connected to the imaging device 2810 or the image processing device 2800, as in the eighth embodiment. Depending on the settings and implementation of the image processing apparatus 2800, the output unit 2807 processes these so that they can be used by the imaging apparatus 2810 and other apparatuses, or converts them into data formats that can be transmitted to an image management system or the like. may be converted.

(Example 10)
Next, an image processing apparatus according to the tenth embodiment will be described with reference to FIGS. 28 and 33. FIG. In Embodiments 8 and 9, the imaging condition acquisition unit 2802 acquires the imaging condition group from the data structure of the input image. On the other hand, in this embodiment, the imaging condition acquisition unit uses the imaging location estimation engine to estimate the imaging part or imaging area of the input image based on the input image.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the ninth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the ninth embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatuses according to the eighth and ninth embodiments, so the same reference numerals are used for the configuration shown in FIG. 28, and description thereof is omitted. do.

An imaging condition acquisition unit 2802 according to this embodiment estimates and acquires an imaging region or an imaging region drawn in an input image acquired by the acquisition unit 2801 using an imaging location estimation engine (estimation engine). In the method of estimating the shooting location provided by the shooting location estimation engine according to the present embodiment, an estimation process using a machine learning algorithm is performed.

In this embodiment, training of a trained model related to an imaging location estimation method using a machine learning algorithm consists of a pair group of input data, which is an image, and output data, which is an imaging region label corresponding to the input data. Use the training data that has been prepared. Here, the input data is an image having specific shooting conditions assumed as a processing target (input image). The input data is preferably an image acquired from an imaging device having the same image quality tendency as the imaging device 2810, and more preferably the same model with the same settings as the imaging device 2810. FIG. The type of imaging region label, which is output data, may be an imaging region at least partially included in the input data. The types of imaging site labels, which are output data, may be, for example, "macula", "optic papilla", "macula and optic papilla", and "others".

The imaging location estimation engine according to the present embodiment includes a trained model that has undergone learning using such teacher data, so that it can determine where the imaging region and imaging region depicted in the input image are. can be output. The imaging location estimation engine can also output the probability of being the imaging region or imaging region for each imaging region label or imaging region label of a required level of detail.

By using the imaging location estimation engine, the imaging condition acquisition unit 2802 can estimate the imaging region and imaging region of the input image based on the input image, and acquire them as the imaging conditions for the input image. When the imaging location estimation engine outputs the probability of being the imaging location or imaging area for each imaging location label or imaging area label, the imaging condition acquisition unit 2802 selects the imaging location or imaging area with the highest probability. Acquired as the shooting conditions of the input image.

Next, as in the ninth embodiment, a series of image processing according to the present embodiment will be described with reference to the flowchart of FIG. Note that the processes in step S3310 and steps S3330 to S3380 according to the present embodiment are the same as those in the ninth embodiment, and thus description thereof is omitted. Note that if the input image is to be subjected to image segmentation processing unconditionally, after the processing of step S3320, the processing of step S3330 may be omitted and the processing may proceed to step S3340.

Once the input image is obtained in step S3310, the process proceeds to step S3320. In step S3320, the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image acquired in step S3310.

Specifically, according to the data format of the input image, the imaging condition group stored in the data structure forming the input image is obtained. If the imaging condition group does not include information about the imaging region or imaging region, the imaging condition acquisition unit 2802 inputs the input image to the imaging region estimation engine, and determines which imaging region and imaging region the input image captures. Estimate what was acquired. Specifically, the imaging condition acquisition unit 2802 inputs an input image to the imaging location estimation engine, evaluates the probabilities output for each imaging region label group or imaging region label group, and determines the imaging condition with the highest probability. A part or imaging region is set/acquired as an imaging condition of an input image.

Note that if the input image does not store imaging conditions other than the imaging site and imaging region, the imaging condition acquisition unit 2802 can acquire imaging information groups from the imaging device 2810 or an image management system (not shown). . Since subsequent processing is the same as the series of image processing according to the ninth embodiment, description thereof is omitted.

As described above, the imaging condition acquiring unit 2802 according to this embodiment functions as an estimating unit that estimates at least one of the imaging part and imaging region from the input image using the imaging location estimation engine including the learned model. do. An imaging condition acquisition unit 2802 inputs an input image to an imaging location estimation engine that includes a trained model in which images labeled with imaging regions and imaging regions are used as learning data. to estimate

Accordingly, the image processing apparatus 2800 according to the present embodiment can acquire imaging conditions for the imaging part and imaging region of the input image based on the input image.

Note that in this embodiment, the imaging condition acquisition unit 2802 estimates the imaging part and imaging area of the input image using the imaging part estimation engine when the imaging condition group does not include information about the imaging part and imaging area. gone. However, the situation in which the imaging site estimation engine is used to estimate the imaging site and imaging area is not limited to this. The imaging condition acquisition unit 2802 uses the imaging part estimation engine to obtain the imaging part and the imaging area even when the information about the imaging part and the imaging area included in the data structure of the input image is insufficient as information of the required level of detail. An estimation may be made for the imaging area.

The imaging condition acquisition unit 2802 estimates the imaging part and imaging area of the input image using the imaging part estimation engine regardless of whether or not the data structure of the input image contains information about the imaging part and imaging area. You may In this case, the output unit 2807 causes the display unit 2820 to display the estimation result output from the imaging location estimation engine and the information about the imaging region and imaging region included in the data structure of the input image, and the imaging condition acquisition unit 2802 performs detection. These imaging conditions may be determined according to the instructions of the operator.

(Example 11)
Next, an image processing apparatus according to the eleventh embodiment will be described with reference to FIGS. In this embodiment, the segmentation processor scales up or down the input image so that the input image has an image size that can be handled by the image segmentation engine. Also, the segmentation processing unit reduces or enlarges the output image from the image segmentation engine so that it has the image size of the input image to generate a region label image.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the eighth embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the eighth embodiment, so the same reference numerals are used for the configuration shown in FIG. 28, and the description thereof is omitted.

The segmentation processing unit 2804 according to this embodiment includes an image segmentation engine similar to the image segmentation engine according to the eighth embodiment. However, in this embodiment, the teacher data used for learning the machine learning model included in the image segmentation engine is different from the teacher data in the eighth embodiment. Specifically, in this embodiment, a group of pairs of input data and output data, which is formed by a group of images obtained by enlarging or reducing an image of input data and an image of output data so as to have a fixed image size, is used as training data. is used.

Now, with reference to FIG. 34, the teacher data of the learned model included in the image segmentation engine according to this embodiment will be described. As shown in FIG. 34, for example, consider a case where there are an input image Im3410 and an area label image Im3420 smaller than a certain image size set for the teacher data. In this case, each of the input image Im3410 and the region label image Im3420 is enlarged so as to have a constant image size set for the teacher data. Then, the enlarged image Im3411 and the enlarged area label image Im3421 are paired, and the pair is used as one of the teacher data.

As in the eighth embodiment, an image having specific shooting conditions assumed as a processing target (input image) is used as the input data of the teacher data. They are the part, the imaging method, and the imaging angle of view. That is, unlike the eighth embodiment, the image size is not included in the specific imaging conditions according to the present embodiment.

The segmentation processing unit 2804 according to this embodiment uses an image segmentation engine trained with such teacher data to perform image segmentation processing on an input image to generate a region label image. At this time, the segmentation processing unit 2804 generates a deformed image by enlarging or reducing the input image to a certain image size set for the teacher data, and inputs the deformed image to the image segmentation engine.

Also, the segmentation processing unit 2804 reduces or enlarges the output image from the image segmentation engine so that it has the image size of the input image, and generates an area label image. For this reason, the segmentation processing unit 2804 according to this embodiment can generate an area label image by performing image segmentation processing using an image segmentation engine even for an input image having an image size that could not be dealt with in the eighth embodiment. can be done.

Next, a series of image processing according to this embodiment will be described with reference to FIGS. FIG. 35 is a flowchart of segmentation processing according to this embodiment. Note that the processes of steps S2910, S2920, and steps S2950 to S2970 according to the present embodiment are the same as those processes in the eighth embodiment, and thus description thereof is omitted. Note that if image segmentation processing is unconditionally performed on the input image with respect to imaging conditions other than the image size, the processing of step S2930 may be omitted after the processing of step S2920, and the processing may proceed to step S2940.

In step S2920, similarly to the eighth embodiment, when the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image, the process proceeds to step S2930. In step S2930, the process propriety determination unit 2803 determines whether or not the input image can be processed by the image segmentation engine using the acquired imaging condition group. Specifically, the process availability determination unit 2803 determines whether or not the imaging conditions of the input image are the imaging region, imaging method, and imaging angle of view that can be dealt with by the image segmentation engine. Unlike the eighth embodiment, the process propriety determination unit 2803 does not determine the image size.

The process propriety determination unit 2803 determines the imaging region, the imaging method, and the imaging angle of view, and if it is determined that the input image can be processed, the process proceeds to step S2940. On the other hand, if the processing enable/disable determining unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970. Note that depending on the settings and implementation of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on part of the imaging region, imaging method, and imaging angle of view, An image segmentation process may be performed.

When the process moves to step S2940, the image segmentation process according to this embodiment shown in FIG. 35 is started. In the image segmentation processing according to this embodiment, first, in step S3510, the segmentation processing unit 2804 enlarges or reduces the input image to a certain image size set for the teacher data to generate a deformed image.

Next, in step S3520, the segmentation processing unit 2804 inputs the generated deformed image to the image segmentation engine, and obtains the first region label image subjected to image segmentation processing.

Then, in step S3530, the segmentation processor 2804 reduces or enlarges the first region label image to the image size of the input image to generate the final region label image. After the segmentation processor 2804 has generated the final region label image in step S3530, the image segmentation process according to this embodiment ends and the process proceeds to step S2950. Since the processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, the description thereof is omitted.

As described above, the segmentation processing unit 2804 according to this embodiment adjusts the image size of the input image to an image size that can be handled by the image segmentation engine, and inputs the image to the image segmentation engine. Also, the segmentation processing unit 2804 generates a region label image by adjusting the output image from the image segmentation engine to the original image size of the input image. As a result, the image processing apparatus 2800 of the present embodiment performs image segmentation processing on an input image having an image size not supported in the eighth embodiment, and includes ROI and VOI information that can be used for image diagnosis and image analysis. A region label image can be generated.

(Example 12)
Next, an image processing apparatus according to the twelfth embodiment will be described with reference to FIGS. In this embodiment, the segmentation processing unit generates region label images through image segmentation processing based on a certain resolution by an image segmentation engine.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the eighth embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the eighth embodiment, so the same reference numerals are used for the configuration shown in FIG. 28, and the description thereof is omitted.

The segmentation processing unit 2804 according to this embodiment includes an image segmentation engine similar to that of the eighth embodiment. However, in this embodiment, the teacher data used for learning the machine learning model included in the image segmentation engine is different from the teacher data in the eighth embodiment. Specifically, after enlarging or reducing the image group to an image size such that the resolution of the image group constituting the pair group of the input data and output data of the teacher data becomes a certain resolution, the image group is enlarged or reduced to a sufficiently large certain size. It is padded to the image size. Here, the resolution of an image group means, for example, the spatial resolution of an imaging device or the resolution of an imaging area.

Now, with reference to FIG. 36, teacher data for the image segmentation engine according to this embodiment will be described. As shown in FIG. 36, for example, consider a case where there are an image Im3610 and a region label image Im3620 having a resolution lower than the fixed resolution set for the teacher data. In this case, each of the image Im3610 and the area label image Im3620 is enlarged so as to have a constant resolution set for the teacher data. Furthermore, each of the enlarged image Im3610 and the region label image Im3620 is padded so as to have a constant image size set for the teacher data. Then, the enlarged and padded image Im3611 and the area label image Im3621 are paired, and the pair is used as one of the teacher data.

Note that the constant image size set for the training data is the maximum possible image size when an image assumed to be a processing target (input image) is enlarged or reduced so as to have a constant resolution. If the fixed image size is not large enough, the trained model may not be able to handle the image size when the image input to the image segmentation engine is scaled up.

In addition, the region to be padded is filled with a constant pixel value, filled with neighboring pixel values, or mirror-padded according to the characteristics of the trained model so that the image segmentation process can be performed effectively. As in the eighth embodiment, an image having specific imaging conditions assumed to be processed is used as the input data. is a corner. That is, unlike the eighth embodiment, the image size is not included in the specific imaging conditions according to the present embodiment.

The segmentation processing unit 2804 according to this embodiment uses an image segmentation engine including a trained model trained with such teacher data to perform image segmentation processing on an input image to generate a region label image. At this time, the segmentation processing unit 2804 generates a deformed image by enlarging or reducing the input image to a certain resolution set for the teacher data. Also, the segmentation processing unit 2804 generates a padding image by padding the deformed image so that it has a constant image size set for the teacher data, and inputs the padding image to the image segmentation engine.

The segmentation processing unit 2804 also trims the first region label image output from the image segmentation engine by the padded region to generate a second region label image. After that, the segmentation processing unit 2804 reduces or expands the generated second region label image to the image size of the input image to generate a final region label image.

For this reason, the segmentation processing unit 2804 according to this embodiment can generate a region label image by performing image segmentation processing using the image segmentation engine even for an input image having an image size that could not be handled in the eighth embodiment. .

Next, a series of image processing according to this embodiment will be described with reference to FIGS. FIG. 37 is a flowchart of image segmentation processing according to this embodiment. Note that the processes of steps S2910, S2920, and steps S2950 to S2970 according to the present embodiment are the same as those processes in the eighth embodiment, and thus description thereof is omitted. Note that if image segmentation processing is unconditionally performed on the input image with respect to imaging conditions other than the image size, the processing of step S2930 may be omitted after the processing of step S2920, and the processing may proceed to step S2940.

In step S2920, similarly to the eighth embodiment, after the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image, the process proceeds to step S2930. In step S2930, the process propriety determination unit 2803 determines whether or not the input image can be processed by the image segmentation engine using the acquired imaging condition group. Specifically, the process availability determination unit 2803 determines whether or not the imaging conditions of the input image are the imaging region, imaging method, and imaging angle of view that can be dealt with by the image segmentation engine. Unlike the eighth embodiment, the process propriety determination unit 2803 does not determine the image size.

The process propriety determination unit 2803 determines the imaging region, the imaging method, and the imaging angle of view, and if it is determined that the input image can be processed, the process proceeds to step S2940. On the other hand, if the processing enable/disable determining unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970. Note that depending on the settings and implementation of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on part of the imaging region, imaging method, and imaging angle of view, An image segmentation process may be performed.

When the process moves to step S2940, the image segmentation process according to this embodiment shown in FIG. 37 is started. In the image segmentation process according to this embodiment, first, in step S3710, the segmentation processing unit 2804 expands or reduces the input image to a certain resolution set for the teacher data, and generates a deformed image.

Next, in step S3720, the segmentation processing unit 2804 generates a padding image by padding the generated deformed image so that it has the image size set for the teacher data. At this time, the segmentation processing unit 2804 fills the region to be padded with a constant pixel value, with neighboring pixel values, or with mirror padding in accordance with the characteristics of the trained model so that the image segmentation processing can be performed effectively. or

In step S3730, the segmentation processing unit 2804 inputs the padding image to the image segmentation engine to obtain the first region label image subjected to image segmentation processing.

Next, in step S3740, the segmentation processing unit 2804 trims the first area label image by the area padded in step S3720 to generate a second area label image.

Then, in step S3750, the segmentation processor 2804 reduces or enlarges the second region label image to the image size of the input image to generate the final region label image. After the segmentation processor 2804 has generated the final region label image in step S3750, the image segmentation process according to this embodiment ends and the process proceeds to step S2950. Since the processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, the description thereof is omitted.

As described above, the segmentation processing unit 2804 according to this embodiment adjusts the image size of the input image so that the input image has a predetermined resolution. In addition, the segmentation processing unit 2804 generates an image by performing padding on the input image whose image size has been adjusted so that the image size can be handled by the image segmentation engine, and inputs the image to the image segmentation engine. After that, the segmentation processing unit 2804 trims the output image from the image segmentation engine by the padded area. Then, the segmentation processing unit 2804 generates a region label image by adjusting the image size of the trimmed image to the original image size of the input image.

As a result, the segmentation processing unit 2804 of this embodiment can generate a region label image by performing image segmentation processing with the image segmentation engine even for an input image having an image size that could not be dealt with in the eighth embodiment. Also, by using an image segmentation engine trained with training data based on resolution, an input image may be segmented more efficiently than the image segmentation engine according to the tenth embodiment, which processes images of the same image size.

(Example 13)
Next, an image processing apparatus according to the thirteenth embodiment will be described with reference to FIGS. In this embodiment, the segmentation processing unit generates region label images by performing image segmentation processing on an input image for each region of a certain image size.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the eighth embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the eighth embodiment, so the same reference numerals are used for the configuration shown in FIG. 28, and the description thereof is omitted.

The segmentation processing unit 2804 according to this embodiment includes an image segmentation engine similar to that of the eighth embodiment. However, in this embodiment, the teacher data used for learning the machine learning model included in the image segmentation engine is different from the teacher data in the eighth embodiment. Specifically, a group of pairs of input data, which is an input image, and output data, which is a region label image corresponding to the input image, which constitutes the training data, is set so that the positional relationship between the input image and the region label image corresponds. It consists of a rectangular area image of a fixed image size.

Here, the training data of the image segmentation engine according to this embodiment will be described with reference to FIG. As shown in FIG. 38, with respect to one pair group forming teacher data, for example, consider a case where there is an image Im3810 that is an input image and an area label image Im3820 that is a corresponding area label image. In this case, in the eighth embodiment, the input data of the teacher data is the image Im3810, and the output data is the region label image Im3820.

In contrast, in this embodiment, the rectangular area image R3811 of the image Im3810 is used as input data, and the rectangular area image R3821, which is the same (corresponding) imaging area as the rectangular area image R3811 in the area label image Im3820, is output data. and A pair of teacher data (hereinafter referred to as a first pair of rectangular region images) is formed by the rectangular region image R3811 as input data and the rectangular region image R3821 as output data.

Here, the rectangular area image R3811 and the rectangular area image R3821 are images of a fixed image size. Note that image Im3810 and region label image Im3820 may be aligned by any method. Also, the corresponding positional relationship between the rectangular area image R3811 and the rectangular area image R3821 may be identified by an arbitrary method such as template matching. Depending on the design of the machine learning model included in the image segmentation engine, the image size and the number of dimensions of the input data and the output data may be different. For example, the input data is part of a B-scan image (two-dimensional image) and the output data is part of an A-scan image (one-dimensional).

The constant image size described above can be determined, for example, from the common divisor of the pixel number groups of the corresponding dimensions for the image size group of images assumed to be processed (input images). In this case, it is possible to prevent overlapping of the positional relationships of the rectangular area images output by the image segmentation engine.

Specifically, for example, the image assumed to be processed is a two-dimensional image, the first image size in the image size group is 500 pixels wide and 500 pixels high, and the second image size is Consider the case of 100 pixels wide and 100 pixels high. Here, a certain image size is selected for the rectangular area images R3811 and R3821 from the common divisor of each side. In this case, for example, a fixed image size is selected from 100 pixels wide and 100 pixels high, 50 pixels wide and 50 pixels high, and 25 pixels wide and 25 pixels high. When the image assumed to be processed is three-dimensional, the number of pixels is determined with respect to width, height, and depth.

A plurality of rectangular areas can be set for one pair of an image corresponding to input data and an area label image corresponding to output data. Therefore, for example, the rectangular area image R3812 in the image Im3810 is used as input data, and the rectangular area image R3822 in the area label image Im3820, which is the same shooting area as the rectangular area image R3812, is used as output data. The rectangular area image R3812 as the input data and the rectangular area image R3822 as the output data form a pair of teacher data. Thereby, a rectangular area image pair different from the first rectangular area image pair can be created.

By creating a large number of pairs of rectangular area images while changing the image of the rectangular area to an image of different coordinates, it is possible to enrich the pair group constituting the teacher data. Efficient image segmentation processing can be expected by an image segmentation engine that has been trained using pairs that constitute the training data. However, pairs that do not contribute to the image segmentation processing of the trained model can be prevented from being added to the training data.

As the input data and the output data of the teacher data, an image that draws one layer or one labeled region can be used as the teacher data. Also, as the input data and output data of the teacher data, it is possible to use an image of an area for drawing a plurality of layers, for example two layers, more preferably three or more layers. Similarly, it is also possible to use an area image in which a plurality of labeled areas are drawn in the area label image. In these cases, compared to the case of using an image that draws one layer or one labeled area as training data, the positional relationship of the learned layers and areas makes it easier to use the trained model. We can expect to be able to perform image segmentation processing appropriately.

Further, for example, consider a case where the structures and positions of the imaging targets drawn in the pair of rectangular area images created from the input image and the rectangular area images created from the area label images are significantly different. In this case, an image segmentation engine trained using such teacher data may output region label images with low accuracy. Therefore, such pairs can be removed from the training data.

As in the eighth embodiment, an image having specific imaging conditions assumed to be processed is used as input data for teacher data. , and the shooting angle of view. That is, unlike the eighth embodiment, the image size is not included in the specific imaging conditions according to the present embodiment.

The segmentation processing unit 2804 according to this embodiment uses an image segmentation engine trained with such teacher data to perform image segmentation processing on an input image to generate a region label image. At this time, the segmentation processing unit 2804 divides the input image into a group of rectangular area images which are continuous without gaps and have a constant image size set for the teacher data. A segmentation processing unit 2804 uses an image segmentation engine to perform image segmentation processing on each of the divided rectangular area image groups, and generates a divided area label image group. After that, the segmentation processing unit 2804 arranges and combines the generated group of divided area label images according to the positional relationship of the input image to generate a final area label image.

In this manner, the segmentation processing unit 2804 of this embodiment performs image segmentation processing on the input image in units of rectangular regions, and combines the images that have undergone the image segmentation processing. As a result, it is possible to generate an area label image by performing image segmentation processing even on an image having an image size that is not supported in the eighth embodiment.

Next, a series of image processing according to this embodiment will be described with reference to FIGS. FIG. 39 is a flowchart of image segmentation processing according to this embodiment. Note that the processes of steps S2910, S2920, and steps S2950 to S2970 according to the present embodiment are the same as those processes in the eighth embodiment, and thus description thereof is omitted. Note that if image segmentation processing is unconditionally performed on the input image with respect to imaging conditions other than the image size, the processing of step S2930 may be omitted after the processing of step S2920, and the processing may proceed to step S2940.

In step S2920, as in the eighth embodiment, after the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image, the process proceeds to step S2930. In step S2930, the process propriety determination unit 2803 determines whether or not the input image can be processed by the image segmentation engine using the acquired imaging condition group. Specifically, the process availability determination unit 2803 determines whether or not the imaging conditions of the input image are the imaging region, imaging method, and imaging angle of view that can be dealt with by the image segmentation engine. Unlike the eighth embodiment, the process propriety determination unit 2803 does not determine the image size.

The process propriety determination unit 2803 determines the imaging region, the imaging method, and the imaging angle of view, and if it is determined that the input image can be processed, the process proceeds to step S2940. On the other hand, if the processing enable/disable determining unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970. Note that depending on the settings and implementation of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on part of the imaging region, imaging method, and imaging angle of view, An image segmentation process may be performed.

When the process moves to step S2940, the image segmentation process according to this embodiment shown in FIG. 39 is started. In the image segmentation process according to the present embodiment, first, in step S3910, as shown in FIG. 40, the input image is divided into a group of continuously continuous rectangular area images having a constant image size set for the teacher data. Here, FIG. 40 shows an example of dividing the input image Im4010 into a group of rectangular area images R4011 to R4026 each having a fixed image size. Depending on the design of the machine learning model included in the image segmentation engine, the image size and the number of dimensions of the input image and the output image may differ. In that case, adjustment is made by overlapping or separating the division positions of the input image described above so that the combined region label image generated in step S3920 is free of defects.

Next, in step S3920, the segmentation processing unit 2804 performs image segmentation processing on each of the rectangular area images R4011 to R4026 using the image segmentation engine to generate a group of divided area label images.

Then, in step S3930, the segmentation processing unit 2804 arranges and joins each of the generated divided region label image groups with the same positional relationship as each of the rectangular region images R4011 to R4026 divided from the input image. This allows the segmentation processing unit 2804 to generate a region label image.

After the segmentation processing unit 2804 has generated the region label image in step S3930, the image segmentation processing according to this embodiment ends, and the process proceeds to step S2950. Since the processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, the description thereof is omitted.

As described above, the segmentation processing unit 2804 according to this embodiment divides the input image into a plurality of rectangular region images R4011 to R4026 each having a predetermined image size. After that, the segmentation processing unit 2804 inputs the plurality of divided rectangular region images R4011 to R4026 to the image segmentation engine to generate a plurality of divided region label images, and integrates the plurality of divided region label images to obtain a region label image. Generate an image. Note that if the positional relationship between the rectangular area groups overlaps at the time of integration, the pixel value groups of the rectangular area groups are integrated or overwritten.

As a result, the segmentation processing unit 2804 of this embodiment can generate a region label image by performing image segmentation processing using the image segmentation engine even for an input image having an image size that could not be dealt with in the eighth embodiment. can. Also, if the teacher data is created from a plurality of images divided into predetermined image sizes, a large amount of teacher data can be created from a small number of images. Therefore, in this case, the number of input images and region label images for creating teacher data can be reduced.

Also, the trained model included in the image segmentation engine according to the present embodiment is a model trained using a tomographic image including two or more layers as input data and a region label image corresponding to the tomographic image as output data. be. For this reason, compared to the case of using an image that draws one layer or one labeled region as training data, the positional relationship of the learned layers and regions makes it possible to use the trained model to create a more appropriate image. We can expect to be able to perform segmentation processing.

(Example 14)
Next, an image processing apparatus according to the fourteenth embodiment will be described with reference to FIGS. In this embodiment, the evaluation unit selects the region label image with the highest accuracy from among the plurality of region label images output from the plurality of image segmentation engines according to the examiner's instructions.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the eighth embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the eighth embodiment, so the same reference numerals are used for the configuration shown in FIG. 28, and the description thereof is omitted.

The segmentation processing unit 2804 according to this embodiment performs image segmentation processing on an input image using two or more image segmentation engines each including a trained model machine-learned using different teacher data.

Here, a method of creating a teacher data group according to this embodiment will be described. Specifically, first, pairs of input data, which are images shot under various shooting conditions, and output data, which are area label images, are prepared. Next, a teacher data group is created by grouping the pair groups according to an arbitrary combination of imaging conditions. For example, first training data composed of pairs acquired by combining the first imaging conditions, second training data composed of pairs acquired by combining the second imaging conditions, and so on. , is created as a training data group.

Each training data is then used to machine-learn a machine-learning model contained in a separate image segmentation engine. For example, providing a first image segmentation engine that includes a trained model trained on first training data. In addition, a group of image segmentation engines are prepared, such as a second image segmentation engine corresponding to the learned model trained with the second teacher data.

Such image segmentation engines differ in the training data used to train the learned models they contain. Therefore, such an image segmentation engine differs in the accuracy with which an input image can be subjected to image segmentation processing depending on the photographing conditions of the image input to the image segmentation engine. Specifically, the first image segmentation engine has high accuracy in image segmentation processing for input images acquired by shooting under the first combination of shooting conditions. On the other hand, the first image segmentation engine has low accuracy in image segmentation processing for images acquired by shooting under the second combination of shooting conditions. Similarly, the second image segmentation engine has high accuracy in image segmentation processing for input images acquired by photographing under the second combination of photographing conditions. On the other hand, the second image segmentation engine has low accuracy in image segmentation processing for images acquired by photographing under the first combination of photographing conditions.

Each of the teacher data is composed of a pair group grouped according to a combination of shooting conditions, so that the image quality tendencies of the image groups forming the pair group are similar. For this reason, the image segmentation engine can perform image segmentation processing with higher accuracy than the image segmentation engine according to the eighth embodiment, provided that the corresponding imaging conditions are combined. Note that the combination of imaging conditions for grouping pairs of teacher data may be arbitrary, and may be, for example, a combination of two or more of the imaging region, imaging angle of view, and image resolution. Also, grouping of teacher data may be performed based on one imaging condition as in the ninth embodiment.

The evaluation unit 2805 evaluates a plurality of region label images generated by the segmentation processing unit 2804 using a plurality of image segmentation engines in the same manner as in the eighth embodiment. After that, when there are a plurality of region label images for which the evaluation result is a true value, the evaluation unit 2805 selects the region label image with the highest accuracy among the plurality of region label images according to the instruction of the examiner, and outputs the region label image. It is judged as the region label image that should be done. Note that the evaluation unit 2805 may perform evaluation using a region label image evaluation engine including a learned model, or a knowledge-based region label image evaluation engine, as in the eighth embodiment. good too.

The analysis unit 2806 uses the input image and the area label image determined by the evaluation unit 2805 as the area label image to be output, and performs image analysis processing on the input image in the same manner as in the eighth embodiment. The output unit 2807 can display the area label image determined as the area label image to be output and the analysis result on the display unit 2820 or output them to another device, as in the eighth embodiment. Note that the output unit 2807 can display a plurality of region label images whose evaluation results are true values on the display unit 2820, and the evaluation unit 2805 displays the most Region label images with high accuracy can be selected.

As a result, the image processing apparatus 2800 can output the region label image with the highest accuracy according to the examiner's instruction, among the plurality of region label images generated using the plurality of image segmentation engines.

A series of image processing according to this embodiment will be described below with reference to FIGS. FIG. 41 is a flowchart of a series of image processing according to this embodiment. It should be noted that the processing of steps S4110 and S4120 according to this embodiment is the same as the processing of steps S2910 and S2920 in the eighth embodiment, and thus description thereof will be omitted. Note that if image segmentation processing is to be performed on the input image unconditionally with respect to the imaging conditions, the processing in step S4130 may be omitted after the processing in step S4120, and the processing may proceed to step S4140.

In step S4120, as in the eighth embodiment, after the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image, the process proceeds to step S4130. In step S4130, the process availability determination unit 2803 determines whether or not any of the image segmentation engines used by the segmentation processing unit 2804 can handle the input image, using the acquired imaging condition group.

If the process propriety determination unit 2803 determines that none of the image segmentation engines can process the input image, the process proceeds to step S4180. On the other hand, if the process propriety determination unit 2803 determines that any one of the image segmentation engines can process the input image, the process proceeds to step S4140. It should be noted that even if the image segmentation engine determines that part of the imaging conditions cannot be dealt with, as in the eighth embodiment, step S4140 may be executed depending on the settings and implementation of the image processing apparatus 2800. good.

In step S4140, the segmentation processing unit 2804 inputs the input image obtained in step S4110 to each of the image segmentation engines to generate a region label image group. Note that the segmentation processing unit 2804 may input the input image only to an image segmentation engine that has been determined by the processability determination unit 2803 to be capable of handling the input image.

In step S4150, the evaluation unit 2805 evaluates the region label image group generated in step S4140 using the region label image evaluation engine as in the eighth embodiment. In step S4160, if there are a plurality of area label images with true evaluation results (image evaluation indices), the evaluation unit 2805 selects/determines the area label image to be output according to the examiner's instruction.

In this case, first, the output unit 2807 causes the user interface of the display unit 2820 to display the region label image group for which the evaluation result is true. Here, FIG. 42 shows an example of the interface. The interface displays an input image UI4210 and area label images UI4220, UI4230, UI4240, and UI4250 for which the evaluation result is true. The examiner operates an arbitrary input device (not shown) to designate the region label image with the highest precision among the image group (region label images UI4220 to UI4250). The evaluation unit 2805 selects the region label image designated by the examiner as the region label image to be output.

If there is only one area label image with a true value as the evaluation result, that area label image is selected/determined as the area label image to be output. If there is no region label image with a true value as the evaluation result, the evaluation unit 2805 determines not to output the region label image generated by the segmentation processing unit 2804, and generates and outputs an image without region label. / is selected, and the process advances to step S4170.

In step S4170, similarly to the eighth embodiment, the analysis unit 2806 uses the input image and the area label image determined by the evaluation unit 2805 to be the area label image to be output, and performs image analysis processing on the input image. Note that if the evaluation unit 2805 outputs an image without region label, the analysis unit 2806 advances the process to step S4180 without performing image analysis processing.

In step S4180, the output unit 2807 causes the display unit 2820 to display the region label image determined as the region label image to be output and the image analysis result. Note that the output unit 2807 may cause the imaging device 2810 or another device to display or store the region label image and the image analysis result instead of displaying the region label image and the image analysis result on the display unit 2820 . Depending on the settings and implementation of the image processing apparatus 2800, the output unit 2807 processes these so that they can be used by the imaging apparatus 2810 and other apparatuses, or converts them into data formats that can be transmitted to an image management system or the like. may be converted. Also, the output unit 2807 is not limited to outputting both the region label image and the image analysis result, and may output only one of them.

On the other hand, if it is determined in step S4130 that the image segmentation processing is impossible, the output unit 2807 outputs an image without region label and causes the display unit 2820 to display it. It should be noted that instead of outputting the region-unlabeled image, a signal indicating that the image segmentation process was not possible may be sent to the imaging device 2810 .

Moreover, even if it is determined in step S4150 that the image segmentation processing could not be performed properly (the generated region label image is not output), the output unit 2807 outputs the region labelless image, and the display unit 2820 to display. In this case as well, the output unit 2807 may send a signal indicating that the image segmentation processing could not be properly performed to the imaging device 2810 instead of outputting the region-unlabeled image. When the output processing in step S4170 ends, the series of image processing ends.

As described above, the segmentation processing unit 2804 according to this embodiment uses multiple image segmentation engines each containing a different trained model to generate multiple region label images from the input image. Further, the evaluation unit 2805 selects at least one of the plurality of pieces of region information determined to be output by evaluating the plurality of pieces of region information according to instructions from the examiner (user). More specifically, when there are a plurality of region label images whose image evaluation index is a true value, the evaluation unit 2805 selects the region label image to be output with the highest accuracy according to the examiner's instruction. Select/determine as Accordingly, the image processing apparatus 2800 can output a highly accurate region label image according to the examiner's instruction, among the plurality of region label images generated using the plurality of image segmentation engines.

In this embodiment, the evaluation unit 2805 selects/determines the region label image with the highest accuracy as the region label image to be output according to the instruction of the examiner. On the other hand, the evaluation unit 2805 may select/determine a plurality of region label images whose evaluation results are true values as region label images to be output in accordance with instructions from the examiner. In this case, the analysis unit 2806 performs image analysis processing on a plurality of area label images selected as area label images to be output. Also, the output unit 2807 outputs the selected plurality of region label images and the analysis results of the plurality of region label images.

Also, in this embodiment, the evaluation unit 2805 selects the area label image to be output from the plurality of area label images whose evaluation results are true values in accordance with the examiner's instruction. In response to this, the output unit 2807 causes the display unit 2820 to display all the region label images generated by the segmentation processing unit 2804, and the evaluation unit 2805 displays the plurality of region label images according to instructions from the examiner. A region label image to be output from the image may be selected. Also in this case, the evaluation unit 2805 may select/determine a plurality of region label images as region label images to be output according to instructions from the examiner.

(Example 15)
Next, an image processing apparatus according to the fifteenth embodiment will be described with reference to FIGS. 28 and 41. FIG. In the image processing apparatus according to the fourteenth embodiment, the evaluation unit 2805 selects/determines an image to be output according to the examiner's instruction from among a plurality of region label images for which the evaluation result by the evaluation unit 2805 is a true value. On the other hand, in this embodiment, the evaluation unit selects/determines the area label image to be output from among the plurality of area label images whose evaluation results are true values, based on a predetermined selection criterion.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the fourteenth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the fourteenth embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatuses according to the eighth and fourteenth embodiments, so the same reference numerals are used for the configuration shown in FIG. 28, and the description thereof is omitted. do.

The evaluation unit 2805 according to the present embodiment uses the region label image evaluation engine to evaluate a plurality of region label images generated by the segmentation processing unit 2804, and output according to the image evaluation index and predetermined selection criteria. Select a region label image.

A series of image processing according to this embodiment will be described below with reference to FIG. Note that the processes other than step S4160 according to the present embodiment are the same as the processes in the fourteenth embodiment, so description thereof will be omitted. Note that if image segmentation processing is to be performed on the input image unconditionally with respect to the imaging conditions, the processing in step S4130 may be omitted after the processing in step S4120, and the processing may proceed to step S4140.

In step S4160, if there are a plurality of region label images for which the evaluation result in step S4150 is a true value, the evaluation unit 2805 selects a region label image to be output among the plurality of region label images according to a predetermined selection criterion. Select/judge. For example, the evaluation unit 2805 selects the region label image for which the true value is output as the first evaluation result in chronological order. Note that the selection criteria are not limited to this, and may be arbitrarily set according to a desired configuration. For example, the evaluation unit 2805 selects the region label generated by the image segmentation engine that has the closest (matching) combination of the shooting conditions of the input image and the shooting conditions of the learning data among the region labeled images for which the evaluation result is a true value. Images may be selected/judged.

Further, when the evaluation results for all region label images are false values, the evaluation unit 2805 determines that the image segmentation processing could not be performed appropriately, generates a regionless label image, and outputs/outputs it. select. Since the processing after step S4170 is the same as the processing after step S4170 in the fourteenth embodiment, the description thereof is omitted.

As described above, in the image processing apparatus 2800 according to this embodiment, the segmentation processing unit 2804 uses a plurality of image segmentation engines to generate a plurality of region label images from the input image. The evaluation unit 2805 selects at least one of the evaluated region-labeled images or the region-unlabeled image to be output based on predetermined selection criteria. The output unit 2807 outputs the region label image selected by the evaluation unit 2805. FIG.

Accordingly, in the image processing apparatus 2800 according to this embodiment, it is possible to prevent an area label image for which image segmentation processing has failed from being output based on the output of the area label image evaluation engine. Further, when there are a plurality of area label images whose image evaluation index output by the area label image evaluation engine is a true value, one of them can be automatically selected and displayed or output.

In this embodiment, at least one of the plurality of area label images whose image evaluation index is the true value is selected and output. may be configured to output all of In this case, the analysis unit 2806 performs image analysis on all region label images output from the evaluation unit 2805 . Also, the output unit 2807 may display all the region label images and the corresponding analysis results output from the evaluation unit 2805 on the display unit 2820, or may output them to another device.

(Example 16)
Next, an image processing apparatus according to the sixteenth embodiment will be described with reference to FIGS. 28 and 29. FIG. In this embodiment, first, a segmentation processing unit divides a three-dimensional input image into a plurality of two-dimensional images (two-dimensional image group). Next, the two-dimensional image group is input to the image segmentation engine, and the segmentation processing unit combines the output image group from the image segmentation engine to generate a three-dimensional region label image.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the eighth embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the eighth embodiment, so the same reference numerals are used for the configuration shown in FIG. 28, and the description thereof is omitted.

The acquisition unit 2801 according to this embodiment acquires a three-dimensional image composed of a group of structurally continuous two-dimensional images. Specifically, the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a group of OCT B-scan images (tomographic images).

The segmentation processing unit 2804 uses the image segmentation engine according to this embodiment to perform segmentation processing on the three-dimensional image, which is the input image, and generates a plurality of two-dimensional region label images. A pair group of input data and output data, which is training data for the image segmentation engine according to the present embodiment, is composed of an image group of two-dimensional images. A segmentation processing unit 2804 divides the obtained three-dimensional image into a plurality of two-dimensional images, and inputs each two-dimensional image to the image segmentation engine. This allows the segmentation processing unit 2804 to generate a plurality of two-dimensional region label images. Furthermore, the segmentation processing unit 2804 arranges and joins a plurality of two-dimensional area label images in the arrangement of the two-dimensional images before division to generate a three-dimensional area label image.

The evaluation unit 2805 uses the region label image evaluation engine to determine whether the three-dimensional region label image is a plausible region label image. If the evaluation result is a true value, the evaluation unit 2805 determines and outputs the three-dimensional area label image as an area label image to be output. On the other hand, if the evaluation result is a false value, the evaluation unit 2805 generates and outputs a three-dimensional area-unlabeled image.

Note that when the region label image evaluation engine includes a trained model, a three-dimensional region label image and an image evaluation index can be used as training data for the trained model. Moreover, the evaluation unit 2805 may evaluate each of the two-dimensional area label images before combination.

The analysis unit 2806 performs image analysis processing on the three-dimensional region label image determined by the evaluation unit 2805 to be a plausible region label image. Note that the analysis unit 2806 may perform image analysis processing on each of the two-dimensional area label images before combining. Also, when the evaluation unit 2805 outputs a three-dimensional area-unlabeled image, the analysis unit 2806 does not perform image analysis.

The output unit 2807 outputs a three-dimensional region label image and analysis results. When the output unit 2807 causes the display unit 2820 to display the generated 3D area label image, the display mode of the 3D area label image may be arbitrary.

Next, a series of image processing according to this embodiment will be described with reference to FIG. Note that the processes of steps S2910 to S2930 and steps S2950 to S2970 according to the present embodiment are the same as those in the eighth embodiment, and therefore description thereof is omitted. However, in step S2910, the acquisition unit 2801 acquires a three-dimensional image. Note that if image segmentation processing is to be performed on the input image unconditionally with respect to the shooting conditions, the processing of step S2930 may be omitted after the processing of step S2920, and the processing may proceed to step S2940.

In step S2930, if the processability determination unit 2803 determines that the input image can be processed by the image segmentation engine, the process proceeds to step S2940. In step S2940, the segmentation processing unit 2804 divides the acquired 3D image into a plurality of 2D images. The segmentation processing unit 2804 inputs each of the plurality of divided two-dimensional images to the image segmentation engine to generate a plurality of two-dimensional region label images. The segmentation processing unit 2804 combines a plurality of generated two-dimensional region label images based on the acquired three-dimensional image to generate a three-dimensional region label image. Since the processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, the description thereof is omitted.

As described above, the segmentation processing unit 2804 according to this embodiment divides the input image into a plurality of images with dimensions lower than that of the input image, and inputs each divided image to the segmentation engine. More specifically, the segmentation processing unit 2804 divides the three-dimensional input image into a plurality of two-dimensional images and inputs them to the image segmentation engine. A segmentation processing unit 2804 combines a plurality of two-dimensional region label images output from the image segmentation engine to generate a three-dimensional region label image.

As a result, the segmentation processing unit 2804 according to this embodiment can perform image segmentation processing on the 3D image using an image segmentation engine that includes a trained model that has been trained using the teacher data of the 2D image. can.

Note that the segmentation processing unit 2804 according to this embodiment divides the three-dimensional input image into a plurality of two-dimensional images and performs image segmentation processing. However, the target for performing the processing related to the division is not limited to the three-dimensional input image. For example, the segmentation processing unit 2804 may divide a two-dimensional input image into a plurality of one-dimensional images and perform image segmentation processing. Also, the segmentation processing unit 2804 may divide a four-dimensional input image into a plurality of three-dimensional images or a plurality of two-dimensional images and perform image segmentation processing.

(Example 17)
Next, an image processing apparatus according to the seventeenth embodiment will be described with reference to FIGS. 28 and 29. FIG. In this embodiment, the segmentation processing unit divides a three-dimensional input image into a plurality of two-dimensional images, and the plurality of two-dimensional images are subjected to image segmentation processing in parallel by a plurality of image segmentation engines. A segmentation processor then generates a three-dimensional region label image by combining the output images from each image segmentation engine.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the sixteenth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the sixteenth embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatuses according to the eighth and sixteenth embodiments, so the same reference numerals are used for the configuration shown in FIG. 28, and the description thereof is omitted. do.

A segmentation processing unit 2804 according to the present embodiment uses a plurality of image segmentation engines similar to those of the sixteenth embodiment to perform image segmentation processing on a three-dimensional image that is an input image and generate a three-dimensional region label image. Note that the plurality of image segmentation engine groups used by the segmentation processing unit 2804 may be implemented in two or more device groups so as to be able to perform distributed processing via circuits or networks, or may be implemented in a single device. may have been

A segmentation processing unit 2804 divides the acquired three-dimensional image into a plurality of two-dimensional images as in the sixteenth embodiment. A segmentation processing unit 2804 uses a plurality of image segmentation engines for a plurality of two-dimensional images to share (in parallel) image segmentation processing and generate a plurality of two-dimensional area label images. A segmentation processing unit 2804 combines a plurality of two-dimensional region label images output from a plurality of image segmentation engines based on a three-dimensional image to be processed to generate a three-dimensional region label image. More specifically, the segmentation processing unit 2804 arranges and joins a plurality of two-dimensional region label images in the arrangement of the two-dimensional images before division to generate a three-dimensional region label image.

Next, a series of image processing according to this embodiment will be described with reference to FIG. Note that the processes of steps S2910 to S2930 and steps S2950 to S2970 according to the present embodiment are the same as those in the sixteenth embodiment, and thus description thereof is omitted. Note that if image segmentation processing is to be performed on the input image unconditionally with respect to the shooting conditions, the processing of step S2930 may be omitted after the processing of step S2920, and the processing may proceed to step S2940.

In step S2930, if the processability determination unit 2803 determines that the input image can be processed by the image segmentation engine, the process proceeds to step S2940. In step S2940, the segmentation processing unit 2804 divides the acquired 3D image into a plurality of 2D images. The segmentation processing unit 2804 inputs each of the plurality of divided two-dimensional images to a plurality of image segmentation engines, performs image segmentation processing in parallel, and generates a plurality of two-dimensional region label images. The segmentation processing unit 2804 combines a plurality of generated two-dimensional region label images based on the acquired three-dimensional image to generate a three-dimensional region label image.

As described above, the segmentation processor 2804 according to this embodiment includes multiple image segmentation engines. A segmentation processing unit 2804 divides a three-dimensional input image into a plurality of two-dimensional images, uses a plurality of image segmentation engines in parallel for the plurality of divided two-dimensional images, and generates a plurality of two-dimensional region labels. Generate an image. A segmentation processing unit 2804 integrates a plurality of two-dimensional region label images to generate a three-dimensional region label image.

As a result, the segmentation processing unit 2804 according to this embodiment can perform image segmentation processing on the 3D image using an image segmentation engine that includes a trained model that has been trained using the teacher data of the 2D image. can. In addition, compared with the sixteenth embodiment, the three-dimensional image can be subjected to image segmentation processing more efficiently.

Note that, as in the sixteenth embodiment, the object for which the segmentation processing by the segmentation processing unit 2804 is to be performed is not limited to the three-dimensional input image. For example, the segmentation processing unit 2804 may divide a two-dimensional input image into a plurality of one-dimensional images and perform image segmentation processing. Also, the segmentation processing unit 2804 may divide a four-dimensional input image into a plurality of three-dimensional images or a plurality of two-dimensional images and perform image segmentation processing.

Also, the training data for the plurality of image segmentation engines may be different training data according to the processing target to be processed by each image segmentation engine. For example, a first image segmentation engine may be trained with training data for a first field of view, and a second image segmentation engine may be trained with training data for a second field of view. In this case, each image segmentation engine can perform image segmentation processing of two-dimensional images with higher accuracy.

Furthermore, similarly to the segmentation processing unit 2804, the evaluation unit 2805 can evaluate three-dimensional region label images in parallel using a plurality of region label image evaluation engines including trained models. In this case, the evaluation unit 2805 evaluates a plurality of two-dimensional region label images generated by the segmentation processing unit 2804 using a plurality of region label image evaluation engines in parallel.

Thereafter, when the image evaluation index for each two-dimensional region label image is a true value, the evaluation unit 2805 can determine that the three-dimensional region label image is a plausible region label image and output it. can. In this case, the trained model training data included in the region label image evaluation engine can be composed of two-dimensional region label images and image evaluation indices. Note that the evaluation unit 2805 determines that the three-dimensional region label image is a plausible region label image and outputs it when the image evaluation index for a part of each two-dimensional region label image is a true value. can also

(Example 18)
Next, an image processing apparatus according to the eighteenth embodiment will be described with reference to FIGS. 29 and 43. FIG. In this embodiment, the acquisition unit acquires the input image from the image management system instead of the imaging device.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the eighth embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus 2800 according to the eighth embodiment, so the same reference numerals are used for the configuration shown in FIG. 28 and the description thereof is omitted.

FIG. 43 shows a schematic configuration of an image processing apparatus 2800 according to this embodiment. An image processing apparatus 2800 according to this embodiment is connected to an image management system 4300 and a display unit 2820 via arbitrary circuits and networks. The image management system 4300 is a device and system for receiving and storing images captured by an arbitrary image capturing device and images subjected to image processing. In addition, the image management system 4300 can transmit images in response to requests from connected devices, perform image processing on saved images, and request other devices to perform image processing. can be done. Image management system 4300 may include, for example, a picture archival communication system (PACS).

The acquisition unit 2801 according to this embodiment can acquire an input image from the image management system 4300 connected to the image processing apparatus 2800 . Also, the output unit 2807 can output the region label image generated by the segmentation processing unit 2804 to the image management system 4300 . Note that the output unit 2807 can also display the area label image on the display unit 2820 as in the eighth embodiment.

Next, a series of image processing according to this embodiment will be described with reference to FIG. It should be noted that the processing of steps S2920 to S2960 according to this embodiment is the same as the processing in the eighth embodiment, so description thereof will be omitted. Note that if image segmentation processing is to be performed on the input image unconditionally with respect to the shooting conditions, the processing of step S2930 may be omitted after the processing of step S2920, and the processing may proceed to step S2940.

In step S2910, the acquisition unit 2801 acquires an image stored in the image management system 4300 as an input image from the image management system 4300 connected via a circuit or network. Note that the acquisition unit 2801 may acquire an input image in response to a request from the image management system 4300 . Such a request may be issued, for example, when image management system 4300 saves an image, displays the saved image on display 2820 before sending the saved image to another device. The request is issued, for example, when the user operates the image management system 4300 to request image segmentation processing, or when the region label image is used in the image analysis function of the image management system 4300. may be

The processing from step S2920 to step S2960 is the same as the processing in the eighth embodiment. In step S2970, the output unit 2807 outputs the area label image to the image management system 4300 as an output image if the evaluation unit 2805 determines in step S2950 to output the area label image. Note that the output unit 2807 may process the output image so that it can be used by the image management system 4300 or convert the data format of the output image, depending on the settings and implementation of the image processing apparatus 2800 . The output unit 2807 can also output the analysis result by the analysis unit 2806 to the image management system 4300 .

On the other hand, if the evaluation unit 2805 determines in step S2950 that the image segmentation processing could not be performed properly, the image without region label is output to the image management system 4300 as an output image. In step S 2930 , the output unit 2807 also outputs the region-unlabeled image to the image management system 4300 when the process availability determination unit 2803 determines that the image segmentation process cannot be performed on the input image.

As described above, the acquisition unit 2801 according to this embodiment acquires an input image from the image management system 4300 . For this reason, the image processing apparatus 2800 of this embodiment creates region labeled images suitable for image diagnosis based on the images stored in the image management system 4300 to increase the invasiveness of the operator and the subject. It can be output without increasing effort. Also, the output area label image and image analysis result can be saved in the image management system 4300 or displayed on a user interface provided in the image management system 4300 . Also, the output region label image can be used for the image analysis function provided in the image management system 4300, or can be transmitted to another device connected to the image management system 4300 via the image management system 4300. .

Note that the image processing device 2800, the image management system 4300, and the display unit 2820 may be connected to other devices (not shown) via circuits or networks. Moreover, although these devices are separate devices in this embodiment, some or all of these devices may be integrated.

(Example 19)
Next, an image processing apparatus according to the nineteenth embodiment will be described with reference to FIGS. 44 and 45. FIG. In this embodiment, the correction unit uses the region label image correction engine to correct incorrect region label values set in the region label image output from the image segmentation engine.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the eighth embodiment.

FIG. 44 shows a schematic configuration of an image processing device 4400 according to this embodiment. The image processing apparatus 4400 according to the present embodiment includes an acquisition unit 2801, an imaging condition acquisition unit 2802, a processing propriety determination unit 2803, a segmentation processing unit 2804, an evaluation unit 2805, an analysis unit 2806, an output unit 2807, and a correction unit. 4408 is provided. Note that the image processing device 4400 may be composed of a plurality of devices provided with some of these components. Here, since the configuration of the image processing apparatus 4400 according to the present embodiment other than the correction unit 4408 is the same as the configuration of the image processing apparatus according to the eighth embodiment, the same reference numerals are used for the configuration shown in FIG. , and the explanation is omitted.

Also, the image processing apparatus 4400 may be connected to the photographing apparatus 2810, the display unit 2820, and other devices (not shown) via arbitrary circuits or networks, similarly to the image processing apparatus 2800 according to the eighth embodiment. Also, these devices may be connected to any other device via a circuit or network, or may be integrally configured with any other device. Although these devices are separate devices in this embodiment, some or all of these devices may be integrated.

The modification unit 4408 according to this embodiment includes an area label image modification engine that modifies the input area label image. The region label image modification engine performs region label value modification by anatomical knowledge-based processing as described above in the explanation of terms. Note that, for example, with regard to the area label value overwriting the continuous area of the area label values to be corrected, the area label value with the largest number of pixels in contact with the continuous area shall be overwritten.

Next, a series of image processing according to this embodiment will be described with reference to FIG. It should be noted that the processing of steps S4510 to S4540 according to this embodiment is the same as the processing of steps S2910 to S2940 in the eighth embodiment, respectively, so description thereof will be omitted. Note that if the image segmentation process is to be performed on the input image unconditionally with respect to the imaging conditions, the process of step S4530 may be omitted after the process of step S4520, and the process may proceed to step S4540.

In step S4540, after the segmentation processing unit 2804 generates the region label image, the process proceeds to step S4550. In step S4550, the evaluation unit 2805 evaluates the generated area label image using the area label image evaluation engine, as in the eighth embodiment. If the evaluation result is a true value, the evaluation unit 2805 determines that the area label image is an area label image to be output. On the other hand, the evaluation unit 2805 according to this embodiment determines that the area label image is an area label image that requires correction when the evaluation result is a false value.

In step S4560, the correction unit 4408 uses the region label correction engine to correct the region label values of the region label image determined in step S4540 as a region label image requiring correction. Specifically, modification unit 4408 inputs the region label image determined to require modification in step S4540 to the region label image modification engine. The region label image correction engine corrects erroneously set region label values in the input region label image according to anatomical knowledge base processing, and outputs a corrected region label image.

If it is determined in step S4550 that the generated area label image is the area label image to be output, modifying unit 4408 advances the process without modifying the area label image.

In step S4570, the analysis unit 2806 uses the region label image determined to be the region label image to be output in step S4540 or the region label image corrected in step S4550 to generate the input image. image analysis processing. Since the content of the image analysis processing may be the same as that of the eighth embodiment, the description is omitted.

In step S4580, the output unit 2807 causes the display unit 2820 to display the area label image determined as the area label image to be output or the area label image with the corrected area label and the image analysis result. Note that the output unit 2807 may cause the imaging device 2810 or another device to display or store the region label image and the image analysis result instead of displaying the region label image and the image analysis result on the display unit 2820 . The output unit 2807 processes these data so that they can be used by the imaging device 2810 or other devices, or converts them into data formats that can be transmitted to an image management system or the like, depending on the settings and implementation of the image processing device 4400 . may be converted. Also, the output unit 2807 is not limited to outputting both the region label image and the image analysis result, and may output only one of them.

On the other hand, if it is determined in step S4530 that the image segmentation processing is impossible, the output unit 2807 outputs an image without region label and causes the display unit 2820 to display it. It should be noted that instead of outputting the region-unlabeled image, a signal indicating that the image segmentation process was not possible may be sent to the imaging device 2810 . When the output processing in step S4580 ends, the series of image processing ends.

As described above, the image processing apparatus 4400 according to this embodiment further includes the correction unit 4408 . The correction unit 4408 corrects the region label image generated by the segmentation processing unit 2804 using a region label image correction engine that performs knowledge base processing according to a predetermined correction method. The output unit 2807 outputs the region label image corrected by the correction unit 4408. FIG.

In particular, the correction unit 4408 according to this embodiment corrects the region label for the region label image determined by the evaluation unit 2805 that the image segmentation processing could not be performed appropriately. Also, the analysis unit 2806 performs image analysis processing on the region label image in which the region label has been corrected.

As a result, in the image processing apparatus 2800 according to this embodiment, the region label image correction engine can correct errors in the region label image for which image segmentation processing has failed, and output the corrected region label image.

Note that in this embodiment, the correction unit 4408 corrects the region label for the region label image for which the evaluation result by the evaluation unit 2805 is a false value. However, the configuration of correction unit 4408 is not limited to this. The correction unit 4408 may correct the region label for the region label image for which the evaluation result by the evaluation unit 2805 is a true value. In this case, the analysis unit 2806 performs image analysis processing on the input image using the modified region label. Also, the output unit 2807 outputs the corrected area label image and its analysis result.

Furthermore, in this case, the evaluation unit 2805 prevents the correcting unit 4408 from correcting the area label of an area label image for which the evaluation result is a false value. Images can also be generated. Once the evaluation unit 2805 has generated the region-unlabeled image, the modification unit 4408 can proceed without modification.

If the correction unit 4408 corrects the region label image when the evaluation result by the evaluation unit 2805 is a true value, the image processing apparatus 2800 can output the region label image and the analysis result with higher accuracy.

In the eighth to nineteenth embodiments described above, the configuration in which the segmentation processing unit 2804 generates region label images as region information capable of identifying an anatomical region has been described. Not limited. The region information generated from the input image by the segmentation processing unit using the image segmentation engine may be a group of numerical data such as coordinate values of pixels having each region label.

Note that the trained models included in the image segmentation engine, region label image evaluation engine, and imaging location estimation engine can be provided in the image processing devices 2800 and 4400, respectively. A trained model may be configured by, for example, a software module or the like executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured by a circuit or the like that performs a specific function such as an ASIC. Also, the learned model may be provided in another server device or the like connected to the image processing devices 2800 and 4400 . In this case, the image processing apparatuses 2800 and 4400 can use the trained model by connecting to a server or the like having the trained model via an arbitrary network such as the Internet. Here, the server provided with the learned model may be, for example, a cloud server, a fog server, an edge server, or the like.

According to the eighth to nineteenth embodiments, image segmentation processing with higher accuracy than conventional image segmentation processing can be performed.

(Modification 1)
In Examples 1 to 7 and their modifications, the processing unit 222 or the first processing unit 822 used the trained model to detect the retinal layer from the tomographic image and generate the boundary image. Also, in Examples 9 to 19, the segmentation processing unit 2804 used an image segmentation engine including a trained model to generate a region label image corresponding to the input image.

On the other hand, the retinal layer information detected using the trained model and the generated boundary image or region label image may be manually corrected according to instructions from the operator. For example, the operator can specify at least a part of the retinal layer detection result, the boundary image, or the region label image displayed on the display unit 50, 2820 to change the position and label of the retinal layer. In this case, correction of the detection result and correction of the boundary image or the region label image may be performed by the processing unit 222, the first processing unit 822, and the segmentation processing unit 2804 according to the operator's instruction. It may be performed by a component such as a correcting unit that is separate from. Therefore, the processing unit 222, the first processing unit 822, the segmentation processing unit 2804, or the correction unit is a correction unit that corrects the structure of the retinal layers detected by the first processing unit according to the operator's instruction. serves as an example of Note that the correction unit and the like may be composed of a software module or the like executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be composed of a circuit or the like that performs a specific function such as an ASIC.

(Modification 2)
The manually corrected data in Modification 1 is used for additional learning of the trained model used by the processing unit 222 or the first processing unit 822 and the trained model included in the image segmentation engine used by the segmentation processing unit 2804. may be Specifically, for the trained model used by the processing unit 222 or the first processing unit 822, the input tomographic image is used as input data for learning data, and the retinal layer whose position is corrected according to the instruction from the operator. Additional learning is performed using the (layer boundary) information as output data (correct data). The output data may be a boundary image whose label has been corrected according to an instruction from the operator. In addition, for the trained model included in the image segmentation engine, the input image is used as the input data for the training data, and the region label image with the label position changed according to the instruction from the operator is used as the output data for additional learning. conduct.

By performing such additional learning on the trained model, it is expected that the accuracy of detection processing and segmentation processing using the trained model can be improved. Moreover, by performing such processing, it is possible to easily perform the labeling processing (annotation processing) on the learning data, and to easily create more accurate learning data.

(Modification 3)
The additional learning described in Modification 2 may be performed according to an operator's instruction. For example, the display control unit 25 or the output unit 2807 uses the corrected detection result of the retinal layer, the region label image, etc. as the learning data when the correction is performed according to the operator's instruction according to the first modification. It is possible to display on the display unit 50, 2820 whether or not there is. By selecting options displayed on the display unit 50, 2820, the operator can instruct whether or not additional learning is required. Accordingly, the image processing apparatuses 20, 80, 2800, and 4400 can determine whether or not additional learning is necessary according to the operator's instruction.

Note that, as described above, the trained model can also be provided in a device such as a server. In this case, the image processing apparatuses 20, 80, 2800, and 4400 combine the input image with the above-described modified detection result or region label image, etc., in accordance with the operator's instruction to perform additional learning. can be transmitted and stored in the server or the like as a pair of learning data. In other words, the image processing apparatuses 20, 80, 2800, and 4400 can determine whether or not to transmit learning data for additional learning to a device such as a server having a trained model in accordance with an instruction from the operator. .

(Modification 4)
In the various embodiments and modified examples described above, the configuration for performing the process of detecting the retinal layer and the process of generating the region label image and the like for the still image has been described. On the other hand, the process of detecting the retinal layer and the process of generating the region label image and the like according to the above-described embodiment and modifications may be repeatedly performed on the moving image. Generally, ophthalmologic apparatuses generate and display a preview image (moving image) for purposes such as positioning of the apparatus before actual photographing. Therefore, for example, the process of detecting the retinal layer and the process of generating the region label image and the like according to the above-described embodiments and modifications may be repeatedly executed for each at least one frame of the moving image of the tomographic image, which is the preview image. .

In this case, the display control unit 25 or the output unit 2807 can cause the display units 50 and 2820 to display the retinal layers, region label images, and the like detected in the preview image. In addition, the image processing apparatuses 20, 80, 2800, and 4400 perform OCT so that the retinal layers detected in the preview image and the regions labeled with the retinal layers are positioned at predetermined positions in the display region of the tomographic image. You can control the device. More specifically, the image processing apparatuses 20, 80, 2800, and 4400 position the retinal layers detected in the preview image and the labeled regions of the retinal layers at predetermined positions in the display region of the tomographic image. Change the coherence gate position as follows. Note that the coherence gate position may be adjusted by, for example, driving the coherence gate stage 14 by the drive control unit 23 .

Note that the adjustment of the coherence gate position may be manually performed according to an instruction from the operator. In this case, the operator sets the adjustment amount of the coherence gate position to the image processing devices 20, 80, 2800, and 4400 based on the retinal layers and region label images detected for the preview images displayed on the display units 50 and 2820. can be entered in

According to such processing, it is possible to appropriately align the OCT apparatus with respect to the subject's eye based on the retinal layers detected using the trained model or the generated region label image.

Note that moving images to which the retinal layer detection processing and region label image generation processing according to the above-described embodiments and modifications can be applied are not limited to live moving images. It can be an image. In addition, during various adjustments such as the coherence gate position, there is a possibility that the object to be imaged, such as the retina of the subject's eye, has not yet been successfully imaged. Therefore, the difference between the medical image input to the trained model and the medical image used as learning data is large, so there is a possibility that the detection result of the retinal layer, the region label image, etc. cannot be obtained with high accuracy. Therefore, when an evaluation value such as image quality evaluation of a tomographic image (B-scan) exceeds a threshold value, the retinal layer detection processing using the trained model and the generation processing of the region label image, etc. are automatically started. may In addition, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold, the button for instructing the image segmentation processing using the trained model is changed to the state where the examiner can specify (active state). may be configured to

(Modification 5)
Diseased eyes have different image features depending on the type of disease. Therefore, the learned models used in the various embodiments and modifications described above may be generated and prepared for each type of disease or for each abnormal site. In this case, for example, the image processing apparatuses 20, 80, 2800, and 4400 select a trained model to be used for processing in accordance with an operator's input (instruction) of the type of disease, abnormal site, etc. of the eye to be examined. can do. Note that the trained models prepared for each disease type and abnormal site are not limited to the trained models used for detecting retinal layers and generating region labeled images. It may be a trained model used in an engine or the like.

Further, the image processing apparatuses 20, 80, 2800, and 4400 may use a separately prepared trained model to identify the type of disease and abnormal site of the subject's eye from the image. In this case, the image processing apparatuses 20, 80, 2800, and 4400 automatically select the learned model used for the above process based on the type of disease and abnormal site identified using the separately prepared learned model. can be selectively selected. The trained model for identifying the type of disease and abnormal site of the eye to be examined uses tomographic images, fundus images, etc. as input data, and learns the type of disease and abnormal sites in these images as output data. Learning may be done in pairs. Here, as input data for learning data, a tomographic image, a fundus image, or the like may be used alone as input data, or a combination thereof may be used as input data.

In addition, when detecting an abnormal site, an adversarial generation network (GAN: Generative Adversarial Networks) or a variational auto-encoder (VAE: Variational auto-encoder) may be used. For example, DCGAN (Deep Convolutional GAN) consisting of a generator obtained by learning to generate a tomographic image and a discriminator obtained by learning to discriminate between a new tomographic image generated by the generator and a real frontal fundus image. ) can be used as a machine learning model.

When DCGAN is used, for example, a classifier encodes an input tomographic image into a latent variable, and a generator generates a new tomographic image based on the latent variable. After that, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal site. When VAE is used, for example, an input tomographic image is encoded by an encoder to generate a latent variable, and a decoder decodes the latent variable to generate a new tomographic image. After that, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal site. Although a tomographic image has been described as an example of input data, a fundus image, a front image of the anterior eye, or the like may be used.

(Modification 6)
In the various embodiments and modifications described above, when detecting a region of the eye to be examined using a trained model by the processing unit 222, the first processing unit 822, and the segmentation processing unit 2804, each detected region can be subjected to predetermined image processing. For example, consider the case of detecting at least two regions of the vitreous region, the retinal region, and the choroidal region. In this case, when performing image processing such as contrast adjustment on at least two detected regions, by using different image processing parameters, it is possible to perform adjustment suitable for each region. By displaying an image adjusted appropriately for each region, the operator can more appropriately diagnose a disease or the like for each region. Regarding the configuration using different image processing parameters for each detected region, for example, the region of the eye to be examined detected by the second processing unit 823 that detects the region of the eye to be examined without using the learned model may be applied as well.

(Modification 7)
The display control unit 25 or the output unit 2807 in the various embodiments and modifications described above may display analysis results such as desired layer thicknesses and various blood vessel densities on the report screen of the display screen. In addition, optic papilla, macular region, vascular region, nerve fiber bundle, vitreous region, macular region, choroidal region, scleral region, cribriform plate region, retinal layer boundary, retinal layer boundary edge, photoreceptors, blood cells, A value (distribution) of parameters relating to a region of interest including at least one of a blood vessel wall, an inner wall boundary of a blood vessel, an outer boundary of a blood vessel, a ganglion cell, a corneal area, an angle area, Schlemm's canal, etc., may be displayed as an analysis result. At this time, for example, by analyzing a medical image to which various artifact reduction processes have been applied, it is possible to display an accurate analysis result. Artifacts include, for example, false image areas caused by light absorption by blood vessel areas, projection artifacts, and strip-shaped artifacts in the front image that occur in the main scanning direction of the measurement light due to the state of the subject's eye (movement, blinking, etc.). There may be. Also, the artifact may be anything, for example, as long as it is an image failure area that occurs randomly in each imaging on a medical image of a predetermined region of the subject. In addition, the display control unit 25 or the output unit 2807 causes the display unit 50 or 2820 to display the parameter values (distribution) regarding the area including at least one of the above-described various artifacts (imaging area) as the analysis result. may Also, parameter values (distribution) relating to an area including at least one of drusen, new blood vessels, vitiligo (hard vitiligo), and abnormal sites such as pseudodrusen may be displayed as the analysis result.

Further, the analysis result may be displayed as an analysis map, a sector indicating statistical values corresponding to each divided area, or the like. The analysis results may be generated using a trained model (analysis result generation engine, trained model for generating analysis results) obtained by learning analysis results of medical images as learning data. . At this time, the trained model is learned using learning data including medical images and analysis results of the medical images, or learning data including medical images and analysis results of medical images of a different type from the medical images. It may be obtained by

Further, the learning data includes the detection result of the retinal layer by the processing unit 222, or the first processing unit 822 and/or the second processing unit 823, the region label image generated by the segmentation processing unit 2804, and the It may also include the analysis results of the medical image obtained. In this case, the image processing apparatus uses, for example, a trained model for generating analysis results to generate the analysis results of the tomographic image from the results obtained by executing the first detection process. It can serve as an example.

Furthermore, the trained model is obtained by learning using learning data including input data that is a set of a plurality of different types of medical images of predetermined regions, such as a luminance front image and a motion contrast front image. good too. Here, the luminance front image corresponds to the luminance En-Face image, and the motion contrast front image corresponds to the OCTA En-Face image.

Further, it may be configured to display analysis results obtained using a high-quality image generated using a trained model for image quality improvement. In this case, the input data included in the learning data may be a high-quality image generated using a trained model for improving image quality, or a set of a low-quality image and a high-quality image. good too. Note that the learning data may be an image obtained by manually or automatically correcting at least a part of an image whose image quality has been improved using a trained model.

In addition, the learning data includes, for example, an analysis value obtained by analyzing the analysis area (e.g., average value, median value, etc.), a table containing the analysis value, an analysis map, the position of the analysis area such as a sector in the image, etc. The input data may be labeled with information containing one as (supervised learning) correct data. Note that the analysis results obtained using the learned model for generating analysis results may be displayed according to instructions from the operator.

Further, the display control unit 25 and the output unit 2807 in the above-described embodiments and modifications may display various diagnostic results such as glaucoma and age-related macular degeneration on the report screen of the display screen. At this time, for example, by analyzing a medical image to which various artifact reduction processes as described above have been applied, it is possible to display a highly accurate diagnosis result. Further, the diagnosis result may display the position of the identified abnormal site or the like on an image, or may display the state of the abnormal site or the like in characters or the like. Further, classification results (for example, Curtin classification) such as abnormal sites may be displayed as diagnosis results.

The diagnosis result may be generated using a trained model (a diagnosis result generating engine, a trained model for generating diagnosis results) obtained by learning the diagnosis results of medical images as learning data. . In addition, the trained model is obtained by learning using learning data including medical images and diagnostic results of the medical images, learning data including medical images and diagnostic results of medical images of a different type from the medical images, and the like. It may be obtained.

Further, the learning data includes the detection result of the retinal layer by the processing unit 222, or the first processing unit 822 and/or the second processing unit 823, the region label image generated by the segmentation processing unit 2804, and the It may also include the diagnostic results of the medical images obtained. In this case, the image processing apparatus uses, for example, a trained model for generating diagnostic results to generate the diagnostic results of the tomographic image from the results obtained by executing the first detection process. It can serve as an example.

Furthermore, it may be configured to display the diagnosis result obtained using the high-quality image generated using the trained model for image quality improvement. In this case, the input data included in the learning data may be a high-quality image generated using a trained model for improving image quality, or a set of a low-quality image and a high-quality image. good too. Note that the learning data may be an image obtained by manually or automatically correcting at least a part of an image whose image quality has been improved using a trained model.

In addition, the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, findings (interpretation findings, etc.), the basis for the diagnosis name (positive information that contains at least one of the following: correct medical support information, etc.), grounds for denying a diagnosis (negative medical support information), etc., as correct data (supervised learning). may Note that the diagnostic results obtained using the learned model for generating diagnostic results may be displayed according to instructions from the examiner.

Further, the display control unit 25 and the output unit 2807 according to the various embodiments and modifications described above display the object recognition result (object detection results) or segmentation results may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, a color or the like may be superimposed on the object in the image and displayed. The object recognition result and segmentation result may be generated using a trained model obtained by learning learning data in which medical images are labeled with information indicating object recognition and segmentation as correct data. . Note that the analysis result generation and diagnosis result generation described above may be obtained by using the object recognition result and segmentation result described above. For example, analysis result generation and diagnosis result generation processing may be performed on a region of interest obtained by object recognition or segmentation processing.

In addition, the trained model for generating diagnostic results in particular may be a trained model obtained by learning using learning data including input data that is a set of a plurality of different types of medical images of a predetermined region of a subject. good. At this time, the input data included in the learning data may be, for example, input data that is a set of a motion contrast front image and a luminance front image (or a luminance tomographic image) of the fundus. As input data included in the learning data, for example, input data such as a set of a fundus tomographic image (B-scan image) and a color fundus image (or a fluorescent fundus image) can be considered. Moreover, the multiple medical images of different types may be acquired by different modalities, different optical systems, or different principles.

In addition, the trained model for diagnosis result generation in particular may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of different parts of the subject. At this time, as the input data included in the learning data, for example, input data that is a set of a tomographic image (B-scan image) of the fundus and a tomographic image (B-scan image) of the anterior segment can be considered. In addition, as input data included in the learning data, for example, input data such as a set of a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic papilla of the fundus. is also conceivable.

The input data included in the learning data may be a plurality of medical images of different regions and different types of the subject. At this time, the input data included in the learning data may be, for example, input data that is a set of a tomographic image of the anterior segment and a color fundus image. Further, the above-described trained model may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of a predetermined part of the subject with different imaging angles of view. The input data included in the learning data may be obtained by pasting together a plurality of medical images obtained by time-dividing a predetermined site into a plurality of regions, such as a panorama image. At this time, by using a wide-angle image such as a panoramic image as training data, it is possible to acquire the feature amount of the image with high accuracy because the amount of information is larger than that of a narrow-angle image. results can be improved. Also, the input data included in the learning data may be input data that is a set of a plurality of medical images of a predetermined part of the subject taken on different dates.

Further, the display screen on which at least one of the above-described analysis result, diagnosis result, object recognition result, and segmentation result is displayed is not limited to the report screen. Such a display screen is, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). may be displayed in For example, by displaying at least one result obtained using the above-described learned model on the photographing confirmation screen, the operator can confirm a highly accurate result even immediately after photographing. Further, the display change between the low image quality image and the high image quality image described in the seventh embodiment may be, for example, the display change between the analysis result of the low image quality image and the analysis result of the high image quality image.

Here, the various trained models described above can be obtained by machine learning using learning data. Machine learning includes, for example, deep learning consisting of multilevel neural networks. In addition, for example, a convolutional neural network (CNN: Convolutional Neural Network) can be used for at least part of the multi-layered neural network. Also, at least a part of the multi-layered neural network may use a technique related to an autoencoder. Also, a technique related to back propagation (error backpropagation method) may be used for learning. However, machine learning is not limited to deep learning, and any learning using a model capable of extracting (expressing) feature amounts of learning data such as images by learning can be used. The machine learning model may be, for example, a Capsule Network (CapsNet). Here, in a general neural network, each unit (each neuron) is configured to output a scalar value, so that spatial information regarding the spatial positional relationship (relative position) between features in an image, for example, can be obtained. configured to be reduced. As a result, for example, learning can be performed in which the effects of local distortion, translation, and the like of an image are reduced. On the other hand, in a capsule network, each unit (each capsule) is configured to output spatial information as a vector, thereby retaining spatial information. As a result, for example, learning can be performed in consideration of the spatial positional relationship between features in the image.

In addition, the high image quality engine (learned model for high image quality) is a trained model obtained by additionally learning learning data containing at least one high quality image generated by the high image quality engine. good. At this time, whether or not to use the high-quality image as learning data for additional learning may be selectable by an instruction from the examiner.

(Modification 8)
The preview screens in the various embodiments and modifications described above may be configured to use the above-described learned model for improving image quality for each of at least one frame of the live moving image. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the learned model corresponding to each live moving image may be used. As a result, the processing time can be shortened even for a live moving image, for example, so that the examiner can obtain highly accurate information before the start of imaging. For this reason, for example, failures in re-imaging can be reduced, so that accuracy and efficiency of diagnosis can be improved. Note that the plurality of live moving images may be, for example, a moving image of the anterior segment for alignment in the XYZ directions, and a front moving image of the fundus for focus adjustment of the fundus oculi observation optical system and OCT focus adjustment. Also, the plurality of live moving images may be, for example, tomographic moving images of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length).

Further, the moving image to which the above-described trained model can be applied is not limited to the live moving image, and may be, for example, a moving image stored (saved) in the storage unit. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving images of the fundus stored (saved) in the storage unit may be displayed on the display screen. For example, when the vitreous body is desired to be properly observed, first, a reference frame may be selected based on conditions such as the presence of as much vitreous body as possible on the frame. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, high-quality images (high-quality frames) sequentially generated using a learned model for image quality improvement may be continuously displayed for each at least one frame of a moving image.

As a method for aligning between frames described above, the same method may be applied to the method of aligning in the X direction and the method of aligning in the Z direction (depth direction), or different methods may be used. may be applied. Also, the alignment in the same direction may be performed multiple times by different techniques. For example, fine alignment may be performed after performing rough alignment. Further, as a method of alignment, for example, there is a (coarse Z-direction) alignment using a retinal layer boundary obtained by segmentation processing of a tomographic image (B-scan image). Furthermore, as a method of alignment, for example, there is also (precise alignment in the X and Z directions) using correlation information (similarity) between a plurality of regions obtained by dividing a tomographic image and a reference image. . Furthermore, as a method of alignment, for example, alignment (in the X direction) using a one-dimensional projection image generated for each tomographic image (B scan image), alignment (in the X direction) using a two-dimensional front image, Alignment, etc. Also, it may be configured such that after rough alignment is performed in units of pixels, fine alignment is performed in units of sub-pixels.

Here, during various adjustments, there is a possibility that the object to be imaged, such as the retina of the subject's eye, has not yet been successfully imaged. For this reason, there is a possibility that a high-quality image cannot be obtained with high accuracy due to the large difference between the medical image input to the trained model and the medical image used as learning data. Therefore, when an evaluation value such as image quality evaluation of a tomographic image (B scan) exceeds a threshold value, display of high-quality moving images (continuous display of high-quality frames) may be automatically started. Further, when an evaluation value such as image quality evaluation of a tomographic image (B-scan) exceeds a threshold value, the image quality improvement button may be changed to a state (active state) in which the examiner can designate.

In addition, a different trained model for improving image quality is prepared for each imaging mode having a different scanning pattern, etc., and the trained model for improving image quality corresponding to the selected imaging mode is selected. good too. Alternatively, one trained model for improving image quality obtained by learning learning data including various medical images obtained in different imaging modes may be used.

(Modification 9)
In the various embodiments and modifications described above, when various trained models are undergoing additional learning, it may be difficult to output (infer/predict) using the learned models themselves that are undergoing additional learning. Therefore, it is preferable to prohibit the input of medical images to the trained model during additional learning. Also, another trained model that is the same as the trained model being additionally learned may be prepared as a backup trained model. At this time, during the additional learning, it is preferable to allow input of medical images to the preliminary trained model. After the additional learning is completed, the trained model after the additional learning is evaluated, and if there is no problem, the spare trained model can be replaced with the trained model after the additional learning. Also, if there is a problem, a backup trained model may be used.

Also, a learned model obtained by learning for each imaging region may be selectively used. Specifically, a first trained model obtained using learning data including a first imaging region (lung, eye to be examined, etc.) and a learning including a second imaging region different from the first imaging region A plurality of trained models can be provided, including a second trained model obtained using the data. The control unit 200 may have selection means for selecting one of these learned models. At this time, the control unit 200 may have control means for performing additional learning on the selected trained model. In response to an instruction from the examiner, the control means searches for data paired with an imaging region corresponding to the selected learned model and a photographed image of the imaging region, and uses the retrieved data as learning data. can be performed as additional learning on the selected trained model. Note that the imaged region corresponding to the selected learned model may be obtained from information in the header of the data or manually input by the examiner. Also, data retrieval may be performed via a network from, for example, a server of an external facility such as a hospital or research institute. As a result, additional learning can be efficiently performed for each imaging part using the photographed image of the imaging part corresponding to the learned model.

The selection means and the control means may be configured by software modules executed by a processor such as the CPU or MPU of the control section 200 . Also, the selection means and the control means may be configured by a circuit such as an ASIC that performs a specific function, an independent device, or the like.

In addition, when acquiring learning data for additional learning from a server of an external facility such as a hospital or research institute via a network, it is necessary to reduce reliability deterioration due to falsification and system troubles during additional learning. is useful. Therefore, the correctness of the learning data for additional learning may be detected by confirming the matching by digital signature or hashing. Thereby, learning data for additional learning can be protected. At this time, if the validity of the learning data for additional learning cannot be detected as a result of confirming the match by digital signature or hashing, a warning to that effect is issued and additional learning is performed using the learning data. Make it not exist. It should be noted that the server may take any form such as a cloud server, a fog server, an edge server, etc., regardless of its installation location.

(Modification 10)
In the various embodiments and modifications described above, the instruction from the examiner may be an instruction by voice or the like in addition to a manual instruction (for example, an instruction using a user interface or the like). At this time, for example, a machine learning model including a speech recognition model (speech recognition engine, trained model for speech recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by character input using a keyboard, touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (a character recognition engine, a learned model for character recognition) obtained by machine learning may be used. Also, the instruction from the examiner may be an instruction by a gesture or the like. At this time, a machine learning model including a gesture recognition model (a gesture recognition engine, a trained model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be the sight line detection result of the examiner on the display unit 50, 2820, or the like. The line-of-sight detection result may be, for example, a pupil detection result using a moving image of the examiner captured from around the display unit 50,2820. At this time, the object recognition engine as described above may be used for pupil detection from moving images. Further, the instructions from the examiner may be instructions based on brain waves, weak electrical signals flowing through the body, or the like.

In such a case, for example, as the learning data, character data or voice data (waveform data) indicating instructions for displaying the results of the processing of the various learned models as described above may be used as input data. The learning data may be learning data in which the correct answer data is an execution command for actually displaying the result of model processing on the display unit. In addition, as learning data, for example, text data or voice data indicating an instruction to display a high-quality image obtained by a trained model for high-quality image is used as input data. 22( a ) and 22 ( b ) may be learning data having an execution instruction for changing the button 2220 to the active state as correct answer data. Any learning data may be used as long as the contents of instructions indicated by character data or voice data and the contents of execution commands correspond to each other. Alternatively, speech data may be converted into character data using an acoustic model, a language model, or the like. Also, waveform data obtained by a plurality of microphones may be used to perform processing for reducing noise data superimposed on audio data. Further, it may be configured such that an instruction by text, voice, or the like and an instruction by a mouse, a touch panel, or the like can be selected according to an instruction from the examiner. Moreover, it may be configured such that ON/OFF of instructions by text, voice, or the like can be selected according to instructions from the examiner.

Machine learning includes deep learning as described above, and a recurrent neural network (RNN), for example, can be used for at least a part of the multi-level neural network. Here, as an example of the machine learning model according to this modified example, an RNN, which is a neural network that handles time-series information, will be described with reference to FIGS. Also, a long short-term memory (hereinafter referred to as LSTM), which is a type of RNN, will be described with reference to FIGS.

FIG. 46(a) shows the structure of RNN, which is a machine learning model. RNN 4620 has a loop structure in the network, receives data x ^t 4610 at time t, and outputs data h ^t 4630 . Since the RNN 4620 has a loop function in the network, it is possible to take over the state of the current time to the next state, so it can handle time-series information. FIG. 46(b) shows an example of input and output of parameter vectors at time t. Data x ^t 4610 includes N (Params1 to ParamsN) data. Data h ^t 4630 output from RNN 4620 includes N pieces of data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information during error backpropagation, LSTM is sometimes used. The LSTM can learn long-term information by having a forget gate, an input gate, and an output gate. Here, FIG. 47(a) shows the structure of the LSTM. In LSTM4740, the information that the network takes over at the next time t is the internal state c ^t-1 of the network called a cell and the output data h ^t-1 . Note that the lower case letters (c, h, x) in the figure represent vectors.

Next, the details of the LSTM 4740 are shown in FIG. 47(b). In FIG. 47(b), a forget gate network FG, an input gate network IG and an output gate network OG are shown, each being a sigmoid layer. Therefore, a vector in which each element is a value between 0 and 1 is output. The forget gate network FG determines how much past information is retained, and the input gate network IG determines which values are updated. Also, in FIG. 47(b), a cell update candidate network CU is shown, and the cell update candidate network CU is the activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gating network OG selects elements of the cell candidates and how much information to convey the next time.

Note that the above-described LSTM model is a basic model, so it is not limited to the network shown here. Coupling between networks may be changed. QRNN (Quasi Recurrent Neural Network) may be used instead of LSTM. Furthermore, machine learning models are not limited to neural networks, and boosting, support vector machines, and the like may be used. In addition, when the instruction from the examiner is input by text, voice, or the like, a technique related to natural language processing (for example, sequence to sequence) may be applied. Also, a dialogue engine (dialogue model, learned model for dialogue) that responds to the examiner with text or voice output may be applied.

(Modification 11)
In the various embodiments and modifications described above, boundary images, region label images, high-quality images, and the like may be stored in the storage unit according to instructions from the operator. At this time, for example, after an instruction from the operator to save a high-quality image, when registering a file name, any part of the file name (for example, the first part, or last part), the file name containing information (for example, characters) indicating that it is an image generated by processing using a trained model for high image quality (high image quality processing) is entered by the operator. It may be displayed in an editable state according to instructions. Similarly, for the boundary image, the area label image, and the like, a file name including information that the image is generated by processing using the trained model may be displayed.

In various display screens such as a report screen, when a high-quality image is displayed on the display unit 50, 2820, the displayed image is a high-quality image generated by processing using a trained model for image quality improvement. An indication indicating that the image is a high quality image may be displayed along with the high quality image. In this case, the operator can easily identify from the display that the displayed high-quality image is not the image itself acquired by photography, thereby reducing misdiagnosis and improving diagnostic efficiency. be able to. It should be noted that the display indicating that the image is a high-quality image generated by processing using a trained model for high image quality should be a display that can distinguish between the input image and the high-quality image generated by the processing. It may be of any aspect. In addition to processing using a trained model for high image quality, processing using various trained models such as those described above is also a result generated by processing using that type of trained model. An indication may be displayed with the results. Also, when displaying the analysis result of the segmentation result using the trained model for image segmentation processing, the display indicating that the analysis result is based on the result of using the trained model for image segmentation May be displayed with results.

At this time, the display screen such as the report screen may be stored in the storage section according to an instruction from the operator. For example, the report screen may be stored in the storage unit as a single image in which high-quality images, etc., and an indication indicating that these images are images generated by processing using the learned model are arranged side by side.

In addition, regarding the display indicating that the image is a high-quality image generated by processing using the trained model for high image quality, what kind of training data the trained model for high image quality learned. A display indicating whether there is any may be displayed on the display unit. The display may include an explanation of the types of input data and correct data of the learning data, and any display related to correct data such as imaging regions included in the input data and correct data. For example, for processing using the various trained models described above, such as image segmentation processing, a display indicating what kind of learning data the trained model of that type has learned is displayed on the display unit. may be

Further, information (for example, characters) indicating that the image is generated by processing using a trained model may be displayed or stored in a state superimposed on the image or the like. At this time, the portion to be superimposed on the image may be any region (for example, the end of the image) that does not overlap the region where the region of interest to be imaged is displayed. Alternatively, a non-overlapping region may be determined and superimposed on the determined region. It should be noted that not only the processing using the trained model for image quality improvement, but also images obtained by processing using the various trained models described above, such as image segmentation processing, may be similarly processed.

22(a) and 22(b) as the initial display screen of the report screen, if the default setting is such that the button 2220 is in an active state (image quality enhancement processing is ON), the A report image corresponding to a report screen including a high-quality image or the like may be transmitted to the server in response to an instruction from the user. Also, if the button 2220 is set to be in an active state by default, at the end of the examination (for example, when the imaging confirmation screen or preview screen is changed to the report screen in response to an instruction from the examiner) Alternatively, the report image corresponding to the report screen including the high-quality image and the like may be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, whether or not the analysis map is superimposed, whether or not it is a high-quality image, the display screen for follow-up observation setting regarding at least one of whether or not the report image generated based on the setting is transmitted to the server. It should be noted that similar processing may be performed when the button 2220 indicates switching of image segmentation processing.

(Modification 12)
In the various embodiments and modifications described above, among the various trained models as described above, an image obtained by the first type of trained model (for example, a high-quality image, an analysis map showing analysis results such as images, images showing object recognition results, images showing retinal layers, images showing segmentation results) may be input to a second type of trained model different from the first type. At this time, the result (for example, analysis result, diagnosis result, object recognition result, retinal layer detection result, segmentation result) of processing of the second type of trained model may be generated.

In addition, among the various trained models described above, the results of the processing of the first type of trained model (for example, analysis results, diagnosis results, object recognition results, retinal layer detection results, segmentation results) are used. Therefore, an image to be input to a first type of trained model different from the first type may be generated from an image input to the first type of trained model. At this time, the generated image is highly likely to be an image suitable for processing using the second type of trained model. For this reason, images obtained by inputting the generated images into the second type of trained model (for example, high-quality images, images showing analysis results such as analysis maps, images showing object recognition results, retinal layers, etc.) images showing segmentation results) can be improved in accuracy.

Further, a similar image search may be performed using an external database stored in a server or the like using the analysis results, diagnosis results, etc. obtained by processing the learned model as described above as a search key. In addition, when the plurality of images stored in the database are already managed in a state in which the feature values of each of the plurality of images are attached as incidental information by machine learning, etc., the image itself is used as the search key. A similar image search engine (similar image search model, trained model for similar image search) may be used.

(Modification 13)
Note that the motion contrast data generation processing in the above embodiments and modifications is not limited to being performed based on the luminance value of the tomographic image. The above-described various processes include an interference signal obtained by the OCT apparatus 10 or the imaging apparatus 2810, a signal obtained by subjecting the interference signal to Fourier transform, a signal obtained by subjecting the signal to arbitrary processing, and a tomographic image based thereon. may be applied to the data. Also in these cases, the same effect as the above configuration can be obtained.

Although a fiber optical system using a coupler is used as the splitting means, a spatial optical system using a collimator and a beam splitter may be used. Also, the configuration of the OCT apparatus 10 or the imaging apparatus 2810 is not limited to the above configuration, and part of the configuration included in the OCT apparatus 10 or the imaging apparatus 2810 may be configured separately from the OCT apparatus 10 or the imaging apparatus 2810. .

In addition, in the above-described embodiments and modifications, the configuration of the Mach-Zehnder interferometer is used as the interference optical system of the OCT apparatus 10 or imaging device 2810, but the configuration of the interference optical system is not limited to this. For example, the interferometric optics of the OCT device 10 or imager 2810 may have the configuration of a Michelson interferometer.

Furthermore, in the above embodiments and modifications, the spectral domain OCT (SD-OCT) apparatus using an SLD as a light source has been described as the OCT apparatus, but the configuration of the OCT apparatus according to the present invention is not limited to this. For example, the present invention can be applied to any other type of OCT apparatus such as a wavelength swept OCT (SS-OCT) apparatus using a wavelength swept light source capable of sweeping the wavelength of emitted light. The present invention can also be applied to a Line-OCT apparatus using line light.

In addition, in the above-described embodiment and modification, the acquisition units 21 and 2801 acquire interference signals acquired by the OCT device 10 or the imaging device 2810, three-dimensional tomographic images generated by the image processing device, and the like. However, the configuration in which the acquisition unit 21, 2801 acquires these signals and images is not limited to this. For example, the acquisition unit 21, 2801 may acquire these signals from a server or imaging device connected to the control unit via LAN, WAN, Internet, or the like.

Note that the learned model can be provided in the image processing apparatuses 20, 80, 152, 172, and 2800. FIG. A trained model can be composed of, for example, a software module or the like executed by a processor such as a CPU. Also, the learned model may be provided in another server or the like connected to the image processing apparatuses 20, 80, 152, 172, and 2800. FIG. In this case, the image processing apparatuses 20, 80, 152, 172, and 2800 connect to a server having a trained model via an arbitrary network such as the Internet, and perform image quality improvement processing using the trained model. It can be carried out.

(Modification 14)
Images processed by the image processing apparatuses or image processing methods according to the various embodiments and modifications described above include medical images acquired using any modality (imaging device, imaging method). Medical images to be processed can include medical images acquired by any imaging device or the like, and images created by the image processing devices or image processing methods according to the above embodiments and modifications.

Furthermore, the medical image to be processed is an image of a predetermined region of a subject (subject), and the image of the predetermined region includes at least part of the predetermined region of the subject. Also, the medical image may include other parts of the subject. Also, the medical image may be a still image or a moving image, and may be a black-and-white image or a color image. Further, the medical image may be an image representing the structure (morphology) of a predetermined site, or an image representing its function. Images representing functions include, for example, images representing blood flow dynamics (blood flow, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. In addition, the predetermined part of the subject may be determined according to the object to be imaged. Includes optional parts such as legs and arms.

Further, the medical image may be a tomographic image or a front image of the subject. The front image is, for example, a front image of the fundus, a front image of the anterior segment of the eye, a fundus image obtained by fluorescence imaging, or data acquired by OCT (three-dimensional OCT data). Includes En-Face images generated using data from The En-Face image is an OCTA En-Face image (motion contrast frontal image) generated using data of at least a partial range in the depth direction of the imaging target for three-dimensional OCTA data (three-dimensional motion contrast data) ) can be used. Three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Also, the imaging device is a device for capturing an image used for diagnosis. The imaging device is, for example, a device that obtains an image of a predetermined portion of a subject by irradiating the predetermined portion of the subject with radiation such as light, X-rays, electromagnetic waves, or ultrasonic waves, or detects radiation emitted from the subject. It includes a device for obtaining an image of a predetermined site by means of More specifically, the imaging apparatuses according to the various embodiments and modifications described above include at least an X-ray imaging apparatus, a CT apparatus, an MRI apparatus, a PET apparatus, a SPECT apparatus, an SLO apparatus, an OCT apparatus, an OCTA apparatus, and a fundus. Including cameras and endoscopes.

The OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Fourier-domain OCT devices may also include spectral-domain OCT (SD-OCT) devices and wavelength-swept OCT (SS-OCT) devices. Further, the SLO device and the OCT device may include a wavefront compensation SLO (AO-SLO) device and a wavefront compensation OCT (AO-OCT) device using a wavefront compensation optical system. In addition, the SLO device and the OCT device may include a polarization SLO (PS-SLO) device and a polarization OCT (PS-OCT) device for visualizing information on polarization phase difference and depolarization.

Further, in the trained models for retinal layer detection and image segmentation processing according to the various embodiments and modifications described above, the magnitude of the luminance value of the tomographic image, the order and inclination of the bright and dark portions, the position, distribution, continuity, etc. It can be considered that the characteristics and the like are extracted as part of the feature amount and used for the estimation process. Similarly, the trained model for evaluation of region label images, image quality improvement, image analysis, and diagnosis result generation also includes the magnitude of the brightness value of the tomographic image, the order and inclination of the bright and dark portions, the position, distribution, It is conceivable that the continuity or the like is extracted as part of the feature amount and used for the estimation process. On the other hand, pre-trained models for voice recognition, character recognition, gesture recognition, etc. use time-series data for learning, so the gradient between input continuous time-series data values is a feature. It is thought that it is extracted as part of the quantity and used for the estimation process. Therefore, such a trained model is expected to be able to perform accurate estimation by using the influence of temporal changes in specific numerical values in the estimation process.

(various implementations)
Embodiment 1 of the present disclosure relates to a medical image processing apparatus. The medical image processing apparatus includes an acquisition unit that acquires a tomographic image of an eye to be inspected, and a trained model obtained by learning data representing at least one retinal layer among a plurality of retinal layers in the tomographic image of the eye to be inspected. and a first processing unit that performs a first detection process for detecting at least one retinal layer among a plurality of retinal layers in the acquired tomographic image using a.

Embodiment 2 includes the medical image processing apparatus according to Embodiment 1, wherein at least one retinal layer among the plurality of retinal layers is detected in the acquired tomographic image without using a trained model obtained by machine learning. It further comprises a second processing unit that executes a second detection process for detecting.

Embodiment 3 includes the medical image processing apparatus according to Embodiment 2, wherein the second detection process detects at least one retina other than at least one retinal layer detected by performing the first detection process. This is a process for detecting layers.

Embodiment 4 includes the medical image processing apparatus according to Embodiment 2 or 3, wherein the first detection process is a process of detecting a retinal region in the acquired tomographic image as the at least one retinal layer. , the second detection process is a process of detecting at least one retinal layer in the retinal region detected by executing the first detection process.

Embodiment 5 includes the medical image processing apparatus according to any one of Embodiments 2 to 4, wherein the first detection processing is performed from the boundary between the inner limiting membrane and the nerve fiber layer of the eye to be examined to the inner segment of photoreceptors. Outer segment junction, retinal pigment epithelium layer, and processing for detecting layers up to one of Bruch's membrane, wherein the second detection processing includes at least one layer between the layers detected by the first detection processing This is a process for detecting retinal layers.

Embodiment 6 includes the medical image processing apparatus according to any one of Embodiments 2 to 5, wherein the second processing unit performs the first detection processing after the first detection processing by the first processing unit. The second detection process is executed.

Embodiment 7 includes the medical image processing apparatus according to Embodiment 2, further comprising a display control unit that controls a display unit, wherein the first detection process and the second detection process detect the same retinal layer. The display control unit causes the display unit to display the processing results of the first detection processing and the second detection processing.

Embodiment 8 includes the medical image processing apparatus according to Embodiment 7, wherein the display control unit causes the display unit to display a mismatched portion of the processing results of the first detection process and the second detection process.

Embodiment 9 includes the medical image processing apparatus according to Embodiment 7 or 8, wherein the first detection process and the second detection process detect photoreceptor cells from the boundary between the inner limiting membrane and the nerve fiber layer of the eye to be examined. This is a process of detecting a layer up to one of the junction of the inner segment and the outer segment, the retinal pigment epithelium layer, and Bruch's membrane, and the second processing unit performs the first detection process according to the operator's instruction. and a third detection process of detecting at least one retinal layer between the layers detected by either one of the second detection processes.

Embodiment 10 includes the medical image processing apparatus according to any one of Embodiments 2 to 9, wherein the first detection process and the second detection process are performed based on imaging conditions related to the acquired tomographic image. further comprising a selection unit that selects at least one of

Embodiment 11 includes the medical image processing apparatus according to any one of Embodiments 2 to 10, wherein the first processing unit processes a plurality of trained models machine-learned using different learning data. Among them, the first detection process is executed using a learned model that has been machine-learned using learning data corresponding to imaging conditions related to the acquired tomographic image.

Embodiment 12 includes the medical image processing apparatus according to Embodiment 10 or 11, wherein the imaging conditions include at least one of imaging region, imaging method, imaging region, imaging angle of view, and image resolution.

Embodiment 13 includes the medical image processing apparatus according to any one of Embodiments 2 to 12, wherein the shape feature of the subject's eye is measured based on the results of the first detection process and the second detection process. be done.

Embodiment 14 includes the medical image processing apparatus according to any one of Embodiments 1 to 13, and corrects the structure of the retinal layer detected by the first processing unit based on the medical characteristics of the retinal layer. A corrector is further provided.

Embodiment 15 includes the medical image processing apparatus according to any one of Embodiments 1 to 14, wherein the first processing unit uses the learned model to pre-process an input image for each imaging region. Detect defined boundaries.

Embodiment 16 includes the medical image processing apparatus according to any one of Embodiments 1 to 15, wherein the detected at least one It further comprises a generation unit that generates a frontal image corresponding to the depth range determined based on the retinal layers.

Embodiment 17 includes the medical image processing apparatus according to Embodiment 16, wherein the generator corresponds to the determined depth range using three-dimensional motion contrast data corresponding to the three-dimensional tomographic image. Generate a motion-contrast frontal image.

Embodiment 18 includes the medical image processing apparatus according to any one of Embodiments 1 to 15, and uses a trained model for improving image quality to obtain the acquired tomographic image from the acquired tomographic image. The first processing unit performs the first detection process on the generated tomographic image.

Embodiment 19 includes the medical image processing apparatus according to any one of Embodiments 1 to 18, and a correction unit that corrects the information of the retinal layers detected by the first processing unit in accordance with an operator's instruction. wherein the modified retinal layer information is used for additional learning of the trained model used by the first processing unit.

Embodiment 20 includes the medical image processing apparatus according to any one of Embodiments 1 to 18, and uses a trained model for generating diagnosis results, and from the result obtained by executing the first detection process and a diagnostic result generating unit for generating a diagnostic result of the acquired tomographic image.

Embodiment 21 relates to a medical image processing method. The medical image processing method includes the steps of acquiring a tomographic image of an eye to be inspected, and using a trained model to detect at least one retinal layer of a plurality of retinal layers of the eye to be inspected in the tomographic image. and performing a detection process for.

Embodiment 22 relates to a program. The program, when executed by a processor, causes the processor to perform each step of the medical image processing method according to Embodiment 21.

A further embodiment 1 of the present disclosure relates to a medical image processing apparatus. The medical image processing apparatus uses a segmentation engine including a trained model to generate region information capable of identifying an anatomical region from an input image, which is a tomographic image of a predetermined region of a subject. and an evaluation unit that evaluates the region information using an evaluation engine that includes a trained model or an evaluation engine that performs knowledge-based processing using anatomical knowledge.

A further embodiment 2 includes the medical image processing apparatus according to the further embodiment 1, further comprising an imaging condition acquisition unit that acquires imaging conditions for the input image, wherein the segmentation processing unit performs , switching between multiple segmentation engines each containing a different pre-trained model.

A further embodiment 3 includes the medical image processing apparatus according to the further embodiment 2, wherein the imaging condition acquisition unit uses an imaging location estimation engine including a learned model to obtain an imaging region and an imaging region from the input image. Estimate at least one of

Further embodiment 4 includes the medical image processing apparatus according to any one of further embodiments 1 to 3, wherein the segmentation processing unit sets the image size of the input image to an image that the segmentation engine can handle. Adjust to size and input to the segmentation engine.

Further embodiment 5 includes the medical image processing apparatus according to any one of further embodiments 1 to 3, wherein the segmentation processing unit is configured such that the image size of the input image is an image that can be handled by the segmentation engine. An image obtained by padding the input image to the size is input to the segmentation engine.

Further embodiment 6 includes the medical image processing apparatus according to any one of further embodiments 1 to 3, wherein the segmentation processing unit divides the input image into images of a plurality of regions, Each image of the region is input to the segmentation engine.

A further embodiment 7 includes the medical image processing apparatus according to any one of the further embodiments 1 to 6, wherein the evaluation unit determines whether or not to output the region information according to the evaluation result. to decide.

Further embodiment 8 includes the medical image processing apparatus according to any one of further embodiments 1 to 7, wherein the segmentation processing unit uses a plurality of segmentation engines each including a different trained model to perform the The plurality of area information are generated from the input image, and the evaluation unit evaluates the plurality of area information and selects at least one of the plurality of area information determined to be output according to a user instruction. select.

Further embodiment 9 includes the medical image processing apparatus according to any one of further embodiments 1 to 7, wherein the segmentation processing unit uses a plurality of segmentation engines each including a different trained model to perform the The plurality of region information are generated from the input image, and the evaluation unit evaluates the plurality of region information based on a predetermined selection criterion and determines at least one of the plurality of region information to be output. to select.

A further embodiment 10 includes the medical image processing apparatus according to any one of further embodiments 1 to 9, wherein the segmentation engine is used to determine whether the region information can be generated from the input image. It further comprises a determination unit for determining.

Further Embodiment 11 includes the medical image processing apparatus according to any one of Further Embodiments 1 to 10, wherein the segmentation processing unit divides the input image into a plurality of segments having a dimension lower than that of the input image. It is divided into images, and each divided image is input to the segmentation engine.

Further embodiment 12 includes the medical image processing apparatus according to further embodiment 11, wherein the segmentation processor processes the plurality of images in parallel using a plurality of segmentation engines.

A further embodiment 13 includes the medical image processing apparatus according to any one of further embodiments 1 to 12, wherein the region information is a labeled image in which regions are labeled for each pixel.

Further embodiment 14 includes the medical image processing apparatus according to further embodiment 13, wherein the segmentation engine receives the tomographic image and outputs the label image.

Further embodiment 15 includes the medical image processing apparatus according to further embodiment 14, wherein the trained model of the segmentation engine receives a tomographic image including two or more layers as input data, and corresponds to the tomographic image. This is a model trained using label images as output data.

Further embodiment 16 includes the medical image processing apparatus according to any one of further embodiments 1 to 15, wherein the input image is acquired from an imaging device, or the predetermined image of the subject is obtained from the imaging device. Part data is obtained, and the input image based on the data is obtained.

Further embodiment 17 includes the medical image processing apparatus according to any one of further embodiments 1 to 15, wherein the input image is obtained from an image management system, and the area information is output to the image management system. or obtaining the input image from the image management system and outputting the area information to the image management system.

Further embodiment 18 includes the medical image processing apparatus according to any one of further embodiments 1 to 17, further comprising a correction unit that corrects the region information by anatomical knowledge base processing.

A further embodiment 19 includes the medical image processing apparatus according to any one of the further embodiments 1 to 18, and performs image analysis of the input image using the region information output from the evaluation unit. It further comprises an analysis unit.

A further embodiment 20 includes the medical image processing apparatus according to any one of further embodiments 1 to 19, outputting that the region information is information generated using a trained model.

A further embodiment 21 relates to a medical image processing method. The medical image processing method uses a segmentation engine including a trained model to generate region information capable of identifying an anatomical region from an input image that is a tomographic image of a predetermined region of a subject; evaluating the regional information using an evaluation engine that includes a pre-existing model or a knowledge-based evaluation engine that uses anatomical knowledge.

A further embodiment 22 relates to a program. The program, when executed by a processor, causes the processor to perform the steps of the medical image processing method according to further embodiment 21.

(Other examples)
The present invention provides a program that implements one or more functions of the above-described embodiments and modifications to a system or device via a network or a storage medium, and the computer of the system or device reads and executes the program. It is feasible. A computer has one or more processors or circuits and may include separate computers or a network of separate processors or circuits for reading and executing computer-executable instructions.

A processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuitry may include a digital signal processor (DSP), data flow processor (DFP), or neural processing unit (NPU).

The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are included to publicize the scope of the invention.

20: Image processing device, 21: Acquisition unit, 222: Processing unit (first processing unit)

Claims (26)

an acquisition unit that acquires a tomographic image of an eye to be inspected;
A trained model obtained by learning using learning data including a tomographic image of an eye to be inspected and an image containing position information about at least one layer in the tomographic image, wherein the acquired model is input data for the trained model. By using the obtained tomographic image, by outputting an image containing position information about at least one layer in the obtained tomographic image as output data of the learned model, at least one layer in the obtained tomographic image a detector that detects positional information about the layer;
generating a front image corresponding to at least a partial depth range in a three-dimensional tomographic image of the eye to be inspected, the depth range being determined using the position information regarding the detected at least one layer; a generation unit that generates a front image with higher image quality than the front image by using the front image as input data for a trained model different from the model;
a display control unit that causes a display unit to display at least one of the front image and the high-quality front image;
A medical image processing apparatus comprising: 2. The medical image processing according to claim 1, wherein said generator generates a motion contrast front image corresponding to said determined depth range using three-dimensional motion contrast data corresponding to said three-dimensional tomographic image. Device. 3. The three-dimensional tomographic image is a plurality of two-dimensional tomographic images including the acquired tomographic image, and is composed of a plurality of two-dimensional tomographic images at different positions of the subject's eye. 3. The medical image processing apparatus according to . 4. The medical image processing according to any one of claims 1 to 3, wherein the generating unit generates an image analysis result regarding the acquired tomographic image using position information regarding the detected at least one layer. Device. The generation unit uses the detected position information about the at least one layer to generate at least one of a thickness graph showing the thickness of the at least one layer in the acquired tomographic image and a thickness map showing the thickness in a color map. is generated as the image analysis result. 6. The medical image processing apparatus according to any one of claims 1 to 5, wherein the detection unit corrects the position information regarding the detected at least one layer in accordance with an instruction from an operator. 7. The detection unit uses the detected position information about the at least one layer to detect position information about another layer that cannot be obtained from the detected position information about the at least one layer. The medical image processing apparatus according to any one of . 8. The detector according to any one of claims 1 to 7 , wherein the detection unit detects position information about other layers that cannot be obtained from the detected position information about the at least one layer without using a trained model obtained by machine learning. or the medical image processing apparatus according to claim 1. By using the acquired tomographic image as input data for the trained model, the detection unit detects a plurality of layers in the acquired tomographic image as position information related to at least one layer in the acquired tomographic image. 9 . The medical image processing apparatus according to claim 7 , wherein the position information about at least one layer among the plurality of layers is detected as the position information about the other layer. The detection unit uses the acquired tomographic images as input data for the trained model to obtain positional information about the plurality of layers from the boundary between the inner limiting membrane and the nerve fiber layer of the eye to be examined to the sclera. 10. The medical image according to claim 9 , wherein a plurality of layer boundaries between the plurality of layer boundaries are detected, and at least one layer boundary between the plurality of layer boundaries is detected as position information regarding at least one layer between the plurality of layers processing equipment. 11. The detector according to any one of claims 7 to 10 , wherein the detecting unit detects the positional information about at least one layer and the positional information about the other layer in the acquired tomographic image using the same arithmetic device or different arithmetic devices. 1. The medical image processing apparatus according to item 1. The display control unit displays a two-dimensional tomographic image corresponding to a position determined according to an instruction from an examiner on the display unit with position information about the detected at least one layer superimposed thereon . 12. The medical image processing apparatus according to any one of claims 1 to 11 , wherein 13. The display control unit according to any one of claims 1 to 12 , wherein the display control unit causes the display unit to display that position information regarding at least one layer in the acquired tomographic image was not properly detected by the detection unit. 1. The medical image processing apparatus according to item 1. 14. The display control unit according to any one of claims 1 to 13 , wherein the display control unit causes the display unit to display that position information regarding the detected at least one layer has been detected using a learned model. Medical image processing equipment. 15. The medical image processing apparatus according to any one of claims 1 to 14 , wherein an image containing position information relating to at least one layer in the tomographic image included in said learning data is an image labeled for each pixel. The acquisition unit further acquires a front fundus image of the subject's eye,
The detection unit uses the acquired front fundus image as input data for a trained model obtained by learning using learning data including a front fundus image of the subject eye and information about an optic papilla region in the front fundus image. 16. The medical image processing apparatus according to any one of claims 1 to 15 , wherein the image is used to detect information about the region of the optic papilla in the acquired front fundus image. The detection unit uses the detected positional information about the at least one layer and the detected information about the optic papilla region to obtain positional information about at least one layer of the optic papilla in the acquired tomographic image. 17. The medical image processing apparatus according to claim 16 , which detects The medical image processing apparatus according to claim 16 or 17 , wherein the detection unit calculates an optic disc recession/optic disc ratio using information about the detected optic disc region. The trained model includes a plurality of trained models,
The detection unit is
selecting a trained model to be used for detecting position information about at least one layer in the acquired tomographic image based on the acquired tomographic image acquisition conditions and learning data of the plurality of learned models;
An image containing position information about at least one layer in the acquired tomographic image as output data of the selected trained model by using the acquired tomographic image as input data of the selected trained model 19. The medical image processing apparatus according to any one of claims 1 to 18 , wherein the position information regarding at least one layer in the acquired tomographic image is detected by outputting . The trained model includes a plurality of trained models,
3. The detection unit selects, as an output, one of a plurality of detection results obtained by using the acquired tomographic image as input data for the plurality of trained models in accordance with an instruction from an operator. 19. The medical image processing apparatus according to any one of 1 to 18 . The detection unit is
by using the acquired tomographic image as input data for a trained model different from the trained model for detecting position information about at least one layer in the acquired tomographic image, also generates high-quality tomographic images,
Position information about at least one layer in the acquired tomographic image, using the generated tomographic image as input data for a trained model for detecting position information about at least one layer in the acquired tomographic image 21. The medical image processing apparatus according to any one of claims 1 to 20 , which detects The acquisition unit acquires a three-dimensional tomographic image of an eye to be examined,
The detection unit divides the acquired three-dimensional tomographic image into a plurality of two-dimensional images, and detects position information regarding at least one layer in the acquired tomographic image for each of the divided two-dimensional images. The medical image processing apparatus according to any one of claims 1 to 21 , which is used as input data for a trained model. The detection unit outputs an image including position information about at least one layer in the acquired tomographic image as output data of the learned model, and uses the output data of the learned model to detect the acquired tomographic image. 23. The medical image processing apparatus of any one of claims 1-22 , wherein the apparatus detects positional information for at least one layer in an image. a photographing device for photographing an eye to be examined;
The medical image processing apparatus according to any one of claims 1 to 23 , communicably connected to the imaging apparatus;
A medical imaging system, comprising: a step of acquiring a tomographic image of an eye to be examined;
A trained model obtained by learning using learning data including a tomographic image of an eye to be inspected and an image containing position information about at least one layer in the tomographic image, wherein the acquired model is input data for the trained model. By using the obtained tomographic image, by outputting an image containing position information about at least one layer in the obtained tomographic image as output data of the learned model, at least one layer in the obtained tomographic image detecting location information about the layer;
generating a front image corresponding to at least a partial depth range in a three-dimensional tomographic image of the eye to be inspected, the depth range being determined using the position information regarding the detected at least one layer; generating a front image with higher image quality than the front image by using the front image as input data for a trained model different from the model;
displaying at least one of the front image and the high-quality front image on a display unit;
A medical image processing method, comprising: A program which, when executed by a processor, causes the processor to perform the steps of the medical image processing method according to claim 25 .

Images