WO2020036182A1 - Medical image processing device, medical image processing method, and program - Google Patents

Medical image processing device, medical image processing method, and program Download PDF

Info

Publication number
WO2020036182A1
WO2020036182A1 PCT/JP2019/031883 JP2019031883W WO2020036182A1 WO 2020036182 A1 WO2020036182 A1 WO 2020036182A1 JP 2019031883 W JP2019031883 W JP 2019031883W WO 2020036182 A1 WO2020036182 A1 WO 2020036182A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
processing
unit
region
label
Prior art date
Application number
PCT/JP2019/031883
Other languages
French (fr)
Japanese (ja)
Inventor
好彦 岩瀬
秀謙 溝部
律也 富田
Original Assignee
キヤノン株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from JP2019147739A external-priority patent/JP7229881B2/en
Application filed by キヤノン株式会社 filed Critical キヤノン株式会社
Priority to KR1020217006768A priority Critical patent/KR102543875B1/en
Priority to DE112019004104.5T priority patent/DE112019004104T5/en
Priority to GB2103260.2A priority patent/GB2591890B/en
Priority to CN201980053915.XA priority patent/CN112601487A/en
Publication of WO2020036182A1 publication Critical patent/WO2020036182A1/en
Priority to US17/168,776 priority patent/US20210158525A1/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/102—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for optical coherence tomography [OCT]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/0016—Operational features thereof
    • A61B3/0025—Operational features thereof characterised by electronic signal processing, e.g. eye models
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/12—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes
    • A61B3/1225—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes using coherent radiation
    • G06T5/70—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10101—Optical tomography; Optical coherence tomography [OCT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30041—Eye; Retina; Ophthalmic

Definitions

  • the present invention relates to a medical image processing device, a medical image processing method, and a program.
  • An ophthalmic tomographic imaging apparatus such as an apparatus (OCT apparatus) using optical coherence tomography (OCT: Optical Coherence Tomography) can three-dimensionally observe a state inside a retinal layer.
  • OCT apparatus optical coherence tomography
  • This tomographic imaging apparatus has been attracting attention in recent years because it is useful for more accurately diagnosing diseases.
  • TD-OCT Time domain OCT
  • Michelson interferometer combines a broadband light source and a Michelson interferometer. This is configured to scan the delay of the reference arm, measure the interference light with the backscattered light of the signal arm, and obtain the information of the depth resolution.
  • TD-OCT Time domain OCT
  • SD-OCT Spectral domain OCT
  • SS-OCT Spept ⁇ Source ⁇ OCT
  • the conventional technology has the following problems.
  • the shape of the retina becomes irregular due to loss of layers, bleeding, and the occurrence of vitiligo and new blood vessels. Therefore, in the conventional image processing method of determining the result of the image feature extraction using the regularity of the shape of the retina and detecting the boundary of the retinal layer, erroneous detection or the like occurs when automatically detecting the boundary of the retinal layer. There is a limit that occurs.
  • an object of the present invention is to provide a medical image processing apparatus, a medical image processing method, and a program that can detect a boundary of a retinal layer regardless of a disease, a part, or the like.
  • the medical image processing apparatus learns an acquisition unit that acquires a tomographic image of an eye to be inspected, and data indicating at least one retinal layer among a plurality of retinal layers in the tomographic image of the eye to be inspected.
  • a first processing unit that executes a first detection process for detecting at least one retinal layer among a plurality of retinal layers in the acquired tomographic image using the learned model obtained in the above.
  • a medical image processing method includes a step of acquiring a tomographic image of the eye to be inspected, and data showing at least one retinal layer among a plurality of retinal layers in the tomographic image of the eye to be inspected. Executing a first detection process for detecting at least one retinal layer among a plurality of retinal layers in the acquired tomographic image, using a learned model obtained by learning the above.
  • FIG. 1 illustrates an example of a schematic configuration of an image processing system according to a first embodiment. It is a figure for explaining an eye part. It is a figure for explaining a tomographic image. It is a figure for explaining a fundus image. 6 is a flowchart of a series of processes according to the first embodiment. It is a figure for explaining an example of a learning image. It is a figure for explaining an example of a learning image. It is a figure for explaining the example of the size of a learning image. It is a figure for explaining the example of the size of a learning image. It is a figure for explaining the example of the size of a learning image. It is a figure for explaining the example of the size of a learning image. FIG. 4 is a diagram for describing an example of a machine learning model according to the first embodiment.
  • FIG. 4 shows an example of a display screen.
  • 9 illustrates an example of a schematic configuration of an image processing system according to a second embodiment.
  • 9 is a flowchart of a series of processes according to the second embodiment.
  • 9 is a flowchart of a boundary detection process according to the second embodiment.
  • FIG. 3 is a diagram for explaining detection of a retinal region.
  • FIG. 3 is a diagram for explaining detection of a retinal region. It is a figure for explaining the example of the size of a learning image. It is a figure for explaining the example of the size of a learning image. It is a figure for explaining the example of the size of a learning image. It is a figure for explaining the example of the size of a learning image.
  • FIG. 14 is a diagram for describing an example of a machine learning model according to a second embodiment.
  • FIG. 9 is a diagram for explaining retinal layer detection according to a second embodiment.
  • FIG. 9 is a diagram for explaining retinal layer detection according to a second embodiment.
  • FIG. 9 is a diagram for explaining retinal layer detection according to a second embodiment.
  • FIG. 9 is a diagram for explaining retinal layer detection according to a second embodiment.
  • It is a figure for explaining an example of an input and output picture in a learned model. It is a figure for explaining an example of an input and output picture in a learned model. It is a figure for explaining an example of an input and output picture in a learned model. It is a figure for explaining an example of an input and output picture in a learned model. It is a figure for explaining an example of an input and output picture in a learned model.
  • 14 illustrates an example of a schematic configuration of an image processing system according to a fourth embodiment.
  • FIG. 13 is a flowchart of a series of processes according to a fourth embodiment. 13 is a flowchart of a series of processes according to a fourth embodiment. 15 shows an example of a schematic configuration of an image processing system according to Embodiment 5.
  • 19 is a flowchart of a series of processes according to a fifth embodiment. 19 is a flowchart of a boundary detection process according to the fifth embodiment. It is a figure for explaining correction processing of a retinal area. It is a figure for explaining correction processing of a retinal area. It is a figure for explaining correction processing of a retinal area. It is a figure for explaining correction processing of a retinal area. It is a figure for explaining correction processing of a retinal area. It is a figure for explaining correction processing of a retinal area.
  • FIG. 19 is a diagram for describing an example of a learning image according to a sixth embodiment.
  • FIG. 5 shows an example of En-Face images of a plurality of OCTAs.
  • 4 shows an example of a tomographic image having a plurality of luminances.
  • 17 illustrates an example of a user interface according to a seventh embodiment. 17 illustrates an example of a user interface according to a seventh embodiment. 17 illustrates an example of a user interface according to a seventh embodiment. 13 shows an example of an area label image according to the explanation of terms. 1 shows an example of the configuration of a neural network according to the explanation of terms. 1 shows an example of the configuration of a neural network according to the explanation of terms. 13 shows an example of an area label image according to the explanation of terms. 17 shows an example of the configuration of an image processing apparatus according to an eighth embodiment.
  • FIG. 19 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to the eighth embodiment.
  • 19 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to the eighth embodiment.
  • FIG. 18 is a diagram illustrating an example of a user interface provided in the imaging device according to the eighth embodiment.
  • FIG. 18 is a diagram illustrating an example of a user interface provided in the imaging device according to the eighth embodiment.
  • 19 is a flowchart illustrating an example of a flow of a process of the image processing apparatus according to the ninth embodiment.
  • 21 shows image processing according to the eleventh embodiment.
  • 33 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to the eleventh embodiment.
  • FIG. 14 shows image processing according to Embodiment 12.
  • 33 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to Embodiment 13.
  • 14 shows image processing according to Embodiment 13.
  • 33 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to Embodiment 13.
  • 14 shows image processing according to Embodiment 13.
  • 24 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to Embodiment 14.
  • FIG. 30 is a diagram illustrating an example of a user interface provided in the imaging device according to Embodiment 15.
  • FIG. 47 shows an example of the configuration of the image processing apparatus according to Embodiment 18.
  • FIG. 39 is a flowchart illustrating an example of the flow of the process of the image processing apparatus according to Embodiment 19.
  • 15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9.
  • 15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9.
  • 15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9.
  • 15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9.
  • 15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9.
  • a machine learning model refers to a learning model based on a machine learning algorithm such as deep learning.
  • the trained model is a model that has been trained (learned) in advance by using appropriate teacher data with respect to a machine learning model using an arbitrary machine learning algorithm. However, it is assumed that the learned model does not perform any further learning and can perform additional learning.
  • teacher data refers to learning data, and is composed of a pair of input data and output data.
  • the correct data refers to output data of learning data (teacher data).
  • FIG. 1 shows an example of a schematic configuration of an image processing system 1 including an image processing device 20 (medical image processing device) according to the present embodiment.
  • the image processing system 1 includes an OCT apparatus 10, an image processing apparatus 20, a fundus image capturing apparatus 30, an external storage device 40, a display unit 50, and an input unit 60, which are examples of a tomographic image capturing apparatus. Is provided.
  • the OCT apparatus 10 is an example of a tomographic image capturing apparatus that captures a tomographic image of an eye to be inspected.
  • Any type of OCT device can be used as the OCT device, and for example, SD-OCT or SS-OCT can be used.
  • the image processing device 20 is connected to the OCT device 10, the fundus image photographing device 30, the external storage device 40, the display unit 50, and the input unit 60 via an interface, and can control them.
  • the image processing device 20 generates various images such as a tomographic image and an En-Face image (front image) of the subject's eye based on various signals acquired from the OCT device 10, the fundus image capturing device 30, and the external storage device 40. can do. Further, the image processing device 20 can perform image processing on these images.
  • the image processing apparatus 20 may be configured by a general-purpose computer, or may be configured by a computer dedicated to the image processing system 1.
  • the fundus image photographing device 30 is a device for photographing a fundus image of the eye to be inspected.
  • a fundus camera for example, a fundus camera, an SLO (Scanning Laser Ophthalmoscope), or the like can be used.
  • the device configuration of the OCT device 10 and the fundus image photographing device 30 may be an integrated type or a separate type.
  • the external storage device 40 stores information relating to the subject's eye (patient's name, age, gender, etc.) in association with various types of captured image data, imaging parameters, image analysis parameters, and parameters set by the operator. I have.
  • the external storage device 40 may be configured by an arbitrary storage device, for example, may be configured by a storage medium such as an optical disk or a memory.
  • the display unit 50 is configured by an arbitrary display, and can display information about the eye to be inspected and various images under the control of the image processing device 20.
  • the input unit 60 is, for example, a mouse, a keyboard, a touch operation screen, or the like.
  • the operator performs image processing on the instruction to the image processing device 20, the OCT device 10, and the fundus image capturing device 30 via the input unit 60. It can be input to the device 20.
  • the input unit 60 is a touch operation screen, the input unit 60 can be configured integrally with the display unit 50.
  • the OCT apparatus 10 includes a light source 11, a galvanometer mirror 12, a focus lens stage 13, a coherence gate stage 14, a detector 15, and an internal fixation lamp 16. Since the OCT apparatus 10 is a known apparatus, a detailed description thereof will be omitted, and here, a description will be given of tomographic image capturing performed in accordance with an instruction from the image processing apparatus 20.
  • the light source 11 emits light.
  • the light from the light source 11 is split into measurement light and reference light using a splitting unit (not shown).
  • the OCT apparatus 10 generates an interference signal including tomographic information of the subject by irradiating the subject (eye to be examined) with measurement light and detecting interference light between the return light from the subject and the reference light. be able to.
  • the galvanomirror 12 is used for scanning the measurement light on the fundus of the eye to be inspected, and the scanning range of the measurement light by the galvanomirror 12 can define an imaging range of the fundus by the OCT imaging.
  • the image processing device 20 controls the driving range and the speed of the galvanomirror 12 so that the imaging range in the planar direction and the number of scanning lines (scanning speed in the planar direction) on the fundus can be defined.
  • the galvanometer mirror 12 is shown as one unit for simplicity of description, but the galvanometer mirror 12 is actually composed of two mirrors for X scan and Y scan, and The desired range above can be scanned with the measuring light.
  • the configuration of the scanning unit for scanning the measurement light is not limited to the galvanomirror, and any other deflecting mirror can be used. Further, as the scanning unit, for example, a deflecting mirror that can scan the measurement light in two-dimensional directions with one sheet such as a MEMS mirror may be used.
  • the focus lens stage 13 is provided with a focus lens (not shown). By moving the focus lens stage 13, the focus lens can be moved along the optical axis of the measurement light. Therefore, the measurement light can be focused on the retinal layer of the fundus through the anterior segment of the subject's eye by the focus lens. The measurement light illuminating the fundus is reflected and scattered by each retinal layer and returns to the optical path as return light.
  • the coherence gate stage 14 is used to adjust the length of the optical path of the reference light or the measurement light in order to cope with a difference in the axial length of the eye to be examined.
  • the coherence gate stage 14 is configured by a stage provided with a mirror, and can move the optical path length of the reference light to correspond to the optical path length of the measurement light by moving in the optical axis direction in the optical path of the reference light. it can.
  • the coherence gate represents a position where the optical distance between the measurement light and the reference light in OCT is equal.
  • the coherence gate stage 14 can be controlled by the image processing device 20.
  • the image processing device 20 can control the imaging range in the depth direction of the subject's eye by controlling the position of the coherence gate by the coherence gate stage 14, and can perform imaging on the retinal layer side or imaging on the deeper side than the retinal layer. Shooting and the like can be controlled.
  • the 15 detector 15 detects an interference light between the reference light and the return light of the measurement light from the subject's eye, which is generated in an interference unit (not shown), and generates an interference signal.
  • the image processing device 20 can generate a tomographic image of the subject's eye by acquiring an interference signal from the detector 15 and performing a Fourier transform or the like on the interference signal.
  • the internal fixation lamp 16 is provided with a display unit 161 and a lens 162.
  • a display unit 161 an example in which a plurality of light emitting diodes (LDs) are arranged in a matrix is used as an example of the display unit 161.
  • the lighting position of the light emitting diode is changed according to a part to be photographed under the control of the image processing device 20.
  • Light from the display unit 161 is guided to the subject's eye via the lens 162.
  • the light emitted from the display unit 161 has a wavelength of, for example, 520 nm, and is displayed in a desired pattern under the control of the image processing device 20.
  • the OCT device 10 may be provided with a drive control unit for the OCT device 10 that controls the driving of each component based on the control of the image processing device 20.
  • FIG. 2A is a schematic diagram of an eyeball.
  • FIG. 2A shows the cornea C, the lens CL, the vitreous V, the macula M (the central portion of the macula represents the fovea), and the optic disc D.
  • the OCT apparatus 10 can also photograph the anterior eye such as the cornea and the crystalline lens.
  • FIG. 2B shows an example of a tomographic image obtained by imaging the retina using the OCT apparatus 10.
  • AS indicates an image unit obtained by one A-scan.
  • the A-scan refers to acquiring the tomographic information in the depth direction at one point of the subject's eye by the above-described series of operations of the OCT apparatus 10.
  • acquiring two-dimensional tomographic information in the transverse direction and the depth direction of the subject's eye by performing the A scan a plurality of times in an arbitrary transverse direction (main scanning direction) is referred to as a B scan.
  • a B scan By collecting a plurality of A-scan images acquired by the A-scan, one B-scan image can be formed.
  • this B-scan image is referred to as a tomographic image.
  • FIG. 2B shows blood vessel Ve, vitreous body V, macula M, and optic disc D.
  • the boundary line L1 is a boundary between the inner limiting membrane (ILM) and the nerve fiber layer (NFL), the boundary line L2 is a boundary between the nerve fiber layer and the ganglion cell layer (GCL), and the boundary line L3 is a photoreceptor inner segment. Represents the outer joint (ISOS).
  • the boundary line L4 represents the retinal pigment epithelium layer (RPE), the boundary line L5 represents the Bruch's membrane (BM), and the boundary line L6 represents the choroid.
  • the horizontal axis OCT main scanning direction
  • the vertical axis depth direction
  • FIG. 2C shows an example of a fundus image acquired by photographing the fundus of the eye to be examined using the fundus image photographing apparatus 30.
  • FIG. 2C shows the macula M and the optic papilla D, and the blood vessels of the retina are represented by thick curves.
  • the horizontal axis OCT main scanning direction
  • the vertical axis OCT sub-scanning direction
  • the image processing device 20 includes an acquisition unit 21, an image processing unit 22, a drive control unit 23, a storage unit 24, and a display control unit 25.
  • the acquisition unit 21 can acquire the data of the interference signal of the subject's eye from the OCT apparatus 10.
  • the data of the interference signal acquired by the acquisition unit 21 may be an analog signal or a digital signal.
  • the image processing device 20 can convert the analog signal into a digital signal.
  • the acquisition unit 21 can acquire various images such as tomographic data, tomographic images, and En-Face images generated by the image processing unit 22.
  • the tomographic data is data including information on a tomographic image of a subject, and is data based on an interference signal by OCT and data obtained by performing fast Fourier transform (FFT: Fast @ Fourier @ Transform) or arbitrary signal processing on the data.
  • FFT Fast @ Fourier @ Transform
  • the acquiring unit 21 may obtain a group of imaging conditions of a tomographic image to be subjected to image processing (for example, imaging date and time, imaging part name, imaging area, imaging angle of view, imaging method, image resolution and gradation, image pixel size, image size, Filter, and information on the data format of the image).
  • image processing for example, imaging date and time, imaging part name, imaging area, imaging angle of view, imaging method, image resolution and gradation, image pixel size, image size, Filter, and information on the data format of the image.
  • the photographing condition group is not limited to the illustrated one. Further, the photographing condition group does not need to include all of the illustrated ones, and may include some of them.
  • the acquisition unit 21 can also acquire data including fundus information acquired by the fundus image photographing device 30 and the like. Further, the acquiring unit 21 can acquire information for identifying the subject's eye such as the subject identification number from the input unit 60 or the like. The acquisition unit 21 can cause the storage unit 24 to store the acquired various data and images.
  • the image processing unit 22 generates a tomographic image, an En-Face image, and the like from the data acquired by the acquiring unit 21 and the data stored in the storage unit 24, and can perform image processing on the generated or acquired image. . Therefore, the image processing unit 22 can function as an example of a generation unit that generates an En-Face image or a motion contrast front image described later.
  • the image processing unit 22 includes a tomographic image generation unit 221 and a processing unit 222 (first processing unit).
  • the tomographic image generating unit 221 can generate tomographic data by performing processing such as Fourier transform on the interference signal acquired by the acquiring unit 21 and generate a tomographic image based on the tomographic data. Note that any known method may be used as a method for generating a tomographic image, and a detailed description thereof will be omitted.
  • the processing unit 222 can include a learned model related to a machine learning model based on a machine learning algorithm such as deep learning. A specific machine learning model will be described later.
  • the processing unit 222 executes a detection process for detecting the retinal layer of the eye to be inspected in the tomographic image using the learned model, and detects each retinal layer.
  • the drive control unit 23 can control the driving of each component of the OCT device 10 and the fundus image capturing device 30 connected to the image processing device 20.
  • the storage unit 24 can store the tomographic data acquired by the acquiring unit 21 and various images and data such as tomographic images generated and processed by the image processing unit 22. Further, the storage unit 24 can also store a program or the like for performing the function of each component of the image processing device 20 by being executed by the processor.
  • the display control unit 25 can control display on the display unit 50 of various information acquired by the acquisition unit 21, tomographic images generated and processed by the image processing unit 22, and information input by an operator. .
  • Each component other than the storage unit 24 of the image processing apparatus 20 may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit).
  • the processor may be, for example, a GPU (Graphical Processing Unit) or an FPGA (Field-Programmable Gate Array).
  • each of the components may be configured by a circuit or the like that performs a specific function such as an ASIC.
  • the storage unit 24 may be configured by an arbitrary storage medium such as an optical disk and a memory.
  • FIG. 3 is a flowchart of a series of processes according to the present embodiment.
  • step S301 the acquiring unit 21 acquires the subject identification number, which is an example of information for identifying the subject's eye, from outside the image processing device 20 such as the input unit 60.
  • the acquiring unit 21 acquires information on the subject's eye held by the external storage device 40 based on the subject identification number and stores the information in the storage unit 24.
  • step S302 the drive control unit 23 controls the OCT apparatus 10 to scan the subject's eye to perform imaging, and the acquisition unit 21 acquires from the OCT apparatus 10 an interference signal including tomographic information of the subject's eye.
  • the scanning of the subject's eye is performed by the drive control unit 23 controlling the OCT device 10 and operating the light source 11, the galvanomirror 12, and the like in response to an instruction to start scanning by the operator.
  • the galvanometer mirror 12 includes a horizontal X scanner and a vertical Y scanner. Therefore, the drive control unit 23 can scan the measuring light in each of the horizontal direction (X) and the vertical direction (Y) in the apparatus coordinate system by changing the directions of these scanners.
  • the drive control unit 23 can scan the measurement light in a direction obtained by combining the horizontal direction and the vertical direction by simultaneously changing the directions of the scanners. Therefore, the drive control unit 23 can scan the measurement light in any direction on the fundus plane.
  • the drive control unit 23 adjusts various shooting parameters when shooting. Specifically, the drive control unit 23 sets at least the position of the pattern displayed by the internal fixation lamp 16, the scan range and scan pattern by the galvanomirror 12, the coherence gate position, and the focus.
  • the drive control unit 23 controls the light emitting diode of the display unit 161 to control the position of the pattern displayed by the internal fixation lamp 16 so as to image the center of the macula of the subject's eye and the optic disc.
  • the drive control unit 23 sets a scan pattern such as a raster scan, a radial scan, or a cross scan for capturing a three-dimensional volume as a scan pattern of the galvanometer mirror 12. Regardless of which scan pattern is selected, one line is repeatedly photographed (the number of repetitions is two or more). In the present embodiment, a case will be described where the scan pattern is a cross scan and the same portion is repeatedly photographed 150 times.
  • the drive control unit 23 controls the OCT apparatus 10 to perform imaging of the subject's eye in response to an instruction to start imaging by the operator.
  • the number of repetitions according to the present embodiment is an example, and may be set to an arbitrary number according to a desired configuration.
  • the OCT apparatus 10 can perform tracking of the subject's eye in order to photograph the same portion for averaging. Thereby, the OCT apparatus 10 can scan the subject's eye while reducing the influence of the fixation tremor.
  • step S303 the tomographic image generation unit 221 generates a tomographic image based on the interference signal acquired by the acquisition unit 21.
  • the tomographic image generation unit 221 can generate a tomographic image by performing general reconstruction processing on each interference signal.
  • the tomographic image generation unit 221 removes fixed pattern noise from the interference signal.
  • the fixed pattern noise removal is performed by averaging the plurality of acquired A-scan signals to extract fixed pattern noise, and subtracting this from the input interference signal.
  • the tomographic image generation unit 221 performs a desired window function process in order to optimize the depth resolution and the dynamic range that are in a trade-off relationship when the interference signal is Fourier-transformed in a finite section.
  • the tomographic image generation unit 221 generates tomographic data by performing fast Fourier transform (FFT) processing on the interference signal that has been subjected to the window function processing.
  • FFT fast Fourier transform
  • the tomographic image generation unit 221 obtains each pixel value of the tomographic image based on the generated tomographic data, and generates a tomographic image.
  • the method of generating a tomographic image is not limited to this, and may be performed by any known method.
  • step S304 the processing unit 222 of the image processing unit 22 performs a retinal layer detection process.
  • the processing of the processing unit 222 will be described with reference to FIGS. 4A and 4B.
  • the processing unit 222 detects a retinal layer boundary in a plurality of tomographic images acquired using the OCT apparatus 10.
  • the processing unit 222 detects each retinal layer using a learned model related to a machine learning model on which machine learning has been performed in advance.
  • the learning data (teacher data) of the machine learning model according to the present embodiment includes one or more pairs of input data and output data.
  • the input data includes a tomographic image 401 obtained by OCT
  • the output data includes a boundary image 402 in which a boundary of a retinal layer is specified for the tomographic image.
  • an image showing a boundary 403 between the ILM and the NFL, a boundary 404 between the NFL and the GCL, the ISSO 405, the RPE 406, and the BM 407 is used as the boundary image 402.
  • boundaries include a boundary between the outer plexiform layer (OPL) and the outer granular layer (ONL), a boundary between the inner plexiform layer (IPL) and the inner granular layer (INL), and a boundary between INL and OPL.
  • OPL outer plexiform layer
  • IPL inner plexiform layer
  • INL inner granular layer
  • the boundary image 402 used as output data may be an image in which a boundary is indicated in a tomographic image by a doctor or the like, or may be an image in which a boundary has been detected by a rule-based boundary detection process.
  • a rule-based boundary detection process may be performed using boundary images for which boundary detection has not been performed properly as output data of teacher data, images obtained using a trained model trained using the teacher data will also be properly detected. There is a possibility that the boundary image will not be obtained. Therefore, by removing the pair including such a boundary image from the teacher data, it is possible to reduce the possibility that an inappropriate boundary image is generated using the learned model.
  • the rule-based processing refers to processing using known regularity
  • the rule-based boundary detection refers to boundary detection processing using known regularity such as, for example, the retina shape regularity.
  • FIGS. 4A and 4B show an example of one XZ cross section in the XY plane of the retina, but the cross section is not limited to this.
  • a tomographic image and a boundary image of an arbitrary plurality of XZ sections in the XY plane are learned in advance, and corresponding to sections taken in various different scan patterns such as a raster scan and a radial scan.
  • scan patterns such as a raster scan and a radial scan.
  • volume data obtained by aligning a plurality of adjacent tomographic images can be used as teacher data.
  • a pair image group of an arbitrary angle can be generated from one volume data (three-dimensional tomographic image) and one corresponding three-dimensional boundary data (three-dimensional boundary image). It is possible. Further, the machine learning model may learn using images actually captured with various scan patterns as teacher data.
  • the creation of the image will be described with reference to FIGS. 5A to 5C.
  • one of the group of pairs forming the teacher data is a tomographic image 401 and a boundary image 402.
  • a pair is formed by using a rectangular area image 501 that is the entire tomographic image 401 as input data and a rectangular area image 502 that is the entire boundary image 402 as output data.
  • a pair of input data and output data is formed by the entirety of each image, but the pair is not limited to this.
  • a pair may be formed by using a rectangular area image 511 of the tomographic image 401 as input data and a rectangular area image 513 as a corresponding imaging area in the boundary image 402 as output data.
  • the rectangular areas of the rectangular area images 511 and 513 are based on A-scan units.
  • the A scan unit may be one A scan unit or several A scan units.
  • FIG. 5B is based on the unit of A-scan, the whole area in the depth direction may not be the area of the image, and a part outside the rectangular area may be provided vertically. That is, the size of the rectangular area in the horizontal direction may be set to several A scans, and the size of the rectangular area in the depth direction may be set smaller than the size of the image in the depth direction.
  • a pair may be formed by using the rectangular area image 521 of the tomographic image 401 as input data and the rectangular area image 523 which is a corresponding imaging area in the boundary image 402 as output data.
  • the scan range (angle of view) and the scan density (the number of A-scans) are normalized to make the image size uniform, so that the rectangular area size at the time of learning can be made uniform.
  • the rectangular area images shown in FIGS. 5A to 5C are examples of the rectangular area size when learning is performed separately.
  • the number of rectangular areas can be set to one in the example shown in FIG. 5A, and a plurality can be set in the examples shown in FIGS. 5B and 5C.
  • a pair may be configured using the rectangular area image 512 of the tomographic image 401 as input data and the rectangular area image 514 that is a corresponding imaging area in the boundary image 402 as output data.
  • a pair can be formed by using the rectangular area image 522 of the tomographic image 401 as input data and the rectangular area image 524 as the corresponding imaging area in the boundary image 402 as output data. .
  • a pair of mutually different rectangular area images can be created from a pair of a tomographic image and a boundary image one by one.
  • the original tomographic image and the boundary image by creating a large number of pairs of rectangular area images while changing the position of the area to different coordinates, it is possible to enrich the group of pairs forming the teacher data.
  • the rectangular areas are discretely shown.
  • the original tomographic image and the boundary image are divided into a group of rectangular area images having a constant image size and without gaps. can do.
  • the original tomographic image and the boundary image may be divided into a group of rectangular area images at random positions corresponding to each other.
  • the image segmentation processing refers to processing for identifying or distinguishing an area or a boundary in an image.
  • FIG. 6 illustrates an example of a configuration 601 of a machine learning model in the processing unit 222.
  • the machine learning model according to the present embodiment for example, FCN (Fully Convolutional Network), SegNet, or the like can be used.
  • a machine learning model that performs object recognition on a region basis according to a desired configuration may be used.
  • RCNN Region @ CNN
  • fastRCNN fastRCNN
  • fastRCNN YOLO (You Look Only Once) or SSD (Single Shot MultiBox Detector) can be used as a machine learning model for performing object recognition in units of regions.
  • SSD Single Shot MultiBox Detector
  • the machine learning model shown in FIG. 6 is composed of a plurality of layer groups responsible for processing the input value group and outputting it.
  • the types of layers included in the configuration 601 of the machine learning model include a convolution layer, a downsampling (Downsampling) layer, an upsampling (Upsampling) layer, and a synthesis (Merger) layer.
  • the convolution layer is a layer that performs a convolution process on an input value group according to parameters such as a set filter kernel size, the number of filters, a stride value, and a dilation value.
  • the number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image.
  • the downsampling layer is a layer that performs processing to reduce the number of output value groups from the number of input value groups by thinning out or combining input value groups. Specifically, for example, there is a Max @ Pooling process.
  • the upsampling layer is a layer that performs processing for increasing the number of output value groups beyond the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. Specifically, as such processing, for example, there is a linear interpolation processing.
  • the composition layer is a layer that performs a process of inputting a value group such as an output value group of a certain layer or a pixel value group constituting an image from a plurality of sources, concatenating them, and adding them to combine them.
  • the parameter setting for the layer group or the node group forming the neural network is different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may be different. In other words, in many cases, appropriate parameters are different depending on the mode of implementation, and can be changed to preferable values as needed.
  • the CNN can obtain better characteristics.
  • the better characteristics include, for example, higher accuracy of the image segmentation process, shorter time for the image segmentation process, shorter time for training of the machine learning model, and the like.
  • the configuration 601 of the CNN used in this embodiment has a function of an encoder having a plurality of layers including a plurality of downsampling layers and a function of a decoder including a plurality of layers including a plurality of upsampling layers.
  • This is a net-type machine learning model.
  • position information spatial information which is ambiguous in a plurality of hierarchies configured as encoders is converted into a same-dimensional hierarchy (hierarchical layers corresponding to each other) in a plurality of hierarchies configured as decoders. ) (Eg, using a skip connection).
  • a batch normalization (Batch @ Normalization) layer or an activation layer using a normalized linear function (Rectifier @ Linear @ Unit) may be incorporated after the convolutional layer. Good.
  • the boundary image 402 is output according to the tendency trained using the teacher data.
  • the processing unit 222 can detect a retinal layer and its boundary in the tomographic image 401 based on the boundary image 402.
  • the processing unit 222 uses the learned model to generate a rectangular region image that is a boundary image corresponding to each rectangular region. Get. Therefore, the processing unit 222 can detect a retinal layer in each rectangular area. In this case, the processing unit 222 arranges and connects each of the rectangular area image groups, which are the boundary images obtained using the learned model, in the same positional relationship as each of the rectangular area image groups. , A boundary image 402 corresponding to the input tomographic image 401 can be generated. Also in this case, the processing unit 222 can detect the retinal layer and its boundary in the tomographic image 401 based on the generated boundary image 402.
  • step S304 when the processing unit 222 performs a retinal layer detection process, the process proceeds to step S305.
  • step S305 the display control unit 25 displays the boundary and the tomographic image detected by the processing unit 222 on the display unit 50.
  • FIG. 1 An example of a screen displayed on the display unit 50 is shown in FIG.
  • FIG. 7 shows a display screen 700.
  • the display screen 700 includes an SLO image 701, a thickness map 702 superimposed on the SLO image 701, an En-Face image 703, a tomographic image 711, and a retinal thickness graph. 712 is shown. Retinal boundaries 715 and 716 are superimposed on the tomographic image 711.
  • the range of the retina is defined as the boundary L1 between the inner limiting membrane and the nerve fiber layer to the retinal pigment epithelium layer L4, and the boundaries 715 and 716 correspond to the boundary L1 and the retinal pigment epithelium layer L4, respectively.
  • the range of the retina is not limited to this, and may be, for example, a range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the choroid L6.
  • the boundaries 715 and 716 may correspond to the boundary L1 and the choroid L6, respectively. it can.
  • the retinal thickness graph 712 is a graph showing the retinal thickness obtained from the boundaries 715 and 716.
  • the thickness map 702 expresses the thickness of the retina obtained from the boundaries 715 and 716 by a color map. Although color information corresponding to the thickness map 702 is not shown in FIG. 7 for the sake of explanation, the thickness map 702 is actually a color map corresponding to the thickness of the retina corresponding to each coordinate in the SLO image 701. It can be displayed according to the map.
  • the En-Face image 703 is a front image generated by projecting data in a range between boundaries 715 and 716 in the XY directions.
  • the front image is at least a part of the depth range of volume data (three-dimensional tomographic image) obtained by using optical interference, and two-dimensionally represents data corresponding to a depth range determined based on two reference planes. It is generated by projecting or integrating on a plane.
  • the En-Face image 703 according to the present embodiment is obtained by converting data corresponding to a depth range (depth range between boundaries 715 and 716) of the volume data determined based on the detected retinal layer into a two-dimensional plane. Is a front image generated by projecting the image on the front side.
  • a representative value of data within the depth range is defined as a pixel value on the two-dimensional plane.
  • the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in a depth direction of a region surrounded by two reference planes.
  • the depth range of the En-Face image 703 shown on the display screen 700 is not limited to the depth range between the boundaries 715 and 716.
  • the depth range of the En-Face image 703 is, for example, a range including a predetermined number of pixels in a deeper or shallower direction with reference to one of the two layer boundaries 715 and 716 related to the detected retinal layer. There may be.
  • the depth range of the En-Face image 703 is, for example, a range changed (offset) from the range between the two layer boundaries 715 and 716 related to the detected retinal layer in accordance with the instruction of the operator. It may be.
  • the front image shown on the display screen 700 is not limited to the En-Face image based on the luminance value (En-Face image of luminance).
  • the front image shown on the display screen 700 is, for example, a motion contrast front image generated by projecting or integrating data corresponding to the above-described depth range onto a two-dimensional plane with respect to motion contrast data between a plurality of volume data. Is also good.
  • the motion contrast data is data indicating a change among a plurality of volume data obtained by controlling the measurement light to scan a plurality of times in the same region (same position) of the eye to be inspected.
  • the volume data is composed of a plurality of tomographic images obtained at different positions.
  • motion contrast data can be obtained as volume data.
  • the motion contrast front image is also called an OCTA front image (OCTA En-Face image) related to OCT angiography (OCTA) for measuring blood flow movement, and the motion contrast data is also called OCTA data.
  • the motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value between two tomographic images or interference signals corresponding thereto (maximum value / minimum value). May be determined by any known method.
  • the two tomographic images can be obtained, for example, by controlling the measurement light to scan a plurality of times in the same region (same position) of the eye to be inspected.
  • three-dimensional OCTA data used when generating an OCTA front image is generated using at least a part of a common interference signal with volume data including a tomographic image for detecting a retinal layer. May be done.
  • the volume data (three-dimensional tomographic image) and the three-dimensional OCTA data can correspond to each other. Therefore, using the three-dimensional motion contrast data corresponding to the volume data, for example, a motion contrast front image corresponding to a depth range determined based on the detected retinal layer can be generated.
  • the display of the thickness map 702, the En-Face image 703, the thickness graph 712, and the boundaries 715 and 716 may be generated by the image processing device 20 based on the boundaries and the retinal layers detected by the processing unit 222.
  • the image processing device 20 may generate the display of the thickness map 702, the En-Face image 703, the thickness graph 712, and the boundaries 715 and 716.
  • any known method may be adopted as a generation method for generating these.
  • the display screen 700 of the display unit 50 may include a patient tab, an imaging tab, a report tab, a setting tab, and the like in addition to the above. In this case, the contents shown on the display screen 700 in FIG. 7 are displayed on the report tab. Further, the display screen 700 can also display a patient information display section, an examination sort tab, an examination list, and the like.
  • the examination list may display thumbnails of a fundus image, a tomographic image, and an OCTA image.
  • step S306 the acquisition unit 21 acquires an instruction from the outside as to whether or not to end a series of processes relating to imaging of a tomographic image by the image processing system 1.
  • This instruction can be input by the operator using the input unit 60.
  • the acquisition unit 21 acquires an instruction to end the processing
  • the image processing system 1 ends a series of processing according to the present embodiment.
  • the process returns to step S302 to continue shooting.
  • the image processing device 20 includes the acquisition unit 21 and the processing unit 222 (first processing unit).
  • the acquisition unit 21 acquires a tomographic image of the subject's eye.
  • the processing unit 222 executes a first detection process for detecting at least one retinal layer among a plurality of retinal layers of the subject's eye in the tomographic image using the learned model.
  • the boundary can be appropriately detected according to the learned tendency. For this reason, in the image processing device 20 according to the present embodiment, by performing the image segmentation process using the learned model, it is possible to perform the boundary detection regardless of the disease, the part, and the like, and to improve the accuracy of the boundary detection. be able to.
  • the image processing device 20 generates a front image corresponding to at least a part of the depth range in the three-dimensional tomographic image of the eye to be inspected and corresponding to the depth range determined based on at least one detected retinal layer.
  • the image processing unit 22 further includes an image processing unit 22.
  • the image processing unit 22 can generate a motion contrast front image corresponding to the determined depth range using the three-dimensional motion contrast data corresponding to the three-dimensional tomographic image.
  • the configuration in which the image segmentation process is performed using one learned model has been described.
  • the image segmentation process may be performed using a plurality of learned models.
  • the trained model generates output data according to the tendency of learning using teacher data as described above. Reproducibility can be improved. Therefore, for example, by performing image segmentation processing on a tomographic image of a corresponding imaging region using a plurality of learned models that have learned for each imaging region, a more accurate boundary image can be generated. . In this case, the image processing system can more accurately detect the retinal layer. In this case, it is also possible to increase the number of learning models additionally, so that it is expected that a version upgrade in which performance is gradually improved can be performed.
  • the processing unit 222 uses a plurality of learning models that have been trained for each region such as a vitreous body region, a retinal region, and a sclera region in the tomographic image, and processes the combined learning model outputs. A final output of the unit 222 may be generated. In this case, a more accurate boundary image can be generated for each region, so that the retinal layer can be detected more accurately.
  • a machine learning model for estimating the imaging region of the tomographic image may be used.
  • the configuration of a machine learning model is not limited to a configuration that outputs an image corresponding to an image that is input data.
  • a machine learning model may be configured to output a type of output data trained using teacher data with respect to input data, or to output a possibility as a numerical value for each of the types.
  • the format and combination of the input data and the output data of the pair group that constitutes the teacher data may be such that one is an image and the other is a numerical value, one is a plurality of image groups and the other is a character string, Both can be images, and so on, which can be suitable for the usage form.
  • teacher data of the machine learning model for estimating the imaging region there is teacher data configured by a pair group of a tomographic image acquired by OCT and an imaging region label corresponding to the tomographic image.
  • the imaging part label is a unique numerical value or character string representing the part.
  • the processing unit 222 further estimates the imaging region of the tomographic image using such a learned model for estimating the imaging region, and uses the learned model corresponding to the estimated imaging region or the imaging region with the highest probability. Image segmentation processing may be performed. In such a configuration, even when the acquisition unit 21 cannot acquire the imaging conditions related to the imaging region of the tomographic image, the imaging region is estimated from the tomographic image, and the image segmentation process corresponding to the imaging region is performed. The retinal layer can be accurately detected.
  • Example 2 In the first embodiment, an image segmentation process for detecting all target retinal layers from a tomographic image is performed using a trained model. On the other hand, in the second embodiment, based on the result of detection of the retinal region by the learned model, boundary detection is performed using rule-based image features.
  • optic papilla it is customary to detect the open end of the Bruch's membrane when detecting Cup (optic papilla) and Disc (optic papilla) using OCT images. In some cases, such as retinal atrophy, it is difficult to detect it.
  • the retinal region is detected using the learned model, and the detected retinal region is used together with the boundary detection based on the image feature.
  • the detection accuracy of the inner layer of the retina is improved, and in the process of machine learning, only the retinal layer or other correct data is created at the time of learning.
  • the image processing system 8 according to the present embodiment will be described with reference to FIGS. 8 to 13D.
  • image processing by the image processing system according to the present embodiment will be described focusing on differences from the image processing according to the first embodiment.
  • the configuration and processing of the image processing system according to the present embodiment that are the same as the configuration and processing of the image processing system 1 according to the first embodiment are denoted by the same reference numerals, and description thereof is omitted.
  • FIG. 8 shows an example of a schematic configuration of the image processing system 8 according to the present embodiment.
  • a first processing unit 822 and a second processing unit 823 are provided instead of the processing unit 222 in the image processing unit 82 of the image processing device 80.
  • the first processing unit 822 has a trained model related to a machine learning model by a machine learning algorithm such as deep learning, and detects a retinal region in a tomographic image using the trained model.
  • the second processing unit 823 determines the result of the image feature extraction for the retinal region detected by the first processing unit 822 on a rule basis, and performs retinal layer boundary detection.
  • FIG. 9A is a flowchart of a series of processes according to the present embodiment
  • FIG. 9B is a flowchart of a boundary detection process in the present embodiment. Note that the processing other than the boundary detection processing is the same as the processing of the first embodiment, and thus the description is omitted.
  • the process proceeds to step S904.
  • step S904 When the boundary detection processing in step S904 is started, the processing shifts to step S941.
  • the first processing unit 822 detects a retinal region in a tomographic image using a learned model as a first boundary detection process.
  • the learning data (teacher data) of the machine learning model according to the present embodiment includes one or more pairs of input data and output data.
  • the teacher data a pair group of a tomographic image 1001 obtained by the OCT imaging illustrated in FIG. 10A and a label image 1002 in which a label is assigned to an arbitrary layer from the tomographic image 1001 illustrated in FIG. 10B is configured.
  • Teacher data and the like are configured.
  • the label image is an image that is labeled for each pixel (an image obtained by annotation), and in the present embodiment, a label related to an image appearing (photographed) at the pixel for each pixel.
  • a label image 1002 as examples of labels, a label 1003 on the shallower side (vitreous body side) than the retina, a label 1004 on the inner layer of the retina, and a label 1005 on the deeper side (choroidal side) than the retina are given.
  • the first processing unit 822 in the present embodiment detects the retinal inner layer based on such a label image.
  • the range of the retina (the range of the inner layer of the retina) is defined as the boundary L1 between the inner limiting membrane and the nerve fiber layer to the retinal pigment epithelium layer L4, but is not limited thereto.
  • the range of the retina is defined as the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the range of the junction between the photoreceptor inner and outer segments L3, the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the Bruch's membrane L5, or It may be defined as a range from the boundary L1 between the limiting membrane and the nerve fiber layer to the choroid L6.
  • 10A and 10B show an example of one XZ cross section in the XY plane of the retina, but the cross section is not limited to this.
  • arbitrary plural XZ sections in the XY plane may be learned in advance so that sections corresponding to various different scan patterns such as a raster scan and a radial scan can be handled. it can.
  • data such as a tomographic image obtained by three-dimensionally photographing the retina by raster scanning
  • volume data obtained by aligning a plurality of adjacent tomographic images can be used as teacher data.
  • the machine learning model may learn using images actually captured with various scan patterns as teacher images.
  • the creation of the image will be described with reference to FIGS. 11A to 11C.
  • one of the group of pairs forming the teacher data is a tomographic image 1001 and a label image 1002.
  • a pair is formed using the rectangular area image 1101 that is the entire tomographic image 1001 as input data and the rectangular area image 1102 that is the entire label image 1002 as output data.
  • a pair of input data and output data is formed by the entire image, but the pair is not limited to this.
  • a pair may be formed using a rectangular area image 1111 of the tomographic image 1001 as input data and a rectangular area image 1113 which is a corresponding imaging area in the label image 1002 as output data.
  • the rectangular area images 1111 and 1113 are based on A-scan units.
  • the A scan unit may be one A scan unit or several A scan units.
  • FIG. 11B is based on the unit of A-scan, the whole area in the depth direction of the image may not be the entire area, but a part outside the rectangular area may be provided at the top and bottom. That is, the size of the rectangular area in the horizontal direction may be set to several A scans, and the size of the rectangular area in the depth direction may be set smaller than the size of the image in the depth direction.
  • a pair may be formed using the rectangular area image 1121 of the tomographic image 1001 as input data and the rectangular area image 1123 which is a corresponding imaging area in the label image 1002 as output data.
  • the size of the rectangular area is a size that includes a plurality of labels in one rectangular area.
  • the scan range (angle of view) and the scan density (the number of A-scans) are normalized to make the image size uniform, so that the rectangular area size at the time of learning can be made uniform.
  • the rectangular area images shown in FIGS. 11A to 11C are examples of the rectangular area size when learning is performed separately.
  • the number of rectangular areas can be set to one in the example shown in FIG. 11A, and a plurality can be set in the examples shown in FIGS. 11B and 11C.
  • a pair can be formed by using a rectangular area image 1112 of the tomographic image 1001 as input data and a rectangular area image 1114 as a corresponding imaging area in the label image 1002 as output data.
  • a pair can be formed using the rectangular area image 1122 of the tomographic image 1001 as input data and the rectangular area image 1124 that is the corresponding imaging area in the label image 1002 as output data. .
  • a pair of mutually different rectangular area images can be created from a pair of a tomographic image and a label image one by one.
  • the original tomographic image and label image by creating a large number of pairs of rectangular area images while changing the position of the area to different coordinates, it is possible to enrich the group of pairs forming the teacher data.
  • the original tomographic image and label image are divided into a group of rectangular area images having a constant image size and without gaps. can do.
  • the original tomographic image and label image may be divided into a group of rectangular area images at random positions corresponding to each other.
  • an image of a smaller area as a rectangular area (or a strip area) as a pair of input data and output data, many pairs are obtained from the tomographic image 1001 and the label image 1002 that constitute the original pair. Can generate data. Therefore, the time required for training the machine learning model can be reduced.
  • the trained model of the completed machine learning model the time of the executed image segmentation process tends to be long.
  • FIG. 12 illustrates an example of a configuration 1201 of a machine learning model in the first processing unit 822.
  • FCN convolutional neural network
  • SegNet convolutional neural network
  • the machine learning model shown in FIG. 12 is configured by a plurality of layer groups that are responsible for processing of processing the input value group and outputting the same as in the example of the machine learning model according to the first embodiment shown in FIG.
  • the types of layers included in the configuration 1201 of the machine learning model include a convolution layer, a downsampling layer, an upsampling layer, and a composite layer. Note that the configurations of these layers and the modifications of the configuration of the CNN are the same as those of the machine learning model according to the first embodiment, and thus detailed descriptions thereof will be omitted.
  • the CNN configuration 1201 used in the present embodiment is a U-net type machine learning model, like the CNN configuration 601 described in the first embodiment.
  • the learned model of the first processing unit 822 when the tomographic image 1001 is input, a label image 1002 is output according to the tendency trained using the teacher data.
  • the first processing unit 822 can detect a retinal region in the tomographic image 1001 based on the label image 1002.
  • the first processing unit 822 uses the learned model to generate a label image corresponding to each rectangular region. Obtain a rectangular area image. Therefore, the first processing unit 822 can detect a retinal layer in each rectangular area. In this case, the first processing unit 822 arranges and combines each of the rectangular area image groups, which are the label images obtained using the learned model, in the same positional relationship as each of the rectangular area image groups. I do. Thereby, the first processing unit 822 can generate the label image 1002 corresponding to the input tomographic image 1001. Also in this case, the first processing unit 822 can detect a retinal region in the tomographic image 1001 based on the generated label image 1002.
  • step S941 when the retinal region is detected by the first processing unit 822, the process proceeds to step S942.
  • the second processing unit 823 performs, as a second detection process, a retinal region by a rule-based image segmentation process based on the retinal region detected by the first processing unit 822 in the tomographic image 1001 illustrated in FIG. 10A. Detect remaining boundaries in inner layers.
  • FIG. 13A shows a tomographic image 1001 which is an example of an input tomographic image.
  • FIG. 13B is a label image 1002 output by the first processing unit 822, which is an image to which labels 1004 of the retina region and labels 1003 and 1005 corresponding to the other are added.
  • the second processing unit 823 according to the present embodiment sets the range of the retinal region indicated by the label 1004 in the label image 1002 as the target region for layer detection.
  • the second processing unit 823 can detect the target boundary by detecting the contour in the retinal region indicated by the label 1004 in the label image 1002.
  • FIG. 13C shows an edge-enhanced image 1303 that has been processed by the second processing unit 823. The processing by the second processing unit 823 will be described below. As shown in FIGS. 13C and 13D, the retinal layer of the optic disc is interrupted, so that the second processing unit 823 does not perform the boundary detection.
  • the second processing unit 823 performs noise removal and edge enhancement on the area corresponding to the label 1004 in the tomographic image 1001 to be processed.
  • the second processing unit 823 applies, for example, a median filter or a Gaussian filter as the noise removal processing. Further, the second processing unit 823 applies a Sobel filter or a Hessian filter as the edge enhancement processing.
  • the Hessian filter can emphasize a secondary local structure of a two-dimensional grayscale distribution based on a relationship between two eigenvalues ( ⁇ 1 , ⁇ 2 ) of the Hessian matrix. Therefore, in this embodiment, the two-dimensional line structure is emphasized using the relationship between the eigenvalues of the Hessian matrix and the eigenvectors (e 1 , e 2 ). Since the line structure in the two-dimensional tomographic image of the eye to be examined corresponds to the structure of the retinal layer, the structure of the retinal layer can be emphasized by applying the Hessian filter.
  • the resolution of the smoothing by the Gaussian function performed when calculating the Hessian matrix may be changed.
  • a two-dimensional Hessian filter When a two-dimensional Hessian filter is applied, it can be applied after data is deformed to match the physical size of XZ of the image.
  • the physical sizes in the XY direction and the Z direction are different. Therefore, the filter is applied by matching the physical size of the retinal layer for each pixel. Since the physical size in the XY direction and the Z direction can be grasped from the design / configuration of the OCT apparatus 10, the data of the tomographic image can be deformed based on the physical size. In the case where the physical size is not normalized, it can be approximately handled by changing the resolution of smoothing by the Gaussian function.
  • the processing using the two-dimensional tomographic image has been described, but the target to which the Hessian filter is applied is not limited to this.
  • the data structure at the time of capturing the tomographic image is a three-dimensional tomographic image obtained by raster scanning
  • a three-dimensional Hessian filter can be applied.
  • the second processing unit 823 outputs the three eigenvalues ( ⁇ 1 , ⁇ 2 , ⁇ 3 ) of the Hessian matrix. Based on the relation, the secondary local structure of the three-dimensional grayscale distribution can be emphasized.
  • edge-enhanced image 1303 a portion where the edge is enhanced appears as a white line 1304. Note that an area that does not correspond to the label 1004 in the tomographic image 1001 can be treated as an area where no edge is detected. Further, here, the configuration in which the edge enhancement processing is performed using the Hessian filter has been described, but the processing method of the edge enhancement processing is not limited to this, and may be performed by any existing method.
  • FIG. 13D shows a boundary image 1305 indicating the boundary of the retinal layer detected by the second processing unit 823 using the label image 1002 and the edge enhancement image 1303.
  • a black line 1306 shows an example of the boundary.
  • the second processing unit 823 detects the edge-enhanced boundary from the edge-enhanced image 1303. In this embodiment, since the first processing unit 822 has already detected the boundary between the ILM and the NFL and the RPE, the second processing unit 823 subsequently detects the boundary between the ISSO, the NFL, and the GCL.
  • other boundaries include a boundary between the outer plexiform layer (OPL) and the outer granular layer (ONL), a boundary between the inner plexiform layer (IPL) and the inner granular layer (INL), and a boundary between the INL and OPL.
  • a boundary, a boundary between GCL and IPL, or the like may be detected.
  • a method for detecting a boundary a plurality of points having high edge strength in each A scan are detected as boundary candidates, and a point (a point having high edge intensity) is converted into a line based on continuity between boundary candidates in adjacent A scans. Perform connection processing. Further, when the points are connected as a line, the second processing unit 823 can remove outliers by evaluating the smoothness of the line. More specifically, for example, the positions of the connected points in the Z direction are compared, and when the difference between the positions in the Z direction is larger than a predetermined threshold, the newly connected point is determined as an outlier, It can be excluded from the connecting process. When an outlier is removed, a boundary candidate in the A-scan adjacent to the A-scan position of the excluded point may be connected as a line.
  • the method of removing outliers is not limited to this, and may be performed by any existing method.
  • the second processing unit 823 determines a corresponding boundary of each line formed by connecting points based on the vertical distance and the positional relationship in the Z direction of the boundary of the retinal layer. If there is no boundary detected as a result of removing outliers in each A scan, the boundary may be obtained by interpolation from surrounding boundaries. Alternatively, a boundary candidate may be searched in the horizontal direction (X or Y direction) from the surrounding boundary by relying on the edge, and the boundary may be determined again based on the boundary candidate searched from the surrounding boundary. .
  • the second processing unit 823 executes a process for smoothly correcting the shape of the detected boundary.
  • the shape of the boundary may be smoothed using an image feature and a shape feature using a dynamic contour model such as Snakes or Level @ Set method.
  • the coordinate values of the boundary shape may be regarded as time-series data based on signals, and the shape may be smoothed by a Savitzky-Golay filter or a smoothing process such as a simple moving average, a weighted moving average, or an exponential moving average.
  • the second processing unit 823 can detect a retinal layer in the retinal region detected by the first processing unit 822.
  • the above-described retinal layer detection processing by the second processing unit 823 is an example, and the retinal layer can be detected using any existing segmentation processing.
  • the process proceeds to step S305. Subsequent processing is the same as in the first embodiment, and a description thereof will not be repeated.
  • the image processing device 80 includes the acquisition unit 21, the first processing unit 822, and the second processing unit 823.
  • the acquisition unit 21 acquires a tomographic image of the subject's eye.
  • the first processing unit 822 executes a first detection process for detecting at least one retinal layer among a plurality of retinal layers of the subject's eye in the tomographic image using the learned model.
  • the second processing unit 823 performs a second detection process for detecting at least one retinal layer among a plurality of retinal layers in the tomographic image without using the learned model.
  • the second processing unit 823 detects at least one retinal layer other than the at least one retinal layer detected by the first detection processing by the second detection processing.
  • the first detection processing is performed from the boundary between the inner limiting membrane and the nerve fiber layer of the eye to be examined to the junction between the photoreceptor inner and outer nodes, the retinal pigment epithelium layer, and any of the Bruch's membranes. This is the process of detecting.
  • the second detection process is performed after the first detection process, and is a process for detecting at least one retinal layer included in the layers detected by the first detection process, that is, between the detected layers. .
  • the image processing apparatus 80 it is possible to perform the boundary detection regardless of a disease or a part.
  • the accuracy of boundary detection can be improved by using boundary detection based on image features together with the region output by the machine learning model.
  • the retinal layer or other correct data need be created, so that learning can be performed efficiently.
  • the possibility of erroneous detection occurring in the output label image or boundary image may increase.
  • the detection of the retinal region using the machine learning model according to the present embodiment since there are few boundaries to be detected, erroneous detection in the output label image or boundary image can be suppressed.
  • the first processing unit 822 may be configured to detect a retinal region using a plurality of machine learning models. In this case, the accuracy of detecting the retinal region can be improved. Further, since it is possible to increase the number of learning models additionally, it is expected that a version upgrade in which performance is gradually improved can be performed.
  • the first processing unit 822 detects the retinal region using the learned model as a previous step. showed that.
  • the order of the processing by the first processing unit 822 and the processing by the second processing unit 823 is not limited to this. For example, when it takes a very long time for the first processing unit 822 to detect a retinal region, the second processing unit 823 may first execute the retinal region detection processing based on a rule base.
  • the second processing unit 823 first detects the boundary between the ILM and the NFL and the RPE or ISSO using the same method as the method by the second processing unit 823 described in the second embodiment. I do. This is because these boundaries are places where the luminance value is high in the retina, and are boundaries located at the shallow layer and the deep layer of the retina.
  • these boundaries may be detected based on a large blurred image subjected to noise processing several times. . In this case, since only global features can be detected, erroneous detection of other boundaries can be prevented.
  • the tomographic image may be binarized by a dynamic threshold to limit the retinal region, and the boundary between the ILM and the NFL and the RPE or ISSO may be detected from the retinal region.
  • the second processing unit 823 may detect the boundary between the ILM and the NFL and the BM.
  • the image processing unit 82 sets a parameter for checking erroneous detection of the retinal region, such as discontinuity of the retinal region boundary, local curvature, or dispersion of boundary coordinates in the local region, with a predetermined threshold. Compare. If these parameters exceed a predetermined threshold, the image processing unit 82 determines that the detection of the retinal region in the second processing unit 823 is an erroneous detection. When the image processing unit 82 determines that the detection of the retinal region by the second processing unit 823 is erroneous detection, the first processing unit 822 is configured to detect the retinal region.
  • a parameter for checking erroneous detection of the retinal region such as discontinuity of the retinal region boundary, local curvature, or dispersion of boundary coordinates in the local region. Compare. If these parameters exceed a predetermined threshold, the image processing unit 82 determines that the detection of the retinal region in the second processing unit 823 is an erroneous detection.
  • the first processing unit 822 is configured to detect
  • the configuration in which the processing of the second processing unit 823 is performed prior to the processing of the first processing unit 822 has been described, but these processings may be started simultaneously.
  • the image processing unit 82 determines that the detection of the retinal region by the second processing unit 823 is erroneous detection, the image processing unit 82 waits for the detection of the retinal region by the first processing unit 822 and 823 performs boundary detection of the inner layer of the retina. If the detection of the retinal region by the second processing unit 823 is performed appropriately, the processing by the first processing unit 822 is interrupted, or the processing result by the first processing unit 822 is discarded. be able to.
  • the display control unit 25 performs the detection by the first processing unit 822 and the second processing unit 823.
  • the processing result of the processing may be displayed on the display unit 50.
  • the second processing is performed on one of the processing results of the detection processing by the first processing unit 822 and the second processing unit 823.
  • the unit 823 may detect the boundary of the inner layer of the retina.
  • the detection processing of the retinal region by the second processing unit 823 may be defined as a second detection processing
  • the subsequent processing of detecting the boundary of the retinal inner layer by the second processing unit 823 may be defined as a third detection processing. .
  • Example 3 In the second embodiment, an example has been described in which the retinal region is detected using the learned model, and the boundary of the retinal inner layer is detected with respect to the detected retinal region.
  • a region to be detected using the learned model not only the retinal region but also a characteristic region with respect to the imaged portion of the image as input data is detected.
  • the image processing by the image processing system according to the present embodiment will be described focusing on the difference from the image processing according to the second embodiment.
  • the configuration and the processing procedure of the image processing system according to the present embodiment are the same as the configuration and the processing procedure of the image processing system 8 according to the second embodiment. .
  • FIGS. 14A to 14D show an example of an image of each part of the eye to be inspected and a label image of a processing result processed by the first processing unit 822.
  • FIG. 14A shows a tomographic image 1401 when the macula is photographed, and a label image 1402 in the macula obtained using the trained model.
  • the label image 1402 shows a vitreous label 1403, a label 1404 ranging from ILM to ISSO, a label 1405 ranging from ISSO to RPE, a choroid label 1406, and a sclera label 1407.
  • labels should be placed for each area where morphological changes are likely Set and let the machine learning model learn in advance.
  • a tomographic image of the macula is used as input data, and, for example, a label image with a vitreous label, a label ranging from ILM to ISSO, a label ranging from ISSO to RPE, a choroid label, and a sclera label Is output data.
  • the first processing unit 822 inputs a tomographic image of the macula into the learned model, thereby acquiring a label image indicating a label for each region where the above-described morphological change is likely to appear, and in units of those labels.
  • the area can be detected.
  • FIG. 14B shows a tomographic image 1411 when the optic papilla is imaged and a label image 1412 in the optic papilla obtained using the trained model.
  • the label image 1412 shows a vitreous label 1413, a label 1414 in the range from the ILM to the boundary between NFL and GCL, and a label 1415 in a range from the boundary between NFL and GCL to the boundary between GCL and IPL. Further, in the label image 1412, a label 1416 in a range from the boundary between the GCL and the IPL to the ISOS and a label 1417 in a range from the IOS to the RPE are shown.
  • the label image 1412 shows a label 1418 in the range from the RPE to the deep layer and a label 1419 in the range of the sieve plate.
  • a tomographic image of the optic papilla is used as input data, for example, a vitreous label, a label in the range from the ILM to the NFL and the GCL boundary, and a range from the boundary between the NFL and the GCL to the boundary between the GCL and the IPL.
  • Label images to which labels, labels ranging from ICL to IPL from the boundary of GCL and IPL, labels ranging from ISSO to RPE, labels ranging from RPE to deep layers, and labels ranging from cribrosa are used as output data.
  • the front image 1421 in FIG. 14C is an image viewed from the front in the XY plane at the time of imaging the optic papilla, and is an image captured using the fundus image capturing apparatus 30.
  • the label image 1422 in FIG. 14C is a label image obtained by using the learned model for the front image 1421 of the optic papilla.
  • the label image 1422 shows a label 1423, a Disc label 1424, and a Cup label 1425 on the periphery of the optic disc.
  • glaucoma tends to cause ganglion cell loss and morphological changes such as the end of the RPE (RPE-tip) or the open end of the Bruch's membrane (BMO), the lamina cribrosa, and the Cup and Disc. Therefore, a label is set for each of these areas, and the machine learning model is trained in advance.
  • learning data a front image of the optic papilla is used as input data, and, for example, a label image with a peripheral label, a Disc label, and a Cup label attached to the optic papilla is used as output data.
  • the first processing unit 822 obtains a label image indicating a label for each region where the above-mentioned morphological change is likely to appear by inputting an image of the optic disc to the learned model, and in units of those labels. The area can be detected.
  • FIG. 14D shows a tomographic image 1431 when the anterior segment is photographed and a label image 1432 in the anterior segment photographing obtained using the trained model.
  • the label image 1432 shows a corneal label 1433, an anterior chamber label 1434, an iris label 1435, and a lens label 1436.
  • the main region as described above is learned in advance by the machine learning model.
  • a tomographic image of the anterior segment is used as input data, and for example, a label image to which a corneal label, an anterior chamber label, an iris label, and a lens label are attached is used as output data.
  • the first processing unit 822 obtains a label image indicating a label for each region where the above-mentioned morphological change is likely to appear by inputting the image of the anterior segment into the learned model, and in units of those labels. The area can be detected.
  • the second processing unit 823 determines the remaining boundary in the tomographic images 1401, 1411, and 1431 shown in FIGS. 14A, 14B, and 14D based on the region detected by the first processing unit 822. To detect. Further, the second processing unit 823 may measure the thickness of the detected boundary or a layer region sandwiched between the boundaries, or the thickness of the region detected by the first processing unit 822.
  • the second processing unit 823 performs measurement on each area classified by the label with respect to the front image of FIG. 14C and calculates the height, width, area, and Cup / Disc ratio of each area. Can be. In addition, about these measurement and calculation of a ratio, you may use the existing arbitrary methods.
  • the second processing unit 823 executes the image processing algorithm corresponding to the region detected by the first processing unit 822, and sets a rule when the image processing algorithm is applied to each region. Can be changed.
  • the rule includes, for example, the type of boundary to be detected.
  • the second processing unit 823 may perform additional boundary detection on the tomographic image 1401 of the macula shown in FIG. 14A in the range from the ILM to the ISSO indicated by the label 1404, as in the first embodiment. Good.
  • the boundary detection method may be the same as the boundary detection method in the first embodiment.
  • the second processing unit 823 detects a boundary between the ILM and the NFL, a boundary between the OPL and the ONL, a boundary between the IPL and the INL, a boundary between the INL and the OPL, and a boundary between the GCL and the IPL for the tomographic image 1401 of the macula.
  • the image processing algorithm and rules may be applied as described above.
  • an arbitrary image processing algorithm and rule may be set for each region according to a desired configuration.
  • the first processing unit 822 detects a predetermined boundary of the input image for each imaging region.
  • the region detected using the learned model is not limited to the retinal region, but is a characteristic region with respect to the imaged region, and thus can cope with variations such as diseases.
  • the first processing unit 822 performs a predetermined boundary for each imaging region as first detection processing for detecting at least one of the retinal layers. Can be detected.
  • the first processing unit 822 detects a predetermined boundary for each imaging part as processing different from the first detection processing. can do.
  • the final output of the first processing unit 822 may be generated by combining the outputs of a plurality of learning models that have learned for each region of the tomographic image.
  • the height (thickness), width, area, Cup / Disc, and the like of each region relating to the subject's eye are determined based on the result of the first detection process or the second detection process. Certain shape features, such as ratios, can be measured.
  • Example 4 In the third embodiment, an example has been described in which the region detected using the learned model is not limited to the retinal region, and a characteristic region in a region to be imaged is detected. On the other hand, in the fourth embodiment, execution of processing using the learned model is selected according to the imaging conditions under which the image is captured, and further, the area to be detected is narrowed down using the learned model.
  • the image processing by the image processing system 150 according to the present embodiment will be described with reference to FIGS. 15 to 16B, focusing on the differences from the image processing according to the second embodiment.
  • the configuration and processing of the image processing system according to the present embodiment that are the same as the configuration and processing of the image processing system 8 according to the second embodiment are denoted by the same reference numerals, and description thereof is omitted.
  • FIG. 15 shows an example of a schematic configuration of an image processing system 150 according to the present embodiment.
  • the selection unit 1524 Is provided in the image processing unit 1520 of the image processing device 152.
  • the selection unit 1524 selects image processing to be performed on a tomographic image based on the imaging condition group acquired by the acquisition unit 21 and the learning content (teacher data) on the learned model of the first processing unit 822. Specifically, based on the imaging condition group and the learning of the learned model, the first processing unit 822 detects only the retinal layer, or the first processing unit 822 detects the retinal region and performs the second processing. Whether the retinal layer is detected by the unit 823 or the retinal layer is detected only by the second processing unit 823 is selected.
  • the selection unit 1524 determines whether the first processing unit 822 has the first processing unit based on the image capturing condition group and the learning content of the first processing unit 822 regarding the learned model. It is possible to select a learned model to be used for the detection process by the 822.
  • FIG. 16A is a flowchart of a series of processes according to the present embodiment
  • FIG. 16B is a flowchart of a boundary detection process according to the present embodiment. Note that the processing other than the boundary detection processing is the same as the processing of the second embodiment, and a description thereof will not be repeated.
  • step S1641 the acquisition unit 21 acquires a group of imaging conditions for the generated tomographic image, and the image processing unit 1520 performs processing by the first processing unit 822 and the second processing unit 823 from the acquired group of imaging conditions. Get the information you need to make a selection.
  • these conditions can include an imaging region, an imaging method, an imaging region, an imaging angle of view, an image resolution, and the like.
  • step S1642 the selection unit 1524 selects whether or not to execute the processing by the first processing unit 822 based on the shooting conditions acquired in step S1641.
  • the trained model of the first processing unit 822 a model for the optic disc and a model for the macula that have been trained using the teacher data for each imaging site of the optic disc and the macula.
  • the first processing unit 822 will be described as not being able to handle a wide-angle image (an image in a range where both the optic papilla and the macula are photographed).
  • the selecting unit 1524 determines that the input image is an image of the optic papilla or the macula alone, based on, for example, information on the imaging region name and the imaging angle of view in the imaging conditions.
  • the selecting unit 1524 selects to execute the processing by the first processing unit 822.
  • the process shifts to step S1643.
  • the selecting unit 1524 determines that the input image is an image obtained by capturing an image other than the optic papilla and the macula or a wide-angle image including both the optic papilla and the macula
  • the selecting unit 1524 Reference numeral 1524 selects that the processing by the first processing unit 822 is not executed. Accordingly, the process proceeds to step S1645.
  • the selection unit 1524 selects an appropriate learned model used by the first processing unit 822, based on the imaging conditions acquired in step S1641.
  • the selection unit 1524 determines that the input image is an image obtained by imaging the optic papilla, for example, based on the information of the imaging site name and the imaging angle of view, the selection unit 1524 generates the model for the optic papilla. select.
  • the selection unit 1524 determines that the input image is an image of the macula
  • the selection unit 1524 selects a model for the macula.
  • the learned model has learned only the image obtained by capturing the optic papilla and the macula has been described, but the learning content of the learned model is not limited to this.
  • a learned model that has been learned for other parts or a learned model that has been learned using a wide-angle image including the optic papilla and the macula may be used.
  • selection of a process or selection of a learned model may be performed according to the photographing method.
  • the photographing method include the SD-OCT and SS-OCT photographing methods, and the image quality, the photographing range, and the depth reach in the depth direction are different depending on the difference between the two photographing methods. Therefore, selection of an appropriate process and selection of a learned model may be performed for these images having different photographing methods. If the learning is performed at the same time regardless of the shooting method, it is not necessary to change the processing according to the shooting method. If there is only one trained model, there is no need to select a learning model in step S1643, so this process can be skipped.
  • step S1644 the first processing unit 822 performs a first boundary detection process using the learned model selected in step S1643.
  • the processing described in the first to third embodiments can be used for this processing.
  • the first processing unit 822 performs the first detection processing as in the first embodiment. All boundaries of interest can be detected.
  • the first processing unit 822 performs the same processing as the second embodiment as the first detection process. Then, the retinal region can be detected.
  • the first processing unit 822 performs As in the third embodiment, a characteristic region can be detected. Note that a specific detection method is the same as the detection method in the first to third embodiments, and thus the description is omitted.
  • step S1645 the selection unit 1524 selects whether or not to execute the processing by the second processing unit 823 based on the shooting conditions acquired in step S1641. If the selecting unit 1524 has selected to execute the process by the second processing unit 823, the process proceeds to step S1646. On the other hand, if the selection unit 1524 selects not to execute the processing by the second processing unit 823, the processing of step S1604 ends, and the processing moves to step S305.
  • step S1645 an example of the selection processing of the selection unit 1524 in step S1645 will be described.
  • the case where the processing by the second processing unit 823 is executed means that, for example, as described in the second and third embodiments, the second processing unit 823 generates a boundary based on the region detected by the first processing unit 822. Is detected.
  • the first processing unit 822 also executes the processing by the second processing unit 823 when an unlearned image is input. In this case, in step S1642, it is selected to skip the processing by the first processing unit 822. Therefore, the second processing unit 823 performs the rule-based image segmentation processing without using the learned model. Performs boundary detection.
  • the case where the processing by the second processing unit 823 is not performed is, for example, the case where the first processing unit 822 can detect all target boundaries using the learned model. In this case, since the processing is completed only by the first processing unit 822, the processing of the second processing unit 823 can be skipped.
  • the second processing unit 823 measures the layer thickness based on the detected boundaries. In the case of performing, the execution of the processing by the second processing unit 823 may be selected.
  • the measurement of the layer thickness based on the detected boundary is not limited to the configuration performed by the second processing unit 823, but may be performed by the image processing unit 1520. Therefore, even when the measurement of the layer thickness or the like is performed, the execution of the process by the second processing unit 823 may not be selected.
  • step S1646 the selection unit 1524 performs image processing necessary for performing the processing by the second processing unit 823 and selection of a rule when applying the image processing based on the imaging conditions acquired in step S1641. Do. For example, when the input image is a tomographic image as shown in FIGS. 14A, 14B, and 14D, the selection unit 1524 detects the remaining boundary based on the region detected by the first processing unit 822. Process and rules to be performed.
  • the selection unit 1524 selects an image process and a rule that enable the macula to be correctly recognized as a layer. Further, for example, in the case of the tomographic image 1411 obtained by photographing the optic papilla illustrated in FIG. Select image processing and rules that take into account exceptional processing of the nipple. Further, for example, in the case of the tomographic image 1431 obtained by photographing the anterior segment shown in FIG. 14D, the selection unit 1524 selects image processing and a rule that can perform further layer recognition of the cornea.
  • step S1647 in addition to the detection of the layer boundary or the measurement of the thickness of the detected boundary or the layer region sandwiched by the boundary, the selection unit 1524 performs such an image measurement function. Image processing required for the image processing can be selected.
  • step S1647 the second processing unit 823 detects a boundary and / or measures the detected boundary or region. Note that, similarly to the description of the second and third embodiments, the description of the process of detecting the boundary in the region based on the region detected by the first processing unit 822 and the process of measuring the boundary or the like will be omitted. .
  • the second processing unit 823 determines the boundary between the ILM and the NFL and the RPE or ISSO as described in the modification of the second embodiment. Is detected first.
  • the second processing unit 823 After detecting the boundary between the ILM and the NFL and the RPE or ISSO, the second processing unit 823 detects the remaining boundary based on the region between the boundaries. Since the detection processing is the same as the detection processing in the second and third embodiments, the description is omitted. When the second processing unit 823 executes these processes, the process proceeds to step S305. Note that, similarly to the modification of the second embodiment, the second processing unit 823 may first detect the boundary between the ILM and the NFL and the BM. Further, the subsequent processing is the same as the processing in the first to third embodiments, and thus the description is omitted.
  • the acquisition unit 21 acquires the imaging conditions for the tomographic image of the subject's eye.
  • the image processing device 152 further includes a selection unit 1524 for selecting a process based on a shooting condition.
  • the selection unit 1524 selects at least one of the first detection processing and the second detection processing based on the imaging conditions.
  • the boundary detection processing can be executed reliably. Therefore, by using at least one of the machine learning models at various stages of maturity created by machine learning and the image processing method of judging the result of the image feature extraction based on a rule base and detecting the boundary of the retinal layer, The accuracy of boundary detection can be improved.
  • the first processing unit 822 includes a plurality of learned models on which machine learning has been performed using different teacher data. Further, the first processing unit 822 executes the first detection process using a learned model that has been machine-learned using teacher data corresponding to the imaging condition, among the plurality of learned models.
  • the retinal layer can be detected using an appropriate learning model based on the imaging conditions, and more appropriate processing can be performed according to the input image. Further, since it is possible to increase the number of learning models additionally, it is expected that a version upgrade in which performance is gradually improved can be performed. Further, according to the selection unit 1524, the image processing and the rule used in the second processing unit 823 can be selected based on the imaging conditions, and more appropriate processing can be performed according to the input image.
  • the first processing unit 822 and the second processing unit 823 select the processing separately, but the procedure for selecting the processing is not limited to this.
  • the selection unit 1524 selects, in one step, processing by only the first processing unit 822, processing by only the second processing unit 823, or processing by the first processing unit 822 and the second processing unit 823. It may be configured as follows.
  • the first processing unit 822 can detect all target boundaries using the learned model, and the second processing unit 823 includes a retinal region that is the first detection target.
  • the first processing unit 822 and the second processing unit 823 output the results of detecting the respective boundaries separately.
  • the display control unit 25 can display the two results side by side on the display unit 50, display the results by switching, or display them in a superimposed manner.
  • the image processing unit 1520 can determine whether the two results match or not, and the display control unit 25 can display the mismatched portion on the display unit 50 in an emphasized manner. In this case, the reliability of the layer detection can be shown to the operator. Further, the display control unit 25 may display the mismatched portion on the display unit 50 so that a more satisfactory result can be selected according to the instruction of the operator.
  • Example 5 In the second to fourth embodiments, an example has been described in which a retinal region is detected using a learned model and a boundary of an inner retinal layer is detected based on a rule based on the detected retinal region. On the other hand, in the fifth embodiment, a region detected using the trained model is corrected based on medical characteristics.
  • image processing performed by the image processing system 170 according to the present embodiment will be described with reference to FIGS. 17 to 19D, focusing on differences from the image processing according to the second embodiment.
  • the configuration and processing of the image processing system according to the present embodiment that are the same as the configuration and processing of the image processing system 8 according to the second embodiment are denoted by the same reference numerals, and description thereof is omitted.
  • FIG. 17 shows an example of a schematic configuration of an image processing system 170 according to the present embodiment.
  • the image processing unit 1720 of the image processing device 172 in the image processing system 170 according to the present embodiment includes a correction unit 1724 in addition to the tomographic image generation unit 221, the first processing unit 822, and the second processing unit 823. Is provided.
  • the correction unit 1724 corrects the labeled area of the label image obtained by the first processing unit 822 using the learned model based on the medical characteristics of the eye. Accordingly, the image processing unit 1720 can more appropriately detect the retinal region and the characteristic region.
  • FIG. 18A is a flowchart of a series of processing according to the present embodiment
  • FIG. 18B is a flowchart of boundary detection according to the present embodiment. Note that the processing other than the boundary detection processing is the same as the processing of the second embodiment, and a description thereof will not be repeated.
  • step S1841 the correction unit 1724 corrects the retinal region detected in step S941 by the first processing unit 822. More specifically, the first processing unit 822 corrects the labeled area of the label image obtained using the learned model based on the medical characteristics of the eye.
  • FIG. 19A shows an example of a tomographic image 1901 to be input to the first processing unit 822.
  • FIG. 19B illustrates an example of a label image 1902 obtained by the first processing unit 822 using the tomographic image 1901 as an input and using the learned model.
  • the label image 1902 shows a label 1904 on the inner layer of the retina, a label 1903 on the shallower side (vitreous side) than the retina, and a label 1905 on the deeper side (choroid side) than the retina.
  • the labeling is based on the label setting at the time of learning of the trained model. Therefore, the type of label is not limited to this, and a plurality of labels may be set in the retinal layer as shown in the third embodiment. Even in that case, the correction processing according to the present embodiment can be applied.
  • the first processing unit 822 performs image segmentation processing in pixel units using the learned model. Therefore, as shown by the labels 1903 'and 1904' in FIG. 19B, erroneous detection may occur partially.
  • the correction unit 1724 corrects these erroneous detections based on the medical characteristics of the eye.
  • the first processing unit 822 performs a labeling process for each detected label, and pixels having the same label in adjacent pixels are integrated as one region.
  • three types of labels are provided: a label 1904 on the inner layer of the retina, a label 1903 on the shallower side (vitreous side) than the retina, and a label 1905 on the deeper side (choroid side) than the retina.
  • the target is an image obtained by capturing a retinal tomographic image
  • the order in which these labels appear is, in order from the top, a label 1903, a label 1904, and a label 1905.
  • the order in which the labels appear is from the top of the image to the labels 1905, 1904, and 1903 In order.
  • the correction unit 1724 specifies the detection result for each of the labeled regions, and corrects the region regarded as erroneous detection to a region estimated based on medical characteristics.
  • the correction unit 1724 specifies the area from the larger area of the labeled area, and determines the area where the area of the labeled area is equal to or smaller than the threshold, or the spatial area from the already specified area. It is determined that an object that is far away is an erroneous detection. After that, the correction unit 1724 resets the label information determined to be an erroneous detection.
  • FIG. 19C shows an example of this case.
  • An area 1910 illustrated in FIG. 19C indicates an area where label information has been reset for the areas indicated by the labels 1903 'and 1904', which are areas that have been determined to be erroneous by the correction unit 1724.
  • the correction unit 1724 assigns label information estimated from surrounding label information to the area 1910 in which the label information has been reset.
  • a label 1903 is assigned to an area 1910 surrounded by a label 1903
  • a label 1905 is assigned to an area 1910 surrounded by a label 1905.
  • a final label image 1920 is output as shown in FIG. 19D. Accordingly, the image processing unit 1720 can more appropriately detect the retinal region.
  • step S942 the second processing unit 823 performs a second boundary detection process based on the corrected retinal region, as in the second embodiment. Subsequent processing is the same as the processing of the second embodiment, and a description thereof will be omitted.
  • the image processing device 172 further includes the correction unit 1724 that corrects the structure of the retinal layer detected by the first processing unit 822 based on the medical characteristics of the retinal layer.
  • the region detected using the learned model can be corrected using the medical features. Therefore, erroneous detection can be reduced even when an image is detected in pixel units.
  • the correction processing by the correction unit 1724 is added to the processing according to the second embodiment.
  • the correction processing may be added to the processing according to the third or fourth embodiment.
  • the boundary of the inner layer of the retina and the retinal region are detected using the learned model in the captured tomographic image.
  • a high-quality image in which the image quality of a tomographic image is improved is generated using another learned model, and the learned model according to the first or second embodiment is used for the high-quality image.
  • the used boundary detection and area detection are performed.
  • the image quality improvement in the present embodiment includes noise reduction, conversion of a photographing target into colors and gradations that are easy to observe, improvement in resolution and spatial resolution, and enlargement of the image size while suppressing a decrease in resolution including.
  • the image processing by the image processing system according to the present embodiment will be described focusing on the difference from the image processing according to the second embodiment.
  • the configuration and the processing procedure of the image processing system according to the present embodiment are the same as the configuration and the processing procedure of the image processing system 8 according to the second embodiment. .
  • the first processing unit 822 improves the image quality of the input image using a learned model related to a high image quality model, which is a machine learning model different from a machine learning model for detecting a retinal region.
  • the image quality improvement model is a trained model that trains a machine learning model using an arbitrary machine learning algorithm using appropriate teacher data in advance, and outputs an image with an improved image quality of an input image.
  • FIG. 20 shows an example of the teacher data of the image quality improvement model according to the present embodiment.
  • a tomographic image 2001 shows an example of a tomographic image acquired by OCT imaging
  • a tomographic image 2002 shows a tomographic image obtained by performing high quality processing on the tomographic image 2001.
  • a tomographic image 2001 shows an example of input data
  • a tomographic image 2002 shows an example of output data
  • teacher data is composed of a group of pairs composed of these images.
  • the image quality improvement processing there is a method of performing position alignment on tomographic images obtained by photographing the same position spatially a plurality of times, and performing an averaging process on the aligned tomographic images.
  • the image quality improvement processing is not limited to the averaging processing, and may be, for example, processing using a smoothing filter, maximum a posteriori probability estimation processing (MAP estimation processing), gradation conversion processing, or the like.
  • MAP estimation processing maximum a posteriori probability estimation processing
  • gradation conversion processing or the like.
  • the image subjected to the high image quality processing for example, an image which has been subjected to filter processing such as noise removal and edge emphasis may be used, or an image in which the contrast is adjusted such that a low luminance image is converted to a high luminance image. May be used.
  • the output data of the teacher data related to the high image quality model only needs to be a high image quality image
  • the image data is captured using an OCT device having higher performance than the OCT device used when capturing the tomographic image as input data.
  • the first processing unit 822 inputs a tomographic image obtained by OCT imaging to the high-quality model trained using such teacher data, and obtains a high-quality tomographic image.
  • the first processing unit 822 can acquire a high-quality volume tomographic image by inputting a tomographic image of a volume obtained by three-dimensionally scanning the retina by raster scanning into a high-quality model. .
  • the first processing unit 822 receives the high-quality image acquired using the high-quality image model, and detects the retinal region or the characteristic region using the learned model as in the second to fifth embodiments.
  • the second processing unit 823 can detect the retinal layer based on the high-quality image acquired by the first processing unit 822 and the detected retinal region or characteristic region.
  • the first processing unit 822 performs the first detection process on the tomographic image of which the image quality has been improved using the learned model.
  • the image processing device 80 can improve the image quality of the input image using the learned model of the machine learning model, and can detect the retinal layer in the image with the improved image quality. Therefore, it is possible to detect the retinal layer using an image with improved image quality such as noise reduction, and to reduce erroneous detection.
  • a process of increasing the image quality of the tomographic image, which is an input image is added to the process of the second embodiment.
  • the process of the first embodiment and the third to fifth embodiments includes the process of improving the image quality. May be added.
  • the image quality improvement model for improving the image quality is a machine learning model different from the machine learning model for performing the detection processing.
  • the machine learning model that performs the detection process may learn the high image quality process, and the machine learning model may be configured to perform both the high image quality process and the detection process.
  • the first processing unit 822 generates a high-quality image in which the image quality of the tomographic image is improved using a learned model (high-quality model) related to the high-quality process.
  • a component that generates a high-quality image using the high-quality model is not limited to the first processing unit 822.
  • a third processing unit (image quality improving unit) different from the first processing unit 822 may be provided, and the third processing unit may generate a high quality image using the image quality improving model. For this reason, the first processing unit 822 or the third processing unit generates a tomographic image having a higher image quality as compared with the tomographic image from the tomographic image using the learned model for improving the image quality.
  • the third processing unit and the high image quality model may be configured by a software module or the like executed by a processor such as a CPU, an MPU, a GPU, or an FPGA, or by a circuit or the like that performs a specific function such as an ASIC. It may be configured.
  • the first processing unit 822 performs the first detection process on the tomographic image of which the image quality has been improved using the image quality improvement model, and has detected the retinal region or the characteristic region.
  • the first processing unit 822 may perform an image quality improvement process on another image using an image quality enhancement model, and the display control unit 25 may display the various image quality enhanced images on the display unit 50. It may be displayed.
  • the first processing unit 822 may generate a luminance En-Face image or an OCTA front image generated based on retinal layer information (eg, a boundary image) detected by the first detection processing and the second detection processing.
  • the display control unit 25 can cause the display unit 50 to display at least one of the tomographic image, the luminance En-Face image, and the OCTA front image that have been subjected to the high image quality processing by the first processing unit 822.
  • the image to be displayed with high image quality may be an SLO image, a fundus image acquired by a fundus camera or the like, a fluorescent fundus image, or the like.
  • the SLO image is a front image of the fundus oculi acquired by an SLO (Scanning Laser Ophthalmoscope) optical system (not shown).
  • the learning data of the image quality improvement model for performing the image quality improvement processing on the various images is the same as the learning data of the image quality improvement model according to the sixth embodiment.
  • the image after the image quality improvement processing is used as input data and output data.
  • the image quality improvement processing on the learning data includes, for example, an averaging processing, a processing using a smoothing filter, a maximum posterior probability estimation processing (MAP estimation processing), a gradation conversion processing, and the like. It may be.
  • MAP estimation processing maximum posterior probability estimation processing
  • a gradation conversion processing and the like. It may be.
  • an image after the image quality improvement processing for example, an image on which filter processing such as noise removal and edge enhancement has been performed may be used, or an image in which contrast is adjusted such that a low-luminance image is changed to a high-luminance image. May be used.
  • the output data of the teacher data relating to the high image quality model only needs to be a high quality image
  • the image was captured using an OCT device having higher performance than the OCT device used when capturing the image as the input data.
  • the image may be an image or an image captured with a high load setting.
  • the image quality improvement model may be prepared for each type of image to be subjected to image quality improvement processing. For example, a high-quality model for a tomographic image, a high-quality model for a luminance En-Face image, and a high-quality model for an OCTA front image may be prepared. Further, the image quality enhancement model for the luminance En-Face image and the image quality enhancement model for the OCTA front image include a learning method in which images of different depth ranges are comprehensively learned for a depth range (generation range) related to image generation. Model may be used.
  • the images in different depth ranges may include, for example, images of the surface layer (Im2110), the deep layer (Im2120), the outer layer (Im2130), and the choroidal vascular network (Im1940), as shown in FIG. 21A. Further, as the image quality improvement model for the luminance En-Face image and the image quality improvement model for the OCTA front image, a plurality of image quality improvement models that have learned images for different depth ranges may be prepared.
  • a learned model that comprehensively learns tomographic images obtained at different positions in the sub-scanning (Y) direction may be used.
  • the tomographic images Im2151 to Im2153 shown in FIG. 21B are examples of tomographic images obtained at different positions in the sub-scanning direction.
  • learning may be separately performed for each part, or the imaging part may be taken care of. You may learn together.
  • the tomographic image to be improved in image quality may include a tomographic image of luminance and a tomographic image of motion contrast data.
  • learning may be separately performed as the respective image quality improvement models.
  • image processing by the image processing system according to the present embodiment will be described focusing on differences from the image processing according to the sixth embodiment.
  • the configuration and the processing procedure of the image processing system according to the present embodiment are the same as the configuration and the processing procedure of the image processing system 8 according to the sixth embodiment. .
  • the display control unit 25 displays an image on which the first processing unit 822 has performed the image quality improvement processing on the display unit 50.
  • description will be given with reference to FIGS. 22A and 22B, but the display screen is not limited to this.
  • the image quality improvement processing can be similarly applied to a display screen in which a plurality of images obtained at different dates and times are displayed side by side.
  • the image quality improvement processing can be similarly applied to a display screen in which the examiner confirms the success or failure of imaging immediately after imaging, such as an imaging confirmation screen.
  • the display control unit 25 can cause the display unit 50 to display the plurality of high-quality images generated by the first processing unit 822 and the low-quality images that have not been subjected to high-quality image processing.
  • the display control unit 25 may select a low-quality image and a high-quality image selected according to an instruction from the examiner, for a plurality of high-quality images and low-quality images that have not been subjected to the high-quality image displayed on the display unit 50. Can be displayed on the display unit 50 respectively.
  • the image processing device 80 can also output the low-quality image and the high-quality image selected according to the instruction of the examiner to the outside.
  • the display screen 2200 shows the entire screen.
  • the display screen 2200 shows a patient tab 2201, an imaging tab 2202, a report tab 2203, and a setting tab 2204.
  • a hatched line in the report tab 2203 indicates an active state of the report screen.
  • an example in which a report screen is displayed will be described.
  • the report screen shown in FIG. 22A shows an SLO image Im2205, OCTA front images Im2207 and 2208, a luminance En-Face image Im2209, tomographic images Im2211 and 2122, and a button 2220. Further, an OCTA front image Im2206 corresponding to the OCTA front image Im2207 is superimposed on the SLO image Im2205. Further, boundary lines 2213 and 2214 of the depth range of the OCTA front images Im2207 and Im2208 are superimposed on the tomographic images Im2211 and Im2212, respectively.
  • the button 2220 is a button for designating execution of the image quality improvement processing. The button 2220 may be a button for instructing display of a high-quality image, as described later.
  • the execution of the image quality improvement processing is performed by designating the button 2220, or the presence or absence of the execution is determined based on information stored (stored) in the database.
  • the button 2220 is designated according to an instruction from the examiner to switch between the display of a high-quality image and the display of a low-quality image.
  • the target image of the image quality improvement processing will be described as an OCTA front image.
  • the depth range of the OCTA front images Im2207 and Im2208 may be determined using the information of the retinal layers detected by the first detection processing and the second detection processing.
  • the depth range may be, for example, a range between two layer boundaries relating to the detected retinal layer, or may be predetermined in a deeper or shallower direction based on one of the two layer boundaries relating to the detected retinal layer. May be included in the range of the number of pixels. Further, the depth range may be, for example, a range that is changed (offset) according to an instruction of the operator from a range between two detected layer boundaries of the retinal layer.
  • the display control unit 25 displays the low-quality OCTA front images Im2207 and Im2208. Thereafter, when the examiner specifies the button 2220, the first processing unit 822 executes the image quality improvement processing on the OCTA front images Im2207 and Im2208 displayed on the screen. After the completion of the high image quality processing, the display control unit 25 displays the high quality image generated by the first processing unit 822 on the report screen. Since the OCTA front image Im2206 is obtained by superimposing and displaying the OCTA front image Im2207 on the SLO image Im2205, the display control unit 25 can also display the OCTA front image Im2206 with a high-quality image. . In addition, the display control unit 25 can change the display of the button 2220 to the active state, and display the button 2220 so that the user can know that the image quality improvement processing has been performed.
  • the execution of the process in the first processing unit 822 does not need to be limited to the timing at which the examiner specifies the button 2220. Since the types of the OCTA front images Im2207 and Im2208 to be displayed when the report screen is opened are known in advance, the first processing unit 822 executes the high image quality processing when the displayed screen transitions to the report screen. You may do. Then, at the timing when the button 2220 is pressed, the display control unit 25 may display a high-quality image on the report screen. Furthermore, there is no need for two types of images to be subjected to the image quality improvement processing in response to an instruction from the examiner or when transitioning to the report screen.
  • Processing is performed on a plurality of OCTA front images such as the surface layer (Im2110), the deep layer (Im2120), the outer layer (Im2130), and the choroidal vascular network (Im2140) as shown in FIG. 21A, which are likely to be displayed. It may be performed.
  • the image on which the image quality improvement processing has been performed may be temporarily stored in a memory or may be stored in a database.
  • the first processing unit 822 displays the high image quality image obtained by executing the image quality improvement process when the display screen transitions to the report screen. 25 is displayed on the display unit 50 by default. Then, the display control unit 25 displays the button 2220 as an active state by default so that the examiner can see that the high-quality image obtained by executing the high-quality processing is displayed. Can be configured.
  • the display control unit 25 causes the display unit 50 to display the low-quality image by specifying the button 2220 and releasing the active state. Can be.
  • the display control unit 25 displays the high-quality image on the display unit 50 again by designating the button 2220 to be in the active state. Let it.
  • the image quality improvement processing is performed on the database is specified for each layer, such as common to all the data stored in the database and for each photographing data (for each inspection). For example, when the state in which the image quality improvement processing is executed is stored for the entire database, the state in which the examiner does not execute the image quality improvement processing is stored for individual imaging data (individual examination). be able to. In this case, the individual imaging data in which the state in which the image quality improvement processing is not performed can be displayed without performing the image quality improvement processing at the next display. According to such a configuration, when it is not specified whether or not the image quality improvement processing is to be performed in units of imaging data (inspection units), it is possible to execute the processing based on the information specified for the entire database. it can. In the case where the image data is specified in units of imaging data (inspection units), it is possible to individually execute processing based on the information.
  • a user interface for example, a save button (not shown) may be used to save the execution state of the image quality improvement processing for each piece of imaging data (for each inspection).
  • the display state for example, the button 2220 Based on the (state), the state in which the image quality improving process is performed may be stored.
  • the OCTA front images Im2207 and Im2208 are displayed as the OCTA front images, but the displayed OCTA front images can be changed by the examiner's designation. Therefore, a description will be given of the change of the image to be displayed when the execution of the image quality improvement processing is specified (the button 2220 is in the active state).
  • the displayed image can be changed using a user interface (for example, a combo box) not shown.
  • a user interface for example, a combo box
  • the first processing unit 822 performs the image quality improvement processing on the choroidal vascular network image
  • the display control unit 25 performs the first image processing.
  • the high-quality image generated by the processing unit 822 is displayed on the report screen. That is, the display control unit 25 changes the display of the high-quality image in the first depth range to the high-quality image in the second depth range that is at least partially different from the first depth range in response to an instruction from the examiner.
  • the display may be changed to an image display.
  • the display control unit 25 changes the first depth range to the second depth range in response to an instruction from the examiner, thereby displaying the high-quality image in the first depth range. May be changed to display a high-quality image in a depth range of. As described above, if a high-quality image has already been generated for an image that is likely to be displayed when the report screen transitions, the display control unit 25 may display the generated high-quality image. .
  • the method of changing the type of image is not limited to the above-described method.
  • An OCTA front image in which a different depth range is set by changing a reference layer or an offset value is generated, and the generated OCTA front image is subjected to a high quality processing. Can be displayed.
  • the first processing unit 822 executes the high image quality processing on an arbitrary OCTA front image, and the display control unit 25 reports the high image image. Display on the screen.
  • the reference layer and the offset value can be changed using a user interface (not shown) (for example, a combo box or a text box).
  • the depth range (generation range) of the OCTA front image can be changed by dragging (moving the layer boundary) any of the boundaries 2213 and 2214 superimposed on the tomographic images Im2211 and Im2212. .
  • the first processing unit 822 may always process the execution instruction, or may execute the processing after the layer boundary is changed by dragging. Alternatively, the execution of the image quality improvement processing is continuously instructed, but when the next instruction comes, the previous instruction may be canceled and the latest instruction may be executed.
  • the high-quality processing may take a relatively long time. Therefore, even if the command is executed at any timing described above, it may take a relatively long time before a high-quality image is displayed. Therefore, after a depth range for generating an OCTA front image according to an instruction from the examiner is set and before a high-quality image is displayed, a low-quality OCTA corresponding to the set depth range is displayed. A front image (low-quality image) may be displayed. That is, when the above-described depth range is set, a low-quality OCTA front image (low-quality image) corresponding to the set depth range is displayed. May be changed to display a high-quality image.
  • information indicating that the high-quality image processing is being performed may be displayed from when the depth range is set until the high-quality image is displayed.
  • these processes are not limited to the configuration applied when it is assumed that the execution of the image quality improvement process has already been specified (the button 2220 is in the active state). For example, when the execution of the image quality improvement processing is instructed in response to the instruction from the examiner, these processings can be applied until the high quality image is displayed.
  • the OCTA front images Im2207 and 2108 relating to different layers are displayed as the OCTA front images, and the low-quality and high-quality images are switched and displayed.
  • the displayed images are not limited thereto.
  • a low-quality OCTA front image may be displayed side by side as the OCTA front image Im2207
  • a high-quality OCTA front image may be displayed as the OCTA front image Im2208.
  • the images are switched and displayed, the images are switched at the same place, so that it is easy to compare the changed portions.
  • the images can be displayed at the same time, so that the entire image can be easily compared.
  • FIG. 22B is a screen example in which the OCTA front image Im2207 in FIG. 22A is enlarged and displayed. Also in FIG. 22B, a button 2220 is displayed as in FIG. 22A.
  • the screen transition from FIG. 22A to FIG. 22B is made, for example, by double-clicking on the OCTA front image Im2207, and is made with the close button 2230 from FIG. 22B to FIG. 22A. Note that the screen transition is not limited to the method shown here, and a user interface (not shown) may be used.
  • the button 2220 When the execution of the image quality improvement processing is designated at the time of the screen transition (the button 2220 is active), the state is maintained even at the time of the screen transition. That is, in a case where a transition is made to the screen of FIG. 22B while the high-quality image is being displayed on the screen of FIG. 22A, the high-quality image is also displayed on the screen of FIG. 22B. Then, the button 2220 is activated. The same applies to the transition from FIG. 22B to FIG. 22B. In FIG. 22B, the display can be switched to a low-quality image by designating a button 2220.
  • the display state of the high-quality image is maintained.
  • the transition can be performed as it is. That is, an image corresponding to the state of the button 2220 on the display screen before the transition can be displayed on the display screen after the transition. For example, if the button 2220 on the display screen before the transition is in the active state, a high-quality image is displayed on the display screen after the transition. Further, for example, if the active state of the button 2220 on the display screen before the transition is released, a low image quality image is displayed on the display screen after the transition.
  • buttons 2220 on the display screen for follow-up observation When the button 2220 on the display screen for follow-up observation is activated, a plurality of images obtained at different dates and times (different inspection dates) displayed side by side on the display screen for follow-up observation are switched to high-quality images. You may. That is, when the button 2220 on the display screen for follow-up observation becomes active, it may be configured to be reflected collectively on a plurality of images obtained at different dates and times.
  • FIG. 23 shows an example of a display screen for follow-up observation.
  • a display screen for follow-up observation is displayed as shown in FIG.
  • the depth range of the OCTA front image can be changed by selecting a desired set from the predetermined depth range sets displayed in the list boxes 2302 and 2303.
  • the retina surface layer is selected in the list box 2302
  • the retina deep layer is selected in the list box 2303.
  • the analysis result of the OCTA front image of the retinal surface is displayed in the upper display area
  • the analysis result of the OCTA front image of the deep retina is displayed in the lower display area.
  • the display of the analysis result is set to the non-selection state, the display may be changed to a parallel display of a plurality of OCTA front images at different dates and times.
  • the button 2220 is designated in response to an instruction from the examiner, the display of a plurality of OCTA front images is changed to the display of a plurality of high-quality images at once.
  • the display of the analysis result When the display of the analysis result is in the selected state, when the button 2220 is designated in response to the instruction from the examiner, the display of the analysis result of the plurality of OCTA front images is changed to the analysis result of the plurality of high-quality images. Will be changed to the display.
  • the display of the analysis result may be a display in which the analysis result is superimposed on the image with an arbitrary transparency.
  • the change from the display of the image to the display of the analysis result may be, for example, a change in a state in which the analysis result is superimposed on the displayed image with arbitrary transparency.
  • the change from the display of the image to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency. Good.
  • the type and offset position of the layer boundary used to specify the depth range can be changed collectively from the user interfaces 2305 and 2306.
  • the user interfaces 2305 and 2306 for changing the type of layer boundary and the offset position are merely examples, and an interface of any other mode may be used.
  • the depth range of a plurality of OCTA front images at different dates and times is collectively changed by displaying the tomographic image together and moving the layer boundary data superimposed on the tomographic image in accordance with an instruction from the examiner. May be. At this time, when a plurality of tomographic images at different dates and times are displayed side by side and the above-described movement is performed on one tomographic image, the layer boundary data may be similarly moved on other tomographic images.
  • the presence or absence of the image projection method or the projection artifact suppression process may be changed by selecting the user interface such as a context menu, for example.
  • the selection button 2307 may be selected to display a selection screen (not shown), and the image selected from the image list displayed on the selection screen may be displayed.
  • an arrow 2304 displayed at the top of FIG. 23 is a mark indicating that the test is currently selected
  • the reference test Baseline
  • a mark indicating the reference inspection may be displayed on the display unit.
  • the measured value distribution (map or sector map) for the reference image is displayed on the reference image. Further, in this case, a difference measurement value map between the measurement value distribution calculated for the reference image and the measurement value distribution calculated for the image displayed in the region is provided in an area corresponding to the other inspection date. Is displayed.
  • a trend graph (a graph of the measurement values for the images on each inspection day obtained by the measurement over time) may be displayed on the report screen. That is, time-series data (for example, a time-series graph) of a plurality of analysis results corresponding to a plurality of images at different dates and times may be displayed.
  • the analysis results regarding the dates and times other than the multiple dates and times corresponding to the multiple displayed images are also distinguished from the multiple analysis results corresponding to the multiple displayed images (for example, time-series (The color of each point on the graph differs depending on whether an image is displayed or not). Further, a regression line (curve) of the trend graph and a corresponding mathematical expression may be displayed on a report screen.
  • the OCTA front image has been described, but the image to which the processing according to the present embodiment is applied is not limited to this.
  • An image related to processing such as display, image quality improvement, and image analysis according to the present embodiment may be an En-Face image of luminance.
  • the En-Face image not only the En-Face image but also a different image such as a B-scan tomographic image, SLO image, fundus image, or fluorescent fundus image may be used.
  • the user interface for executing the image quality improvement processing is to instruct execution of the image quality improvement processing for a plurality of different types of images, or to select an arbitrary image from the plurality of types of the different images. There may be one that instructs execution of the high image quality processing.
  • the tomographic images Im2211 and Im2212 shown in FIG. 22A may be displayed with high image quality.
  • a high-quality tomographic image may be displayed in an area where the OCTA front images Im2207 and Im2208 are displayed.
  • only one tomographic image to be displayed with high image quality may be displayed, or a plurality of tomographic images may be displayed.
  • tomographic images acquired at different positions in the sub-scanning direction may be displayed, or a plurality of tomographic images obtained by, for example, a cross scan may be displayed with high image quality.
  • images in different scanning directions may be displayed.
  • a plurality of tomographic images partially selected (for example, two tomographic images at positions symmetrical to each other with respect to a reference line). ) May be displayed. Further, a plurality of tomographic images are displayed on a display screen for follow-up observation as shown in FIG. 23, and an instruction for higher image quality or an analysis result (for example, the thickness of a specific layer, etc. ) May be displayed. Further, the image quality improvement processing may be performed on the tomographic image based on the information stored in the database by a method similar to the above-described method.
  • the SLO image Im2205 may be displayed with high image quality.
  • the luminance En-Face image 2209 may be displayed with high image quality.
  • a plurality of SLO images and En-Face images of luminance are displayed on the display screen for follow-up observation as shown in FIG. 23, and an instruction for higher image quality or an analysis result (for example, The display of the thickness of a specific layer, etc.) may be performed.
  • the image quality improvement processing may be performed on the SLO image or the luminance En-Face image based on the information stored in the database by the same method as the method described above.
  • the display of the tomographic image, the SLO image, and the luminance En-Face image is merely an example, and these images may be displayed in any manner according to a desired configuration. Further, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the luminance En-Face image may be displayed with high image quality by a single instruction.
  • the display control unit 25 can display the image on which the first processing unit 822 according to the present embodiment has performed the image quality improvement processing on the display unit 50.
  • the display screen is displayed. May be transited, or the selected state may be maintained.
  • the display control unit 25 changes the display of the analysis result of the low image quality image to high according to an instruction from the examiner (for example, when the button 2220 is designated). The display may be changed to a display of the analysis result of the image quality image.
  • the display control unit 25 displays the analysis result of the high-quality image in response to an instruction from the examiner (for example, when the designation of the button 2220 is released). May be changed to the display of the analysis result of the low-quality image.
  • the display control unit 25 responds to an instruction from the examiner (for example, when the display of the analysis result is released), and The display of the analysis result may be changed to the display of a low-quality image.
  • the display control unit 25 displays the low-quality image in response to an instruction from the examiner (for example, when the display of the analysis result is specified). The display of the analysis result of the low-quality image may be changed.
  • the display control unit 25 analyzes the high-quality image in response to an instruction from the examiner (for example, when the display of the analysis result is released). The display of the result may be changed to the display of a high-quality image.
  • the display control unit 25 increases the display of the high-quality image according to the instruction from the examiner (for example, when the display of the analysis result is specified). The display may be changed to a display of the analysis result of the image quality image.
  • the display control unit 25 converts the first type of analysis result of the low-quality image into an image.
  • the display may be changed to the display of the second type of analysis result of the low image quality image. It is also assumed that the display of the high-quality image is in the selected state and the display of the first type of analysis result is in the selected state.
  • the display control unit 25 responds to an instruction from the examiner (for example, when the display of the second type of analysis result is specified), the first type of analysis result of the high-quality image is displayed.
  • the display may be changed to the display of the second type of analysis result of the high quality image.
  • the display screen for follow-up observation may be configured so that these display changes are collectively reflected on a plurality of images obtained at different dates and times.
  • the display of the analysis result may be a display in which the analysis result is superimposed on the image with an arbitrary transparency.
  • the display of the analysis result may be changed, for example, to a state where the analysis result is superimposed on the displayed image with arbitrary transparency.
  • the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.
  • the first processing unit 822 generates a high-quality image in which the image quality of the tomographic image is improved using a learned model (high-quality model) related to the high-quality process.
  • a component that generates a high-quality image using the high-quality model is not limited to the first processing unit 822.
  • a third processing unit (image quality improving unit) different from the first processing unit 822 may be provided, and the third processing unit may generate a high quality image using the image quality improving model.
  • the third processing unit and the high image quality model may be configured by a software module or the like executed by a processor such as a CPU, an MPU, a GPU, or an FPGA, or a circuit that performs a specific function such as an ASIC. May be configured.
  • the display control unit 25 displays, on the display unit 50, an image selected from the high-quality image and the input image generated by the first processing unit 822 in accordance with an instruction from the examiner. Can be done.
  • the display control unit 25 may switch the display on the display unit 50 from a captured image (input image) to a high-quality image according to an instruction from the examiner. That is, the display control unit 25 may change the display of the low-quality image to the display of the high-quality image according to an instruction from the examiner.
  • the display control unit 25 may change the display of the high-quality image to the display of the low-quality image according to an instruction from the examiner.
  • the first processing unit 822 executes the start of image quality improvement processing (input of an image to the image quality improvement engine) by the image quality improvement engine (image quality improvement model) in accordance with an instruction from the examiner,
  • the display control unit 25 may cause the display unit 50 to display the generated high-quality image.
  • the image quality improvement engine includes a learned model that performs the above-described image quality improvement processing (image quality improvement processing).
  • the display control unit 25 may change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image according to an instruction from the examiner. In addition, the display control unit 25 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image according to an instruction from the examiner. Further, the display control unit 25 may change the display of the analysis result of the low-quality image to the display of the low-quality image according to an instruction from the examiner. In addition, the display control unit 25 may change the display of the low-quality image to the display of the analysis result of the low-quality image according to an instruction from the examiner.
  • the display control unit 25 may change the display of the analysis result of the high-quality image to the display of the high-quality image according to an instruction from the examiner.
  • the display control unit 25 may change the display of the high-quality image to the display of the analysis result of the high-quality image according to an instruction from the examiner.
  • the display control unit 25 may change the display of the analysis result of the low-quality image to the display of another type of analysis result of the low-quality image according to an instruction from the examiner.
  • the display control unit 25 may change the display of the analysis result of the high-quality image to the display of another type of analysis result of the high-quality image according to an instruction from the examiner.
  • the analysis result of the high-quality image may be displayed by superimposing and displaying the analysis result of the high-quality image on the high-quality image with any transparency.
  • the display of the analysis result of the low-quality image may be a display in which the analysis result of the low-quality image is superimposed and displayed on the low-quality image with arbitrary transparency.
  • the display of the analysis result may be changed, for example, to a state where the analysis result is superimposed on the displayed image with arbitrary transparency.
  • the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.
  • the first processing unit 822 generates a high-quality image in which the image quality of the tomographic image is improved by using a learned model (high-quality model) related to the high-quality processing.
  • a component that generates a high-quality image using the high-quality model is not limited to the first processing unit 822.
  • a third processing unit different from the first processing unit 822 may be provided, and the third processing unit may generate a high quality image using the high quality image model.
  • the third processing unit and the high image quality model may be configured by a software module or the like executed by a processor such as a CPU, an MPU, a GPU, or an FPGA, or a circuit that performs a specific function such as an ASIC. May be configured.
  • the image on which the image quality improvement processing using the image quality enhancement model has been performed is displayed according to the active state of the button 2220 on the display screen.
  • an analysis value using the result of the image segmentation processing using the learned model may be displayed.
  • the display control unit 25 executes the image segmentation process performed by the second processing unit 823.
  • An analysis result using the result is displayed on the display unit 50.
  • the display control unit 25 performs the image segmentation processing performed by the first processing unit 822 alone or by the first processing unit 822 and the second processing unit 823. An analysis result using the result is displayed on the display unit 50.
  • the analysis result using the result of the image segmentation processing using the learned model and the analysis result using the result of the image segmentation processing using the learned model are switched according to the active state of the button. Is displayed. Since these analysis results are based on the results of the processing by the learned model and the image processing by the rule base, there may be a difference between the two results. Therefore, by switching and displaying these analysis results, the examiner can compare the two and use the analysis results that are more satisfactory for the diagnosis.
  • the image segmentation process when the displayed image is a tomographic image, the numerical value of the layer thickness analyzed for each layer may be switched and displayed. Further, for example, when a tomographic image divided by a color, a hatching pattern, or the like is displayed for each layer, a tomographic image in which the shape of the layer is changed according to the result of the image segmentation processing may be switched and displayed. . Furthermore, when a thickness map is displayed as an analysis result, a thickness map in which the color indicating the thickness has changed according to the result of the image segmentation process may be displayed. Further, a button for specifying the image quality improvement processing and a button for specifying the image segmentation processing using the learned model may be provided separately, one of them may be provided, and both buttons may be provided. It may be provided as one button.
  • the switching of the image segmentation process may be performed based on information stored (recorded) in the database, similarly to the switching of the image quality improvement process described above. It should be noted that the switching of the image segmentation process may be performed in the same manner as the above-described switching of the image quality improvement process also at the time of the screen transition.
  • the obtaining unit 21 obtains the interference signal obtained by the OCT apparatus 10 and the three-dimensional tomographic data generated by the tomographic image generating unit 221.
  • the configuration in which the acquisition unit 21 acquires these signals and data is not limited to this.
  • the acquisition unit 21 may acquire these signals from a server or an imaging device connected to the image processing apparatus 20 via a LAN, a WAN, the Internet, or the like. In this case, it is possible to omit the processing related to imaging, and acquire three-dimensional tomographic data that has been captured. Then, the boundary detection processing can be performed in step S304, step S904, or the like. Therefore, a series of processing times from acquisition of tomographic information to display of a front image, a thickness map, and the like can be shortened.
  • the learned model for image segmentation and the learned model for improving image quality used by the processing unit 222 and the first processing unit 822 can be provided in the image processing devices 20, 80, and 152.
  • the learned model may be configured by, for example, a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may be configured by a circuit that performs a specific function such as an ASIC.
  • these learned models may be provided in a device of another server or the like connected to the image processing devices 20, 80, 152.
  • the image processing apparatuses 20, 80, and 152 can use the learned model by connecting to a server or the like having the learned model via an arbitrary network such as the Internet.
  • the server including the learned model may be, for example, a cloud server, a fog server, an edge server, or the like.
  • a label image labeled for each pixel has been described as a label image, but a label image labeled for each region may be used as a label image.
  • the configuration of the OCT apparatus 10 is not limited to the above configuration, and a part of the configuration included in the OCT apparatus 10 may be configured separately from the OCT apparatus 10.
  • the layout of the user interface such as buttons and the display are not limited to those described above.
  • Example 8 Description of the Related Art
  • image diagnosis using images acquired by various imaging devices is performed to identify a disease of a subject or to observe the degree of the disease.
  • Examples of the type of imaging apparatus include, for example, an X-ray imaging apparatus, an X-ray computed tomography apparatus (CT), a magnetic resonance imaging apparatus (MRI), and a positron emission tomography apparatus (PET) in the field of radiology.
  • CT X-ray computed tomography
  • MRI magnetic resonance imaging apparatus
  • PET positron emission tomography apparatus
  • ophthalmology for example, there are a fundus camera, a scanning laser ophthalmoscope (SLO), an optical coherence tomography (OCT) device, an OCT angiography (OCTA) device, and the like.
  • SLO scanning laser ophthalmoscope
  • OCT optical coherence tomography
  • OCTA OCT angiography
  • Image diagnosis is basically performed by medical staff observing lesions and the like drawn in the image, but in recent years, various information useful for diagnosis has been obtained by improving image analysis technology. .
  • image analysis helps detect small lesions that may be overlooked, assists healthcare professionals, performs quantitative measurements on the shape and volume of lesions, and does not require observation by healthcare professionals And to identify the disease.
  • image segmentation processing for identifying a region such as an organ or a lesion depicted in an image is performed when performing many image analysis methods. This is a necessary process.
  • image segmentation process is also simply referred to as a segmentation process for simplification.
  • the conventional image segmentation process is performed by an image processing algorithm based on medical knowledge and image characteristics regarding a target organ or lesion, as disclosed in Patent Document 1.
  • an image acquired from an imaging device may not be able to be captured neatly due to various factors such as the condition of the subject, the imaging environment of the imaging device, lack of skill of the photographer, and the like. . Therefore, in the conventional image segmentation processing, a target organ or lesion is not depicted as expected, and a specific region may not be accurately extracted.
  • the shape of the retinal layer is irregularly drawn. There was something. In such a case, erroneous detection may occur in the retinal layer area detection processing, which is a type of image segmentation processing.
  • One of the objects of the following eighth to nineteenth embodiments is to provide a medical image processing apparatus, a medical image processing method, and a program capable of performing image segmentation processing with higher accuracy than conventional image segmentation processing.
  • each device may be connected by a wired or wireless line.
  • a line connecting each device in the network includes, for example, a dedicated line, a local area network (hereinafter referred to as LAN) line, a wireless LAN line, an Internet line, Wi-Fi (registered trademark), and Bluetooth (registered trademark). ).
  • LAN local area network
  • Wi-Fi registered trademark
  • Bluetooth registered trademark
  • the medical image processing device may be configured by two or more devices that can communicate with each other, or may be configured by a single device.
  • Each component of the medical image processing apparatus may be configured by a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA.
  • a processor such as a CPU, an MPU, a GPU, and an FPGA.
  • each of the components may be configured by a circuit or the like that performs a specific function such as an ASIC. Also, it may be configured by a combination of any other hardware and any software.
  • the medical image processed by the medical image processing apparatus or the medical image processing method is a tomographic image of the subject acquired using the OCT apparatus.
  • the OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device.
  • the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a wavelength sweep type OCT (SS-OCT) device.
  • a wavefront compensation OCT (AO-OCT) apparatus using a wavefront compensation optical system a line OCT apparatus in which measurement light irradiated to a subject is formed in a line, and the measurement light is formed in a plane
  • Any OCT device, such as a formed full-field OCT device, may be included.
  • the medical image includes a tomographic image of the subject's eye (the subject's eye).
  • the tomographic image of the subject's eye is not limited to a tomographic image of the retina and the like in the posterior segment of the subject's eye, and includes a tomographic image of the anterior eye of the subject's eye and the eye chamber.
  • a tomographic image of the skin or the organ of the subject may be used as a medical image to be processed by the medical image processing apparatus or the medical image processing method according to the following embodiments. .
  • the image management system is a device and a system that receives and stores an image captured by an imaging device such as an OCT device or an image processed image.
  • the image management system can transmit an image in response to a request from a connected device, perform image processing on a stored image, and request an image processing request to another device. It can.
  • the image management system can include, for example, an image storage and communication system (PACS).
  • PACS image storage and communication system
  • the image management system according to the embodiment described below includes a database that can store various information such as information on a subject and an imaging time associated with the received image.
  • the image management system is connected to a network, and can transmit and receive images, convert images, and transmit and receive various types of information associated with stored images in response to requests from other devices. .
  • the photographing conditions are various information at the time of photographing the image acquired by the photographing device.
  • the imaging conditions include, for example, information on the imaging device, information on the facility where the imaging was performed, information on the inspection related to the imaging, information on the photographer, information on the subject, and the like.
  • the imaging conditions include, for example, information on the date and time of imaging, the name of the imaging region, the imaging region, the imaging angle of view, the imaging method, the resolution and gradation of the image, the image size, the applied image filter, and the data format of the image.
  • the imaging region may include a peripheral region shifted from a specific imaging region, an region including a plurality of imaging regions, and the like.
  • the imaging method may include any OCT imaging method such as a spectral domain method or a wavelength sweep method.
  • the shooting conditions can be stored in a data structure constituting the image, stored as shooting condition data different from the image, or stored in a database or an image management system related to the shooting device. Therefore, the photographing condition can be acquired by a procedure corresponding to the photographing condition storage unit of the photographing device.
  • the shooting conditions are, for example, to analyze the data structure of an image output by the shooting device, to obtain shooting condition data corresponding to the image, and to obtain shooting conditions from a database related to the shooting device. It is obtained by accessing the interface of.
  • Some shooting conditions may not be available for some shooting devices because they are not stored.
  • the photographing apparatus does not have a function of acquiring or storing a specific photographing condition, or such a function is invalidated.
  • the photographing condition is not stored because the photographing condition is not related to the photographing device or photographing.
  • the shooting conditions are hidden, encrypted, or cannot be obtained without the right.
  • it may be possible to acquire even shooting conditions that are not stored. For example, by performing image analysis, it is possible to specify an imaging part name and an imaging region.
  • a region label image refers to a label image in which a region is labeled for each pixel.
  • an arbitrary region is divided by a group of pixel values (hereinafter, region label values) that can be specified.
  • Image Im2420 the specified arbitrary region includes a region of interest (ROI: Region ⁇ Interest), a volume of interest (VOI: Volume ⁇ Of ⁇ Interest), and the like.
  • the coordinate group of the pixel having an arbitrary region label value is specified from the image Im2420
  • the coordinate group of the pixel depicting the corresponding region such as the retina layer in the image Im2410
  • the region label value indicating the ganglion cell layer constituting the retina is 1
  • a coordinate group whose pixel value is 1 is specified from the pixel group of the image Im2420
  • the coordinate group is determined from the image Im2410. Extract a pixel group corresponding to the group.
  • the area of the ganglion cell layer in the image Im2410 can be specified.
  • some embodiments include a process of performing a reduction or enlargement process on an area label image.
  • the image complement processing method used for reducing or enlarging the region label image uses a nearest neighbor method or the like that does not erroneously generate an undefined region label value or a region label value that should not exist at the corresponding coordinates. Shall be.
  • the image segmentation process is a process of specifying an area called an ROI or a VOI, such as an organ or a lesion depicted in an image, for use in image diagnosis or image analysis.
  • an ROI or a VOI such as an organ or a lesion depicted in an image
  • the number of specified regions is 0 if no region to be specified is depicted in the image. Further, if a plurality of region groups to be specified are depicted in the image, the number of specified regions may be plural or one region surrounding the region group may be included. Good.
  • the specified area group is output as information that can be used in other processes.
  • a coordinate group of a pixel group constituting each of the specified region groups can be output as a numerical data group.
  • a coordinate group indicating a rectangular area, an elliptical area, a rectangular parallelepiped area, an elliptical area, and the like including each of the specified area groups can be output as a numerical data group.
  • a coordinate group indicating a straight line, a curve, a plane, a curved surface, or the like, which is a boundary of the specified region group can be output as a numerical data group.
  • an area label image indicating the specified area group can be output.
  • a region label-less image is a type of region label image, and is a region label image that does not include information corresponding to ROIs and VOIs used for image diagnosis and image analysis. Specifically, as an example, a case will be described in which it is desired to know a region of a ganglion cell layer constituting a retina depicted in a medical image for use in image analysis.
  • the region label value indicating the region of the ganglion cell layer is 1, and the region label value indicating the other region is 0.
  • the region label image is an image without a region label because there is no region having a region label value 1 corresponding to the ROI of the ganglion cell layer to be used for image analysis in the region label image.
  • the image without the region label may be a numerical data group or the like indicating a coordinate group having the same information as the image instead of the image.
  • the machine learning model refers to a learning model based on a machine learning algorithm.
  • Specific algorithms for machine learning include a nearest neighbor method, a naive Bayes method, a decision tree, a support vector machine, and the like.
  • deep learning deep learning in which a feature amount for learning and a connection weighting coefficient are generated by themselves using a neural network is also included.
  • any of the above algorithms that can be used can be applied to the learning model according to the embodiment.
  • the learned model is a model in which training (learning) has been performed in advance on a machine learning model according to an arbitrary machine learning algorithm using appropriate teacher data (learning data). However, it is assumed that the learned model does not perform any further learning and can perform additional learning.
  • the teacher data is composed of one or more pairs of input data and output data.
  • the format and combination of input data and output data of a pair group that constitutes teacher data may be such that one is an image and the other is a numerical value, one is a plurality of image groups and the other is a character string, May be an image suitable for a desired configuration.
  • teacher data configured by a pair group of an image acquired by OCT and an imaging part label corresponding to the image.
  • the imaging region label is a unique numerical value or character string representing the region.
  • teacher data constituted by a pair group of an image acquired by OCT imaging of the posterior segment of the eye and an area label image of a retinal layer corresponding to the image.
  • second teacher data a pair group of a low-quality image with a lot of noise obtained by normal imaging of OCT and a high-quality image obtained by performing multiple image capturing and high-quality processing by OCT is configured.
  • Teacher data hereinafter, third teacher data).
  • output data according to the design of the learned model is output.
  • the trained model outputs output data having a high possibility of corresponding to the input data, for example, according to the tendency trained using the teacher data. Further, the trained model can output, for example, as a numerical value the possibility corresponding to the input data for each type of output data according to the tendency trained using the teacher data.
  • the trained model when an image acquired by OCT is input to a trained model trained with the first teacher data, the trained model outputs an imaging region label of the imaging region photographed in the image. Or output the probability for each radiographic part label. Further, for example, when an image depicting a retinal layer obtained by OCT imaging of the posterior segment is input to a learned model trained by the second teacher data, the learned model is rendered in the image. An area label image for the retinal layer is output. Further, for example, when a low-quality image with much noise acquired by normal OCT imaging is input to the learned model trained with the third teacher data, the learned model is photographed a plurality of times by OCT to improve the image quality. A high quality image equivalent to the processed image is output.
  • the machine learning algorithm includes a technique related to deep learning such as a convolutional neural network (CNN).
  • CNN convolutional neural network
  • the probability of outputting a correct radiographed part label may be higher.
  • a more appropriate parameter is set, a more accurate region label image may be output in some cases.
  • a higher quality image may be output in some cases.
  • the parameters in the CNN are, for example, the filter kernel size, the number of filters, the stride value, and the dilation value set for the convolutional layer, and the number of nodes output from the fully connected layer. Etc. can be included.
  • the parameter group and the number of training epochs can be set to values suitable for the use form of the learned model based on the teacher data. For example, based on teacher data, set a parameter group and epoch number that can output a correct radiographed part label with a higher probability, output a more accurate area label image, and output a higher quality image can do.
  • ⁇ ⁇ ⁇ ⁇
  • 70% of the pair group constituting the teacher data is used for training, and the remaining 30% is randomly set for evaluation.
  • the trained model is trained using the training pair group, and at the end of each training epoch, a training evaluation value is calculated using the evaluation pair group.
  • the training evaluation value is, for example, an average value of a group of values obtained by evaluating an output when input data constituting each pair is input to a trained model being trained and output data corresponding to the input data using a loss function. It is.
  • the parameter group and the number of epochs when the training evaluation value becomes the smallest are determined as the parameter group and the number of epochs of the learned model. In this way, by deciding the number of epochs by dividing the pair group constituting the teacher data into the one for training and the other for evaluation, the trained model may overlearn the pair group for training. Can be prevented.
  • the image segmentation engine is a module that performs image segmentation processing and outputs an area label image corresponding to the input image that has been input.
  • the input image include an OCT B-scan image and a three-dimensional tomographic image (three-dimensional OCT volume image).
  • the region label image include a region label image indicating each layer of the retinal layer when the input image is an OCT B-scan image and a layer label image indicating each layer of the retinal layer when the input image is a three-dimensional tomographic image of OCT.
  • the image processing method constituting the image segmentation processing method in the following embodiments, processing using a learned model according to various machine learning algorithms such as deep learning is performed.
  • the image processing method may be performed not only with a machine learning algorithm but also with other existing arbitrary processing.
  • the image processing includes, for example, various image filtering processes, a matching process using a database of region label images corresponding to similar images, an image registration process of a reference region label image, and a knowledge base image process. .
  • a configuration 2500 shown in FIG. 25 is an example of a convolutional neural network (CNN) that generates a region label image Im2520 by performing image segmentation processing on a two-dimensional image Im2510 input as an input image.
  • the configuration 2500 of the CNN includes a plurality of layer groups that are responsible for processing the input value group and outputting the processed value group.
  • the types of layers included in the configuration 2500 include a convolution layer, a downsampling (Downsampling) layer, an upsampling (Upsampling) layer, and a synthesis (Merger) layer.
  • the CNN configuration 2500 used in the present embodiment is a U-net type machine learning model, like the CNN configuration 601 described in the first embodiment.
  • the convolution layer is a layer that performs a convolution process on an input value group according to parameters such as the set filter kernel size, number of filters, stride value, and dilation value.
  • the downsampling layer is a layer that performs processing to reduce the number of output value groups from the number of input value groups by thinning out or combining input value groups. As a process performed in the downsampling layer, specifically, for example, there is a Max @ Pooling process.
  • the upsampling layer is a layer that performs processing for increasing the number of output value groups beyond the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group.
  • the processing performed in the upsampling layer specifically, for example, there is a linear interpolation processing.
  • the synthesis layer is a layer that performs processing of inputting a value group such as an output value group of a certain layer or a pixel value group forming an image from a plurality of sources, and connecting or adding them to synthesize.
  • parameters set in the convolutional layer group included in the configuration of the CNN for example, by setting the kernel size of the filter to 3 pixels in width, 3 pixels in height, and 64 to the number of filters, an image with constant accuracy can be obtained. Segmentation processing is possible.
  • the parameter setting for the layer group or the node group forming the neural network is different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may be different. That is, in many cases, appropriate parameters for each layer group and each node group differ depending on the embodiment, and may be changed as necessary.
  • not only the method of changing the parameters as described above but also a change in the configuration of the CNN may provide the CNN with better characteristics.
  • the better characteristics include, for example, a higher accuracy of the image segmentation process, a shorter time for the image segmentation process, a shorter time for training the learned model, and the like.
  • a batch normalization (Batch @ Normalization) layer or an activation layer using a normalized linear function (Rectifier @ Linear @ Unit) is incorporated after the convolutional layer.
  • machine learning model used by the image segmentation engine for example, FCN or SegNet can be used as in the first embodiment. Further, according to a desired configuration, a machine learning model that performs object recognition in units of regions as described in the first embodiment may be used.
  • the filter kernel size may correspond to one-dimensional, three-dimensional, or four-dimensional.
  • the four-dimensional image includes, for example, a three-dimensional moving image or an image in which parameters at each pixel position of the three-dimensional image are indicated by different hues.
  • the image segmentation processing may be performed by only one image processing method, or may be performed by combining two or more image processing methods. Furthermore, a plurality of image segmentation processing methods can be performed to generate a plurality of region label images.
  • the input image is divided into small area groups, image segmentation processing is performed on each of them to obtain small area area label images, and the small area area label images are combined.
  • image segmentation processing is performed on each of them to obtain small area area label images
  • the small area area label images are combined.
  • parameters may be input to the image segmentation engine together with the input image.
  • the input parameters may include, for example, a parameter that specifies a degree of a range in which the image segmentation process is performed, such as an upper limit of a lesion size, and a parameter that specifies an image filter size used in an image processing method. it can.
  • the image segmentation engine may output another image or a coordinate data group capable of specifying an area instead of the area label image in some embodiments.
  • the processing time can be reduced by performing the image segmentation processing in parallel.
  • the input image and the output region label image may require different image sizes in order to prevent a problem such as a problem that the peripheral portion of the region label image is not segmented with sufficient accuracy. It is.
  • the image size may be appropriately adjusted. It has been adjusted. More specifically, padding is performed on an input image such as an image used as teacher data for training a trained model or an image input to an image segmentation engine, or a shooting area around the input image is combined. Or adjust the image size.
  • the area to be padded is filled with a fixed pixel value, filled with neighboring pixel values, or mirror-padded according to the characteristics of the image segmentation processing method so that the image segmentation processing can be performed effectively.
  • the shooting location estimation engine is a module that estimates a shooting site and a shooting area of an input image.
  • the imaging location estimation engine determines the location of the imaging region or imaging region drawn in the input image, or the probability of being the imaging region or imaging region for each imaging region label or imaging region label of a required level of detail. Can be output.
  • the imaging region and the imaging region may not be stored as the imaging conditions depending on the imaging device, or may not be able to be acquired and stored by the imaging device. Further, even when the imaging region and the imaging region are stored, the imaging region and the imaging region at the necessary detailed level may not be stored. For example, only the “posterior segment” is stored as the imaging region, and in detail, whether it is the “macular region”, the “optic nerve head”, or the “macular and optic nerve head”, Others may not know what it is. Also, in another example, it is only stored as “breast” as an imaging part, and it is not known in detail whether it is “right breast”, “left breast”, or “both”. is there. Therefore, by using the imaging location estimation engine, it is possible to estimate the imaging site and the imaging area of the input image in these cases.
  • the image and data processing method that constitutes the estimation method of the shooting location estimation engine, processing using a learned model according to various machine learning algorithms such as deep learning is performed.
  • any known known processing such as natural language processing, matching processing using a database of similar images and similar data, knowledge base processing, and the like.
  • An estimation process may be performed.
  • the teacher data for training the trained model constructed using the machine learning algorithm may be an image to which a label of an imaging region or an imaging region is attached. In this case, regarding the teacher data, an image for estimating an imaging region or an imaging region is set as input data, and a label of the imaging region or the shooting region is set as output data.
  • the configuration 2600 of the CNN includes a plurality of convolution processing blocks 2620 each including a convolution layer 2621, a batch normalization layer 2622, and an activation layer 2623 using a normalized linear function.
  • the configuration 2600 of the CNN includes a last convolution layer 2630, a full connection layer 2640, and an output layer 2650.
  • the full coupling layer 2640 fully couples the output value group of the convolution processing block 2620.
  • the output layer 2650 outputs a probability of each assumed imaging region label with respect to the input image Im2610 using the Softmax function as an estimation result 2660 (Result).
  • the input image Im2610 is an image obtained by imaging the “macula”
  • the highest probability is output for the imaging region label corresponding to the “macula”.
  • the filter kernel size is 3 pixels in width, 3 pixels in height, and the number of filters is 64, the constant accuracy is maintained. Can be used to estimate the imaging region. However, in practice, as described in the description of the learned model, a better parameter group can be set using the teacher data according to the use form of the learned model. When it is necessary to process a one-dimensional image, a three-dimensional image, or a four-dimensional image, the filter kernel size may be extended to one-dimensional, three-dimensional, or four-dimensional. Note that the estimation method may be performed using only one image and data processing method, or may be performed using a combination of two or more image and data processing methods.
  • the area label image evaluation engine is a module that evaluates whether or not an input area label image is likely to be subjected to image segmentation processing. Specifically, the area label image evaluation engine outputs, as an image evaluation index, a true value if the input area label image is likely, and outputs a false value otherwise.
  • Examples of the method for performing the evaluation include processing using a learned model according to various machine learning algorithms such as deep learning, or knowledge base processing.
  • One of the methods of the knowledge base processing uses, for example, anatomical knowledge. For example, evaluation of a region label image using known anatomical knowledge such as regularity of a retinal shape is performed. I do.
  • an area label image corresponding to an OCT tomographic image in which the posterior segment is imaged is evaluated by the knowledge base processing.
  • anatomical tissue groups have fixed positions. Therefore, there is a method of checking the pixel value group in the area label image, that is, the coordinates of the area label value group, and evaluating whether or not the position is correctly output.
  • the evaluation method for example, if there is a region label value corresponding to the crystalline lens at coordinates close to the anterior segment in a certain range and a region label value corresponding to the retinal layer group at distant coordinates, it is possible to perform the image segmentation process likelihood. Evaluate On the other hand, when these region label values are not at such assumed positions, it is evaluated that the image segmentation processing has not been properly performed.
  • the evaluation method of the knowledge base will be described more specifically using an area label image Im2710 of a layer group constituting the retinal layer corresponding to an OCT tomographic image of the posterior segment shown in FIG. I do.
  • the anatomical tissue group has a fixed position.Therefore, by checking the pixel value group in the region label image, that is, the coordinates of the region label value group, whether the region label image is likely to be an image Can be determined.
  • the region label image Im2710 includes a region Seg 2711, a region Seg 2712, a region Seg 2713, and a region Seg 2714 in which a pixel group having the same region label value is continuously formed. Although the region Seg 2711 and the region Seg 2714 have the same region label value, the region group constituting the retinal layer anatomically has a layer structure. It is evaluated that the image segmentation processing has been performed. In this case, the area label image evaluation engine outputs a false value as the image evaluation index.
  • a method of the knowledge-based evaluation processing there is a method of evaluating whether or not a pixel having an area label value corresponding to an area that is supposed to be present in the imaging target is included in the area label image. Further, for example, there is a method of evaluating whether or not a predetermined number or more of pixels having an area label value corresponding to an area that is supposed to be present in the imaging target are included in the area label image.
  • the region label image evaluation engine selects one of the most likely region label images from the plurality of label images. Can also be output.
  • the area label evaluation engine can select one area label image with a predetermined priority, for example.
  • the area label evaluation engine can, for example, weight a plurality of area label images and merge them into one area label image.
  • the area label evaluation engine displays a plurality of area label images on a user interface provided on an arbitrary display unit or the like, and selects one of them according to an instruction of an examiner (user). Good.
  • the region label image evaluation engine may output all of the plurality of likely region images.
  • the region label image correction engine is a module that corrects a region in an input region label image that has been erroneously subjected to image segmentation processing.
  • a knowledge base process or the like.
  • One of the methods of the knowledge base processing uses, for example, anatomical knowledge.
  • the knowledge base correction for the region label image is corrected.
  • the method will be described more specifically.
  • the region label image Im2710 since the layer group anatomically forming the retinal layer has a layer structure, the region Seg2714 is erroneously determined from the shape and the positional relationship with other regions. It can be seen that the area has been subjected to image segmentation processing.
  • the region label image correction engine detects a region that has been erroneously segmented, and overwrites the detected region with another region label value. For example, in the case of FIG. 27, the area Seg 2714 is overwritten with an area label value indicating that the area is not any of the retinal layers.
  • the region label image correction engine may detect or specify an erroneously segmented region using the evaluation result of the region label evaluation engine. Further, the region label image correction engine may overwrite the detected label value of the erroneously segmented region with the label information estimated from the label information around the region. In the example of FIG. 27, when label information is attached to an area surrounding the area Seg2714, the label information of the area Seg2714 can be overwritten with the label information of the area.
  • the peripheral label information is not limited to the label information of the area completely surrounding the area to be corrected, and may be the label information having the largest number among the label information of the areas adjacent to the area to be corrected. .
  • FIG. 28 illustrates an example of a schematic configuration of an image processing apparatus according to the present embodiment.
  • the image processing device 2800 is connected to the imaging device 2810 and the display unit 2820 via a circuit or a network. Further, the imaging device 2810 and the display unit 2820 may be directly connected. Although these devices are separate devices in this embodiment, a part or all of these devices may be integrally configured. These devices may be connected to any other device via a circuit or a network, or may be configured integrally with any other device.
  • the image processing apparatus 2800 includes an acquisition unit 2801, a shooting condition acquisition unit 2802, a processing availability determination unit 2803, a segmentation processing unit 2804, an evaluation unit 2805, an analysis unit 2806, and an output unit 2807 (display control unit).
  • the image processing device 2800 may be configured by a plurality of devices in which some of these components are provided.
  • the acquisition unit 2801 can acquire various data and images from the imaging device 2810 and other devices, and can acquire input from the examiner via an input device (not shown).
  • an input device not shown
  • a mouse, a keyboard, a touch panel, and any other input device may be employed as the input device.
  • display portion 2820 may be configured as a touch panel display.
  • the imaging condition acquisition unit 2802 acquires the imaging conditions of the medical image (input image) acquired by the acquisition unit 2801. More specifically, a group of imaging conditions stored in a data structure forming a medical image is acquired according to the data format of the medical image. Note that when the imaging conditions are not stored in the medical image, the imaging information group can be acquired from the imaging device 2810 or the image management system via the acquisition unit 2801.
  • the processing availability determination unit (determination unit) 2803 determines whether the medical image can be handled by the segmentation processing unit 2804 using the imaging condition group acquired by the imaging condition acquisition unit 2802.
  • the segmentation processing unit 2804 performs image segmentation processing on a medical image that can be dealt with using an image segmentation engine (segmentation engine) including the learned model, and generates an area label image (area information).
  • the evaluation unit 2805 evaluates the area label image generated by the segmentation processing unit 2804 using an area label image evaluation engine (evaluation engine), and determines whether to output the area label image based on the evaluation result. .
  • the area label image evaluation engine outputs a true value as the image evaluation index if the input area label image is likely, and outputs a false value otherwise.
  • the evaluation unit 2805 determines that the area label image is output when the image evaluation index obtained by evaluating the area label image is a true value.
  • the analysis unit 2806 performs an image analysis process on the input image using the area label image or the input image determined to be output by the evaluation unit 2805.
  • the analysis unit 2806 can calculate, for example, a shape change and a layer thickness of the tissue included in the retinal layer by the image analysis processing. Note that any known image analysis process may be used as the image analysis process.
  • the output unit 2807 causes the display unit 2820 to display the region label image and the analysis result by the analysis unit 2806.
  • the output unit 2807 may store the area label image and the analysis result in a storage device connected to the image processing device 2800, an external device, or the like.
  • the segmentation processing unit 2804 generates an area label image corresponding to the input image (input) using the image segmentation engine.
  • a process using a learned model is performed.
  • the training of the machine learning model is configured by a pair group of input data that is an image acquired under a specific imaging condition assumed as a processing target and output data that is an area label image corresponding to the input data.
  • the specific imaging conditions specifically include a predetermined imaging region, imaging method, imaging angle of view, image size, and the like.
  • the input data of the teacher data is an image acquired with the same model as the imaging device 2810 and the same settings as the imaging device 2810.
  • the input data of the teacher data may be an image acquired from a photographing device having the same image quality tendency as the photographing device 2810.
  • the output data of the teacher data is an area label image corresponding to the input data.
  • the input data is a tomographic image Im2410 of a retinal layer captured by OCT.
  • the output data is an area label image Im2420 obtained by attaching an area label value representing the type of the retinal layer to the corresponding coordinates according to the type of the retinal layer depicted in the tomographic image Im2410, and dividing each area.
  • the area label image is created by a specialist with reference to a tomographic image, created by an arbitrary image segmentation process, or created by a specialist correcting an area label image created by the image segmentation process. Can be prepared.
  • the input data group of the teacher data pair group comprehensively includes input images having various conditions that are assumed to be processed.
  • the various conditions are, specifically, image conditions caused by a combination of variations such as a disease state of a subject, a photographing environment of a photographing device, and a technical level of a photographer.
  • the machine learning model is trained. Therefore, by using the image segmentation engine including the trained model that has undergone such training, the segmentation processing unit 2804 generates a stable and highly accurate region label image for images under various conditions. be able to.
  • pairs that do not contribute to the image segmentation process can be removed from the teacher data. For example, if the region label value of the region label image, which is output data forming a pair of teacher data, is incorrect, the region label of the region label image obtained using the trained model learned using the teacher data is used. It is more likely that the value will be wrong. That is, the accuracy of the image segmentation processing is reduced. Therefore, there is a possibility that the accuracy of the trained model included in the image segmentation engine can be improved by removing the pair having the area label image having the incorrect area label value as the output data from the teacher data.
  • the segmentation processing unit 2804 can specify an organ or a lesion depicted in the medical image when a medical image acquired by imaging is input.
  • An area label image can be output.
  • FIG. 29 is a flowchart of a series of image processing according to the present embodiment.
  • the process proceeds to step S2910.
  • the acquiring unit 2801 acquires, as an input image, an image photographed by the photographing device 2810 from the photographing device 2810 connected via a circuit or a network.
  • the acquisition unit 2801 may acquire an input image in response to a request from the imaging device 2810.
  • a request may be made, for example, when the imaging device 2810 generates an image, before or after storing the image generated by the imaging device 2810 in the recording device of the imaging device 2810, and then displays the stored image on the display unit 2820. It may be issued at the time of display, when an area label image is used for image analysis processing, or the like.
  • the obtaining unit 2801 may obtain data for generating an image from the imaging device 2810, and obtain an image generated based on the data by the image processing device 2800 as an input image.
  • any existing image generation method may be adopted as an image generation method for the image processing apparatus 2800 to generate various images.
  • the photographing condition acquiring unit 2802 acquires a photographing condition group of the input image. More specifically, a group of photographing conditions stored in a data structure forming the input image is acquired according to the data format of the input image. Note that, as described above, when the imaging conditions are not stored in the input image, the imaging condition acquisition unit 2802 can acquire the imaging information group from the imaging device 2810 or an image management system (not illustrated).
  • step S2930 the processability determination unit 2803 determines whether or not the input image can be subjected to image segmentation processing by the image segmentation engine used by the segmentation processing unit 2804, using the acquired shooting condition group. Specifically, the processing possibility determination unit 2803 determines whether the imaging region, imaging method, imaging angle of view, and image size of the input image match the conditions that can be dealt with using the learned model of the image segmentation engine. Is determined.
  • the processing availability determination unit 2803 determines all shooting conditions, and if it is determined that it can be dealt with, the process proceeds to step S2940. On the other hand, when the processability determination unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970.
  • Step S2940 may be performed.
  • the image segmentation engine can comprehensively deal with any imaging region of the subject, and can cope with the case where the input data includes an unknown imaging region. Such a process may be performed when it is implemented.
  • the processing availability determination unit 2803 determines whether at least one of a shooting region, a shooting method, a shooting angle of view, and an image size of an input image matches a condition that can be handled by the image segmentation engine. May be determined.
  • step S2940 the segmentation processing unit 2804 performs an image segmentation process on the input image using the image segmentation engine, and generates a region label image from the input image. Specifically, the segmentation processing unit 2804 inputs the input image to the image segmentation engine.
  • the image segmentation engine generates an area label image as area information that can specify an organ or a lesion depicted in an input image, based on a learned model in which machine learning has been performed using teacher data.
  • the segmentation processing unit 2804 inputs parameters together with the input image to the image segmentation engine according to the shooting condition group, and adjusts the extent of the image segmentation processing and the like. May be.
  • the segmentation processing unit 2804 may input a parameter corresponding to the input of the examiner to the image segmentation engine together with the input image to adjust the extent of the image segmentation processing.
  • step S2950 the evaluation unit 2805 evaluates whether or not the region label image generated by the segmentation processing unit 2804 is a likely image using the region label image evaluation engine.
  • the evaluation unit 2805 evaluates whether or not the area label image is a likely image using an area label evaluation engine that uses a knowledge-based evaluation method.
  • the area label evaluation engine checks the pixel value group in the area label image, that is, the coordinates of the area label value group, and evaluates whether or not the position is output to an anatomically correct position. In this case, for example, if there is an area label value corresponding to the crystalline lens at coordinates close to the anterior segment in a certain range and an area label value corresponding to the retinal layer group at distant coordinates, it is likely that the image segmentation process has been performed. evaluate. On the other hand, when these region label values are not at such assumed positions, it is evaluated that the image segmentation processing has not been properly performed.
  • the area label evaluation engine outputs a true value as an image evaluation index when the area label is evaluated to be likely to be able to perform the image segmentation processing, and a false value when the area label is evaluated to be unlikely to be able to perform the image segmentation processing. Is output.
  • the evaluation unit 2805 determines whether to output an area label image based on the image evaluation index output from the area label evaluation engine. Specifically, if the image evaluation index is a true value, the evaluation unit 2805 determines to output the area label image. On the other hand, when the image evaluation index is a false value, it is determined that the region label image generated by the segmentation processing unit 2804 is not output. When the evaluation unit 2805 determines that the region label image generated by the segmentation processing unit 2804 is not output, the evaluation unit 2805 can generate an image without a region label.
  • step S2960 when the evaluating unit 2805 determines that the area label image is to be output, the analyzing unit 2806 performs image analysis processing of the input image using the area label image and the input image.
  • the analysis unit 2806 calculates, for example, a change in a layer thickness or a tissue shape depicted in an input image by image analysis processing.
  • the method of the image analysis processing may employ any known processing. If the evaluation unit 2805 determines that an area label image is not to be output, or if an image without an area label is generated, the process proceeds without performing image analysis.
  • step S2970 when the output unit 2807 determines that the area label image is output by the evaluation unit 2805, the output unit 2807 outputs the area label image and the image analysis result, and causes the display unit 2820 to display the image.
  • the output unit 2807 may display or store these on the imaging device 2810 or another device instead of displaying the region label image and the image analysis result on the display unit 2820.
  • the output unit 2807 may process the image processing device 2800 so that it can be used by the image capturing device 2810 or another device, or may transmit the data to an image management system or the like, depending on the setting or implementation form of the image processing device 2800. May be converted.
  • the output unit 2807 is not limited to a configuration that outputs both the area label image and the image analysis result, and may output only one of them.
  • step S2930 if it is determined in step S2930 that the image segmentation process cannot be performed, the output unit 2807 outputs an image without an area label, which is a type of area label image, and causes the display unit 2820 to display the image. Note that, instead of outputting the image without the region label, a signal indicating that the image segmentation processing has been impossible may be transmitted to the imaging device 2810.
  • the output unit 2807 when it is determined in step S2950 that the image segmentation process has not been properly performed, the output unit 2807 outputs an image without an area label and causes the display unit 2820 to display the image. Also in this case, the output unit 2807 may transmit a signal indicating that the image segmentation processing has not been properly performed to the imaging device 2810 instead of outputting an image without an area label.
  • the output processing in step S2970 ends, a series of image processing ends.
  • the image processing apparatus 2800 includes the acquisition unit 2801 and the segmentation processing unit 2804.
  • the acquisition unit 2801 acquires an input image that is a tomographic image of a predetermined part of the subject.
  • the segmentation processing unit 2804 generates an area label image, which is area information capable of identifying an anatomical area, from the input image using an image segmentation engine including the learned model.
  • the image segmentation engine includes a learned model in which tomographic images under various conditions and region label images are used as learning data.
  • the image segmentation engine receives a tomographic image as input and outputs a region label image.
  • the image processing device 2800 further includes an evaluation unit 2805.
  • the evaluation unit 2805 evaluates the area label image using a knowledge-based evaluation engine using anatomical features, and determines whether to output the area label image according to the evaluation result.
  • the image processing apparatus 2800 uses the segmentation engine including the learned model to generate a region label image as region information used for specifying ROIs and VOIs that can be used for image diagnosis and image analysis. Can be generated. Therefore, it is possible to output a highly accurate region label image even for an input image having a bad condition in the conventional segmentation processing, and to provide an ROI or VOI that can be used for image diagnosis or image analysis.
  • the image processing apparatus 2800 can prevent the use of an inappropriate area label image for image diagnosis or image analysis by evaluating whether or not the area label image is a likely image by the evaluation unit 2805.
  • the image processing apparatus 2800 further includes a photographing condition acquisition unit 2802 and a processing availability determination unit 2803.
  • the imaging condition acquisition unit 2802 acquires an imaging condition including at least one of an imaging part, an imaging method, an imaging angle of view, and an image size of an input image.
  • the processing availability determination unit 2803 determines whether an area label image can be generated from an input image using an image segmentation engine. The processing availability determination unit 2803 makes the determination based on the shooting conditions of the input image.
  • the image processing apparatus 2800 can omit an input image that cannot be processed by the segmentation processing unit 2804 from the image segmentation processing, and reduce the processing load and the occurrence of errors of the image processing apparatus 2800. it can.
  • the image processing apparatus 2800 further includes an analysis unit 2806 that performs an image analysis of the input image using the area label image as the area information.
  • the image processing device 2800 can perform image analysis using the highly accurate region label image generated by the segmentation processing unit 2804, and can obtain a highly accurate analysis result.
  • the processing availability determination unit 2803 determines whether the input image can be subjected to image segmentation processing by the image segmentation engine. After that, when the processing availability determination unit 2803 determines that the input image can be processed by the segmentation processing unit 2804, the segmentation processing unit 2804 performs image segmentation processing. On the other hand, when the image capturing device 2810 performs image capturing only under image capturing conditions that allow image segmentation processing, the image acquired from the image capturing device 2810 may be unconditionally subjected to image segmentation processing. In this case, as shown in FIG. 30, the processing of steps S2920 and S2930 can be omitted, and step S2940 can be performed after step S2910.
  • the output unit 2807 displays the generated area label image and the analysis result on the display unit 2820.
  • the operation of the output unit 2807 is not limited to this.
  • the output unit 2807 can output the region label image and the analysis result to another device connected to the imaging device 2810 or the image processing device 2800. For this reason, the region label image and the analysis result can be displayed on the user interface of these devices, saved in any recording device, used for arbitrary image analysis, and transmitted to the image management system. it can.
  • the output unit 2807 displays the area label image and the image analysis result on the display unit 2820.
  • the output unit 2807 may cause the display unit 2820 to display the region label image or the image analysis result according to an instruction from the examiner.
  • the output unit 2807 may cause the display unit 2820 to display an area label image or an image analysis result in response to the examiner pressing an arbitrary button on the user interface of the display unit 2820.
  • the output unit 2807 may display the region label image by switching to the input image.
  • the output unit 2807 may display the area label image UI3120 side by side with the input image UI3110 as shown in FIG. 31, or may display the translucent area label with the input image as any one of UI3210 to UI3240 in FIG.
  • the image may be superimposed and displayed.
  • the translucent method may be any known method.
  • the area label image can be made translucent by setting the transparency of the area label image to a desired value.
  • the output unit 2807 indicates that the region label image is generated using the learned model and that the output unit 2807 is a result of image analysis performed based on the region label image generated using the learned model. May be displayed on the display unit 2820. Furthermore, the output unit 2807 may cause the display unit 2820 to display a display indicating what kind of teacher data the learned model has learned. The display may include an explanation of the types of the input data and the output data of the teacher data, and an arbitrary display related to the teacher data such as an imaging part included in the input data and the output data.
  • the process proceeds to step S2940, and the image segmentation process by the segmentation processing unit 2804 is started.
  • the output unit 2807 may cause the display unit 2820 to display the determination result by the processing availability determination unit 2803, and the segmentation processing unit 2804 may start the image segmentation process according to an instruction from the examiner.
  • the output unit 2807 can also display, on the display unit 2820, the input image and the imaging conditions such as the imaging region acquired for the input image, along with the determination result.
  • the image segmentation process is performed after the examiner determines whether or not the determination result is correct, so that the image segmentation process based on the erroneous determination can be reduced.
  • the output unit 2807 may cause the display unit 2820 to display imaging conditions such as an input image and an imaging part acquired for the input image without performing the determination by the processing availability determination unit 2803.
  • the segmentation processing unit 2804 can start the image segmentation processing according to the instruction from the examiner.
  • the evaluation unit 2805 evaluated the area label image generated by the segmentation processing unit 2804 by using an area label image evaluation engine that employs a knowledge-based evaluation method.
  • the evaluation unit 2805 uses an area label image evaluation engine including a trained model in which training is performed using the area label image and an image evaluation index based on a predetermined evaluation method as teacher data, and the area label image is likely to be an image. It may be evaluated whether or not.
  • a region label image or a fake image like a region label image is used as input data, and an image evaluation index for each image is used as output data.
  • the image evaluation index is a true value when the input data is an appropriate area label image, and a false value when the input data is a false image.
  • a method of generating a fake image a method of using an arbitrary generator of an area label image in which inappropriate conditions are set, a method of intentionally overwriting an appropriate area label image with an inappropriate area label, and the like May be adopted.
  • the evaluation unit 2805 evaluates whether or not the region label image is a plausible image using the region label image evaluation engine including the trained model that has performed such learning, it is not suitable for image diagnosis or image analysis. It is possible to prevent a region label image from being used.
  • the image without region label may be generated by the output unit 2807 or may be generated by the processing availability determination unit 2803. Good. If the evaluation unit 2805 determines that image segmentation has not been performed properly, the output unit 2807 may display on the display unit 2820 that the image segmentation has not been performed properly in step S2970. .
  • the segmentation processing unit 2804 includes one image segmentation engine.
  • the segmentation processing unit uses a plurality of image segmentation engines including respective learned models that have performed machine learning using different teacher data, and uses a plurality of image segmentation engines with respect to the input image. Generate a label image.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
  • the segmentation processing unit 2804 performs image segmentation processing on an input image using two or more image segmentation engines including respective trained models that have been machine-learned using different teacher data. Do.
  • a method of creating the teacher data group will be described. Specifically, first, a group of pairs of an image as input data and an area label image as output data, in which various imaging parts are imaged, is prepared. Next, a teacher data group is created by grouping a pair group for each imaging region. For example, first teacher data composed of a pair group acquired by imaging the first imaging region, and second teacher data composed of a pair group acquired by imaging the second imaging region. Thus, a teacher data group is created.
  • the machine learning model included in the separate image segmentation engine performs machine learning.
  • a first image segmentation engine including a trained model trained with first teacher data is provided.
  • an image segmentation engine group is prepared, such as preparing a second image segmentation engine including a trained model trained with the second teacher data.
  • Such image segmentation engines differ in the teacher data used for training the learned models included in each. Therefore, such an image segmentation engine differs in the degree to which the input image can be subjected to the image segmentation process depending on the imaging conditions of the image input to the image segmentation engine.
  • the first image segmentation engine has a high accuracy of the image segmentation process for the input image obtained by imaging the first imaging region, and is obtained by imaging the second imaging region. The accuracy of the image segmentation process is low for images that have been corrupted.
  • the second image segmentation engine has a high accuracy of the image segmentation process for an input image obtained by imaging the second imaging region, and obtains an image obtained by imaging the first imaging region. , The accuracy of the image segmentation process is low.
  • each of the teacher data is constituted by a group of pairs grouped by the imaged region, the image quality tendencies of the images constituting the group of pairs are similar. For this reason, if the image segmentation engine is a corresponding imaging part, the image segmentation processing can be performed with higher accuracy than the image segmentation engine according to the eighth embodiment.
  • the imaging conditions for grouping pairs of teacher data are not limited to imaging regions, and may be imaging angles of view, image resolutions, or a combination of two or more of these. Good.
  • FIG. 33 is a flowchart of a series of image processing according to the present embodiment. Note that the processing in steps S3310 and S3320 is the same as that in steps S2910 and S2920 according to the eighth embodiment, and a description thereof will not be repeated.
  • the process of step S3330 may be omitted after the process of step S3320, and the process may proceed to step S3340.
  • step S3330 the processing availability determination unit 2803 determines whether any of the above-described image segmentation engine groups can deal with the input image using the shooting condition group acquired in step S3320.
  • step S3380 If the processing availability determination unit 2803 determines that none of the image segmentation engine groups can deal with the input image, the process moves to step S3380. On the other hand, if the process availability determination unit 2803 determines that any of the image segmentation engine groups can handle the input image, the process proceeds to step S3340. Note that, depending on the settings and the implementation form of the image processing apparatus 2800, as in the eighth embodiment, even if the image segmentation engine determines that some shooting conditions cannot be dealt with, even if step S3340 is performed. Good.
  • the segmentation processing unit 2804 selects an image segmentation engine to be processed based on the shooting conditions of the input image acquired in step S3320 and the information of the teacher data of the image segmentation engine group. More specifically, for example, the image segmentation information for the imaging region in the imaging condition group acquired in step S3320 has information of teacher data regarding the imaging region or surrounding imaging regions, and the image segmentation processing has high accuracy. Select an engine. In the above example, when the imaging region is the first imaging region, the segmentation processing unit 2804 selects the first image segmentation engine.
  • step S3350 the segmentation processing unit 2804 uses the image segmentation engine selected in step S3340 to generate an area label image obtained by subjecting the input image to image segmentation processing.
  • Steps S3360 and S3370 are the same as steps S2950 and S2960 in the eighth embodiment, and a description thereof will be omitted.
  • step S3380 when the evaluation unit 2805 determines that the area label image is to be output, the output unit 2807 outputs the area label image and the analysis result, and causes the display unit 2820 to display the image.
  • the output unit 2807 may display a message indicating that the region label image is a region label image generated by using the image segmentation engine selected by the segmentation processing unit 2804. Note that the output unit 2807 may output only one of the area label image and the analysis result.
  • the output unit 2807 outputs an image without an area label, which is a type of area label image, and causes the display unit 2820 to display the image. Note that, instead of generating an image without a region label, a signal indicating that the image segmentation processing has been impossible may be transmitted to the imaging device 2810.
  • the output unit 2807 when it is determined in step S3360 that the image segmentation process has not been properly performed, the output unit 2807 outputs an image without an area label, which is a type of area label image, and causes the display unit 2820 to display the image. In this case as well, the output unit 2807 may transmit a signal indicating that the image segmentation processing has not been properly performed to the imaging device 2810 instead of generating an image without an area label.
  • the output processing in step S3380 ends, a series of image processing ends.
  • the segmentation processing unit 2804 uses at least one of a plurality of image segmentation engines including each trained model that has performed learning using different learning data to generate a region label image. Generate.
  • each of the plurality of image segmentation engines includes a trained model that has been trained using different learning data for at least one of the imaging region, imaging angle of view, and image resolution.
  • the segmentation processing unit 2804 generates an area label image using an image segmentation engine corresponding to at least one of the imaging region of the input image, the imaging angle of view, and the resolution of the image.
  • the image processing device 2800 can perform more accurate image segmentation processing in accordance with the shooting conditions.
  • the segmentation processing unit 2804 selects the image segmentation engine used for the image segmentation processing based on the shooting conditions of the input image, but the selection processing of the image segmentation engine is not limited to this.
  • the output unit 2807 may cause the user interface of the display unit 2820 to display the shooting conditions of the acquired input image and the group of image segmentation engines.
  • the segmentation processing unit 2804 may select an image segmentation engine to be used for the image segmentation process according to an instruction from the examiner.
  • the output unit 2807 may cause the display unit 2820 to display information of teacher data used for learning of each image segmentation engine together with the image segmentation engine group.
  • the display mode of the information of the teacher data used for the learning of the image segmentation engine may be arbitrary.
  • the group of the image segmentation engines may be displayed using the name related to the teacher data used for the learning.
  • the output unit 2807 may display the image segmentation engine selected by the segmentation processing unit 2804 on the user interface of the display unit 2820, and may receive an instruction from the examiner.
  • the segmentation processing unit 2804 may determine whether or not to finally select the image segmentation engine as an image segmentation engine to be used for the image segmentation process, according to an instruction from the examiner.
  • the output unit 2807 may output the generated region label image and the evaluation result to another device connected to the imaging device 2810 or the image processing device 2800, as in the eighth embodiment.
  • the output unit 2807 may process the image processing device 2800 so that it can be used by the image capturing device 2810 or another device, or may transmit the data to an image management system or the like, depending on the setting or implementation form of the image processing device 2800. May be converted.
  • the imaging condition acquisition unit 2802 acquires the imaging condition group from the data structure of the input image and the like.
  • the imaging condition acquisition unit estimates the imaging region or the imaging region of the input image based on the input image using the imaging position estimation engine.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the ninth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the ninth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatuses according to the eighth and ninth embodiments, the configuration illustrated in FIG. I do.
  • the imaging condition acquisition unit 2802 estimates and acquires an imaging part or an imaging region drawn on the input image acquired by the acquisition unit 2801 using an imaging location estimation engine (estimation engine).
  • an estimation process using a machine learning algorithm is performed.
  • the training of the learned model according to the imaging location estimation method using a machine learning algorithm includes a pair group of input data which is an image and output data which is an imaging part label corresponding to the input data.
  • the input data is an image having a specific shooting condition assumed as a processing target (input image).
  • the input data is preferably an image acquired from a photographing device having the same image quality tendency as the photographing device 2810, and more preferably the same model having the same settings as the photographing device 2810.
  • the type of the imaging region label that is the output data may be an imaging region in which at least a part is included in the input data.
  • the type of the imaging region label that is the output data may be, for example, “macular part”, “optic nerve head”, “macular and optic nerve head”, and “other”.
  • the imaging location estimation engine includes a learned model that has performed learning using such teacher data to determine where the imaging region and the imaging region depicted in the input image are. Can be output.
  • the imaging location estimation engine can also output, for each required imaging level label or imaging area label at a required level of detail, the probability of being the imaging area or imaging area.
  • the imaging condition acquiring unit 2802 can estimate the imaging region and the imaging region of the input image based on the input image, and can acquire it as the imaging condition for the input image.
  • the imaging location estimation engine outputs, for each imaging region label or imaging region label, the probability of the imaging region or imaging region, the imaging condition acquisition unit 2802 determines the imaging region or imaging region with the highest probability. It is acquired as the shooting condition of the input image.
  • step S3310 and steps S3330 to S3380 according to the present embodiment are the same as these processes in the ninth embodiment, and thus description thereof will be omitted.
  • the process of step S3330 may be omitted after the process of step S3320, and the process may proceed to step S3340.
  • step S3310 the process proceeds to step S3320.
  • step S3320 the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image acquired in step S3310.
  • a group of photographing conditions stored in a data structure constituting the input image is obtained. If the imaging condition group does not include information on the imaging region or the imaging region, the imaging condition acquisition unit 2802 inputs the input image to the imaging location estimation engine, and captures the imaging region or imaging region of the input image. Estimate whether it was acquired. Specifically, the imaging condition acquisition unit 2802 inputs an input image to the imaging location estimation engine, evaluates the probability output for each of the imaging region label group or the imaging region label group, and determines the imaging probability with the highest probability. A part or an imaging region is set and acquired as an imaging condition of an input image.
  • the imaging condition acquisition unit 2802 can acquire the imaging information group from the imaging device 2810 or an image management system (not shown). . Subsequent processing is the same as a series of image processing according to the ninth embodiment, and a description thereof will not be repeated.
  • the imaging condition acquisition unit 2802 functions as an estimation unit that estimates at least one of an imaging region and an imaging region from an input image using the imaging location estimation engine including the learned model. I do.
  • the imaging condition acquisition unit 2802 inputs an input image to an imaging location estimation engine including a learned model in which an image to which an imaging region and an imaging region are labeled is used as learning data, thereby obtaining an imaging region and an imaging region of the input image. Is estimated.
  • the image processing apparatus 2800 can acquire the imaging conditions for the imaging region and the imaging region of the input image based on the input image.
  • the imaging condition acquisition unit 2802 estimates the imaging region and the imaging region of the input image using the imaging region estimation engine when the imaging condition group does not include information on the imaging region and the imaging region. went.
  • the situation in which the imaging site and the imaging region are estimated using the imaging location estimation engine is not limited to this.
  • the imaging condition acquisition unit 2802 can use the imaging location estimation engine to determine the imaging site and the imaging region even when the information on the imaging region and the imaging region included in the data structure of the input image is insufficient as the required level of detail. The estimation may be performed on the shooting area.
  • the imaging condition obtaining unit 2802 estimates the imaging region and the imaging region of the input image using the imaging region estimation engine. May be.
  • the output unit 2807 causes the display unit 2820 to display the estimation result output from the imaging location estimation engine and information about the imaging region and the imaging region included in the data structure of the input image, and the imaging condition acquisition unit 2802 performs the inspection.
  • These photographing conditions may be determined according to a user's instruction.
  • the segmentation processing unit enlarges or reduces the input image so that the input image has an image size that can be handled by the image segmentation engine. Further, the segmentation processing unit generates an area label image by reducing or enlarging the output image from the image segmentation engine so as to have the image size of the input image.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
  • the segmentation processing unit 2804 includes an image segmentation engine similar to the image segmentation engine according to the eighth embodiment.
  • the teacher data used for learning the machine learning model included in the image segmentation engine is different from the teacher data in the eighth embodiment.
  • a pair group of input data and output data is configured as teacher data by a group of images obtained by enlarging or reducing an image of input data and an image of output data so as to have a fixed image size. Is used.
  • the teacher data of the learned model included in the image segmentation engine according to the present embodiment will be described.
  • FIG. 34 for example, consider a case where there are an input image Im3410 and an area label image Im3420 smaller than a certain image size set for teacher data.
  • each of the input image Im3410 and the region label image Im3420 is enlarged so as to have a fixed image size set for the teacher data.
  • the enlarged image Im3411 and the enlarged region label image Im3421 are paired, and the pair is used as one of the teacher data.
  • the input data of the teacher data is an image having a specific shooting condition assumed as a processing target (input image), and the specific shooting condition is a predetermined shooting condition.
  • the segmentation processing unit 2804 generates an area label image by performing image segmentation processing on an input image using an image segmentation engine that has been trained with such teacher data. At this time, the segmentation processing unit 2804 generates a deformed image in which the input image is enlarged or reduced so as to have a fixed image size set for the teacher data, and inputs the deformed image to the image segmentation engine.
  • the segmentation processing unit 2804 reduces or enlarges the output image from the image segmentation engine so as to have the image size of the input image, and generates an area label image. For this reason, the segmentation processing unit 2804 according to the present embodiment generates an area label image by performing an image segmentation process using an image segmentation engine even for an input image having an image size that cannot be dealt with in the eighth embodiment. Can be.
  • FIG. 35 is a flowchart of the segmentation process according to the embodiment. Note that the processes in steps S2910, S2920, and steps S2950 to S2970 according to the present embodiment are the same as those in the eighth embodiment, and thus description thereof will be omitted. If the input image is subjected to the image segmentation processing unconditionally for the photographing conditions other than the image size, the processing in step S2930 may be omitted after the processing in step S2920, and the processing may proceed to step S2940.
  • step S2920 similarly to the eighth embodiment, when the shooting condition obtaining unit 2802 obtains the shooting condition group of the input image, the process proceeds to step S2930.
  • the processing possibility determination unit 2803 determines whether the input image can be handled by the image segmentation engine using the acquired shooting condition group. More specifically, the processing availability determination unit 2803 determines whether or not the imaging condition of the input image is an imaging region, an imaging method, and an imaging angle of view that can be handled by the image segmentation engine. Unlike the eighth embodiment, the processing availability determination unit 2803 does not determine the image size.
  • the processing availability determination unit 2803 determines the imaging region, the imaging method, and the imaging angle of view, and if it is determined that the input image can be handled, the process proceeds to step S2940.
  • the processability determination unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970. Note that, depending on the setting and the implementation form of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on a part of the imaging part, the imaging method, and the imaging angle of view, it is determined in step S2940 that the input image cannot be processed. An image segmentation process may be performed.
  • step S2940 the image segmentation process according to the embodiment shown in FIG. 35 is started.
  • the segmentation processing unit 2804 enlarges or reduces the input image to a fixed image size set for the teacher data, and generates a deformed image.
  • step S3520 the segmentation processing unit 2804 inputs the generated deformed image to the image segmentation engine, and acquires a first region label image that has been subjected to image segmentation.
  • step S3530 the segmentation processing unit 2804 reduces or enlarges the first region label image to the image size of the input image, and generates a final region label image.
  • the image segmentation processing according to the present embodiment ends, and the processing shifts to step S2950.
  • the processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, and a description thereof will not be repeated.
  • the segmentation processing unit 2804 adjusts the image size of the input image to an image size that can be handled by the image segmentation engine, and inputs the image size to the image segmentation engine.
  • the segmentation processing unit 2804 generates an area label image by adjusting the output image from the image segmentation engine to the original image size of the input image.
  • the image processing apparatus 2800 according to the present embodiment also performs image segmentation processing on an input image having an image size not supported in the eighth embodiment, and includes information on ROIs and VOIs that can be used for image diagnosis and image analysis. An area label image can be generated.
  • the segmentation processing unit generates an area label image by image segmentation processing based on a certain resolution by the image segmentation engine.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
  • the segmentation processing unit 2804 includes an image segmentation engine similar to that of the eighth embodiment.
  • the teacher data used for learning the machine learning model included in the image segmentation engine is different from the teacher data in the eighth embodiment. Specifically, after enlarging or reducing the image group to an image size such that the resolution of the image group forming the pair group of the input data and the output data of the teacher data becomes a constant resolution, a sufficiently large constant Padded to the image size.
  • the resolution of the image group refers to, for example, the spatial resolution of the imaging device and the resolution for the imaging region.
  • each of the image Im3610 and the region label image Im3620 is enlarged so as to have a fixed resolution set for the teacher data. Furthermore, padding is performed on each of the enlarged image Im3610 and the region label image Im3620 so as to have a fixed image size set for the teacher data. Then, the image Im3611 that has been enlarged and padded is paired with the region label image Im3621, and the pair is used as one of the teacher data.
  • the fixed image size set for the teacher data is the maximum image size when an image assumed as a processing target (input image) is enlarged or reduced to have a certain resolution. If the certain image size is not large enough, when the image input to the image segmentation engine is enlarged, there is a possibility that the learned model has an image size that cannot be dealt with.
  • the area where padding is performed is padded with a fixed pixel value, padded with neighboring pixel values, or mirror-padded according to the characteristics of the learned model so that image segmentation processing can be performed effectively.
  • an image having a specific imaging condition assumed as a processing target is used as input data, and the specific imaging condition includes a predetermined imaging part, imaging method, and imaging image. Is the corner. That is, unlike the eighth embodiment, the specific photographing condition according to the present embodiment does not include the image size.
  • the segmentation processing unit 2804 performs an image segmentation process on an input image using an image segmentation engine including a trained model trained with such teacher data to generate a region label image. At this time, the segmentation processing unit 2804 generates a deformed image obtained by enlarging or reducing the input image so as to have a fixed resolution set for the teacher data. In addition, the segmentation processing unit 2804 performs padding on the deformed image so as to have a fixed image size set for the teacher data, generates a padded image, and inputs the padded image to the image segmentation engine.
  • the segmentation processing unit 2804 trims the first region label image output from the image segmentation engine by the padded region to generate a second region label image. After that, the segmentation processing unit 2804 reduces or enlarges the generated second region label image to have the image size of the input image, and generates a final region label image.
  • the segmentation processing unit 2804 can generate an area label image by performing an image segmentation process using an image segmentation engine even for an input image having an image size that cannot be dealt with in the eighth embodiment. .
  • FIG. 37 is a flowchart of the image segmentation process according to the embodiment. Note that the processes in steps S2910, S2920, and steps S2950 to S2970 according to the present embodiment are the same as those in the eighth embodiment, and thus description thereof will be omitted.
  • the process of step S2930 may be omitted after the process of step S2920, and the process may proceed to step S2940.
  • step S2920 when the shooting condition obtaining unit 2802 obtains the shooting condition group of the input image, the process proceeds to step S2930.
  • the processing possibility determination unit 2803 determines whether the input image can be handled by the image segmentation engine using the acquired shooting condition group. More specifically, the processing availability determination unit 2803 determines whether or not the imaging condition of the input image is an imaging region, an imaging method, and an imaging angle of view that can be handled by the image segmentation engine. Unlike the eighth embodiment, the processing availability determination unit 2803 does not determine the image size.
  • the processing availability determination unit 2803 determines the imaging region, the imaging method, and the imaging angle of view, and if it is determined that the input image can be handled, the process proceeds to step S2940.
  • the processability determination unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970. Note that, depending on the setting and the implementation form of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on a part of the imaging part, the imaging method, and the imaging angle of view, it is determined in step S2940 that the input image cannot be processed. An image segmentation process may be performed.
  • step S2940 the image segmentation process according to the embodiment shown in FIG. 37 is started.
  • the segmentation processing unit 2804 enlarges or reduces the input image to have a fixed resolution set for the teacher data, and generates a deformed image.
  • step S3720 the segmentation processing unit 2804 generates a padding image by performing padding on the generated deformed image so as to have the image size set for the teacher data.
  • the segmentation processing unit 2804 fills the area to be padded with a fixed pixel value, a neighboring pixel value, or a mirror padding in accordance with the characteristics of the learned model so that the image segmentation processing can be performed effectively.
  • step S3730 the segmentation processing unit 2804 inputs the padding image to the image segmentation engine, and obtains a first region label image that has been subjected to image segmentation processing.
  • step S3740 the segmentation processing unit 2804 performs trimming on the first area label image by the area padded in step S3720 to generate a second area label image.
  • step S3750 the segmentation processing unit 2804 reduces or enlarges the second region label image to the image size of the input image, and generates a final region label image.
  • the image segmentation processing according to the present embodiment ends, and the processing shifts to step S2950.
  • the processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, and a description thereof will not be repeated.
  • the segmentation processing unit 2804 adjusts the image size of the input image so that the resolution of the input image becomes a predetermined resolution.
  • the segmentation processing unit 2804 generates a padded image of the input image whose image size has been adjusted so that the image size can be handled by the image segmentation engine, and inputs the image to the image segmentation engine. Thereafter, the segmentation processing unit 2804 trims the output image from the image segmentation engine by the padded area. Then, the segmentation processing unit 2804 generates an area label image by adjusting the image size of the trimmed image to the original image size of the input image.
  • the segmentation processing unit 2804 of this embodiment can generate an area label image by performing an image segmentation process using an image segmentation engine even for an input image having an image size that cannot be dealt with in the eighth embodiment.
  • the input image may be more efficiently subjected to image segmentation processing than the image segmentation engine according to the tenth embodiment that processes images of the same image size.
  • the segmentation processing unit generates an area label image by performing image segmentation processing on an input image for each area of a fixed image size.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
  • the segmentation processing unit 2804 includes an image segmentation engine similar to that of the eighth embodiment.
  • the teacher data used for learning the machine learning model included in the image segmentation engine is different from the teacher data in the eighth embodiment.
  • a pair group of input data, which is an input image, and output data, which is an area label image corresponding to the input image, which constitutes teacher data has a positional relationship between the input image and the area label image. It consists of a rectangular area image of a fixed image size.
  • the teacher data of the image segmentation engine according to the present embodiment will be described with reference to FIG. 38, regarding one of the pairs forming the teacher data, it is assumed that, for example, there is an image Im3810 as an input image and an area label image Im3820 as a corresponding area label image.
  • the input data of the teacher data is the image Im3810
  • the output data is the area label image Im3820.
  • a rectangular area image R3811 of the image Im3810 is used as input data
  • a rectangular area image R3821 which is the same (corresponding) shooting area as the rectangular area image R3811 in the area label image Im3820, is output data.
  • a pair of teacher data hereinafter, a first rectangular region image pair
  • a first rectangular region image pair is configured by the rectangular region image R3811 as input data and the rectangular region image R3821 as output data.
  • the rectangular region image R3811 and the rectangular region image R3821 are images having a fixed image size.
  • the image Im3810 and the region label image Im3820 may be aligned by an arbitrary method.
  • the positional relationship between the rectangular area image R3811 and the rectangular area image R3821 may be specified by an arbitrary method such as template matching.
  • the image size and the number of dimensions of the input data and the output data may be different. For example, there is a case where the input data is a part of the B-scan image (two-dimensional image) and the output data is a part of the A-scan image (one-dimensional).
  • the above-mentioned fixed image size can be determined, for example, from a common divisor of a corresponding pixel number group of each dimension, for an image size group of an image assumed as a processing target (input image). In this case, it is possible to prevent the positional relationships of the rectangular area image groups output by the image segmentation engine from overlapping.
  • the image assumed as the processing target is a two-dimensional image
  • the first image size in the image size group is 500 pixels in width and 500 pixels in height
  • the second image size is Consider a case where the width is 100 pixels and the height is 100 pixels.
  • a fixed image size for the rectangular area images R3811, R3821 is selected from the common divisor of each side.
  • a fixed image size is selected from a width of 100 pixels, a height of 100 pixels, a width of 50 pixels, a height of 50 pixels, a width of 25 pixels, a height of 25 pixels, and the like. If the image to be processed is three-dimensional, the number of pixels is determined for the width, height, and depth.
  • a plurality of rectangular areas can be set for one of a pair of an area label image corresponding to input data and an area label image corresponding to output data. Therefore, for example, the rectangular area image R3812 of the image Im3810 is set as input data, and the rectangular area image R3822 which is the same shooting area as the rectangular area image R3812 in the area label image Im3820 is set as output data. Then, a pair of teacher data is formed by the rectangular area image R3812 as input data and the rectangular area image R3822 as output data. Thus, a rectangular area image pair different from the first rectangular area image pair can be created.
  • the group of pairs forming the teacher data can be enhanced. Then, an efficient image segmentation process can be expected by the image segmentation engine that has been trained using the pair group forming the teacher data. However, pairs that do not contribute to the image segmentation processing of the trained model can be prevented from being added to the teacher data.
  • an image that draws one layer or an area to which one label is attached can be used as the teacher data.
  • an image of an area where a plurality of layers, for example, two layers, and more preferably three or more layers are drawn can be used.
  • an image of an area in which a plurality of areas into which labels are divided in the area label image is drawn can be used.
  • the position of the learned layer or area is more easily used by using the trained model. It can be expected that image segmentation processing can be performed appropriately.
  • the input data of the teacher data uses an image having a specific imaging condition assumed as a processing target, and the specific imaging condition is determined by a predetermined imaging part and imaging method. , And the shooting angle of view. That is, unlike the eighth embodiment, the specific photographing condition according to the present embodiment does not include the image size.
  • the segmentation processing unit 2804 generates an area label image by performing image segmentation processing on an input image using an image segmentation engine that has been trained with such teacher data. At this time, the segmentation processing unit 2804 divides the input image into a continuous rectangular area image group having a fixed image size set for the teacher data without any gap. The segmentation processing unit 2804 performs an image segmentation process on each of the divided rectangular area image groups using an image segmentation engine, and generates a divided area label image group. Thereafter, the segmentation processing unit 2804 arranges the generated divided area label images according to the positional relationship of the input images and combines them to generate a final area label image.
  • the segmentation processing unit 2804 of the present embodiment performs an image segmentation process on an input image in units of a rectangular area, and combines the images that have been subjected to the image segmentation process. As a result, it is possible to generate an area label image by performing image segmentation processing on an image having an image size not supported in the eighth embodiment.
  • FIG. 39 is a flowchart of the image segmentation process according to the embodiment. Note that the processes in steps S2910, S2920, and steps S2950 to S2970 according to the present embodiment are the same as those in the eighth embodiment, and thus description thereof will be omitted. If the input image is subjected to the image segmentation processing unconditionally for the photographing conditions other than the image size, the processing in step S2930 may be omitted after the processing in step S2920, and the processing may proceed to step S2940.
  • step S2920 when the shooting condition obtaining unit 2802 obtains the shooting condition group of the input image, the process proceeds to step S2930.
  • the processing availability determination unit 2803 determines whether the input image can be handled by the image segmentation engine using the acquired shooting condition group. More specifically, the processing availability determination unit 2803 determines whether or not the imaging condition of the input image is an imaging region, an imaging method, and an imaging angle of view that can be handled by the image segmentation engine. Unlike the eighth embodiment, the processing availability determination unit 2803 does not determine the image size.
  • the processing availability determination unit 2803 determines the imaging region, the imaging method, and the imaging angle of view, and if it is determined that the input image can be handled, the process proceeds to step S2940.
  • the processability determination unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970. Note that, depending on the setting and the implementation form of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on a part of the imaging part, the imaging method, and the imaging angle of view, it is determined in step S2940 that the input image cannot be processed. An image segmentation process may be performed.
  • step S3910 the input image is divided into a group of rectangular area images that are continuous without gaps and have a fixed image size set for teacher data.
  • FIG. 40 shows an example in which the input image Im4010 is divided into groups of rectangular area images R4011 to R4026 having a fixed image size. Note that, depending on the design of the machine learning model included in the image segmentation engine, the image size and the number of dimensions of the input image and the output image may be different. In that case, the above-described division positions of the input image are adjusted by overlapping or separating so that the combined area label image generated in step S3920 has no loss.
  • step S3920 the segmentation processing unit 2804 performs image segmentation processing on each of the rectangular area images R4011 to R4026 by the image segmentation engine to generate a divided area label image group.
  • step S3930 the segmentation processing unit 2804 arranges and combines the generated divided region label image groups in the same positional relationship as the divided rectangular region images R4011 to R4026 group of the input image. Accordingly, the segmentation processing unit 2804 can generate an area label image.
  • step S2950 is the same as the processing after step S2950 in the eighth embodiment, and a description thereof will not be repeated.
  • the segmentation processing unit 2804 divides an input image into a plurality of rectangular area images R4011 to R4026 having a predetermined image size. After that, the segmentation processing unit 2804 inputs the plurality of divided rectangular area images R4011 to R4026 to the image segmentation engine to generate a plurality of divided area label images, and integrates the plurality of divided area label images to obtain the area label. Generate an image. If the positional relationship between the rectangular area groups overlaps during integration, the pixel value groups of the rectangular area groups are integrated or overwritten.
  • the segmentation processing unit 2804 can generate an area label image by performing image segmentation processing using an image segmentation engine even for an input image having an image size that cannot be dealt with in the eighth embodiment. it can.
  • the teacher data is created from a plurality of images divided into a predetermined image size, a large number of teacher data can be created from a small number of images. Therefore, in this case, the number of input images and area label images for creating teacher data can be reduced.
  • the trained model included in the image segmentation engine according to the present embodiment is a model in which a tomographic image including two or more layers is used as input data, and a region label image corresponding to the tomographic image is used as output data to perform learning. is there. For this reason, compared to the case where an image that draws one layer or one label-attached area is used as teacher data, the image is more appropriately used using the trained model based on the positional relationship of the learned layers or areas. It can be expected that segmentation processing will be performed.
  • the evaluator selects the most accurate area label image from among the plurality of area label images output from the plurality of image segmentation engines in accordance with the instruction of the examiner.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
  • the segmentation processing unit 2804 performs image segmentation processing on an input image using two or more image segmentation engines including respective learned models that have been machine-learned using different teacher data.
  • a method of creating the teacher data group will be described. Specifically, first, a pair group of input data, which is an image photographed under various photographing conditions, and output data, which is an area label image, is prepared. Next, a teacher data group is created by grouping the pair groups according to an arbitrary combination of imaging conditions. For example, first teacher data composed of a pair group acquired by a combination of first imaging conditions, second teacher data composed of a pair group acquired by a combination of second imaging conditions, and so on. Is created as a teacher data group.
  • the machine learning model included in the separate image segmentation engine performs machine learning.
  • a first image segmentation engine including a trained model trained with first teacher data is provided.
  • a group of image segmentation engines is prepared, such as preparing a second image segmentation engine corresponding to the trained model trained with the second teacher data.
  • Such image segmentation engines differ in the teacher data used for training the learned models included in each. Therefore, in such an image segmentation engine, the accuracy with which the input image can be subjected to the image segmentation process differs depending on the imaging conditions of the image input to the image segmentation engine.
  • the first image segmentation engine has high accuracy of the image segmentation processing on an input image obtained by shooting under a combination of the first shooting conditions.
  • the first image segmentation engine has a low accuracy of the image segmentation process for an image captured and acquired under the combination of the second capturing conditions.
  • the second image segmentation engine has high accuracy of the image segmentation process for an input image captured and acquired under the combination of the second capturing conditions.
  • the second image segmentation engine has a low accuracy of the image segmentation processing for an image captured and acquired under the combination of the first capturing conditions.
  • each of the teacher data is constituted by a pair group grouped by a combination of the photographing conditions, the image quality tendency of the image group constituting the pair group is similar. Therefore, if the image segmentation engine is a combination of the corresponding shooting conditions, the image segmentation processing can be performed more accurately than the image segmentation engine according to the eighth embodiment.
  • the combination of the imaging conditions for grouping the pair of teacher data may be arbitrary, and may be, for example, a combination of two or more of the imaging region, the imaging angle of view, and the image resolution. Further, the grouping of the teacher data may be performed based on one shooting condition, as in the ninth embodiment.
  • the evaluation unit 2805 evaluates a plurality of region label images generated by the segmentation processing unit 2804 using the plurality of image segmentation engines in the same manner as in the eighth embodiment. After that, when there are a plurality of area label images for which the evaluation result is a true value, the evaluator 2805 selects an area label image with the highest accuracy among the plurality of area label images according to the instruction of the examiner, and outputs the selected area label image. It is determined as an area label image to be performed.
  • the evaluation unit 2805 may perform evaluation using an area label image evaluation engine including a learned model, or may perform evaluation using a knowledge-based area label image evaluation engine, as in the eighth embodiment. Is also good.
  • the analysis unit 2806 performs an image analysis process on the input image in the same manner as in the eighth embodiment using the input image and the area label image determined as the area label image to be output by the evaluation unit 2805.
  • the output unit 2807 can display the region label image determined as the region label image to be output and the analysis result on the display unit 2820 or output to another device, as in the eighth embodiment. Note that the output unit 2807 can display a plurality of area label images whose evaluation results are true values on the display unit 2820, and the evaluation unit 2805 responds to the instruction from the examiner who has checked the display unit 2820 most frequently. A highly accurate area label image can be selected.
  • the image processing apparatus 2800 can output the most accurate area label image corresponding to the instruction of the examiner among the plurality of area label images generated using the plurality of image segmentation engines.
  • FIG. 41 is a flowchart of a series of image processing according to the present embodiment. Note that the processing in steps S4110 and S4120 according to the present embodiment is the same as the processing in steps S2910 and S2920 in the eighth embodiment, and a description thereof will not be repeated.
  • the process of step S4130 may be omitted after the process of step S4120, and the process may proceed to step S4140.
  • step S4120 similarly to the eighth embodiment, when the shooting condition obtaining unit 2802 obtains the shooting condition group of the input image, the process proceeds to step S4130.
  • the processing possibility determination unit 2803 determines whether any of the image segmentation engine groups used by the segmentation processing unit 2804 can handle the input image, using the acquired imaging condition group.
  • step S4180 If the processing availability determination unit 2803 determines that none of the image segmentation engine groups can deal with the input image, the process moves to step S4180. On the other hand, if the process determination unit 2803 determines that any of the image segmentation engine groups can handle the input image, the process proceeds to step S4140. Note that, depending on the settings and the implementation form of the image processing apparatus 2800, as in the eighth embodiment, even if the image segmentation engine determines that some shooting conditions cannot be dealt with, even if step S4140 is executed. Good.
  • step S4140 the segmentation processing unit 2804 inputs the input image obtained in step S4110 to each of the image segmentation engine groups, and generates an area label image group. Note that the segmentation processing unit 2804 may input the input image only to the image segmentation engine that has been determined by the processability determination unit 2803 to be able to handle the input image.
  • step S4150 the evaluation unit 2805 evaluates the area label image group generated in step S4140 using the area label image evaluation engine, as in the eighth embodiment.
  • step S4160 when there are a plurality of region label images for which the evaluation result (image evaluation index) is a true value, the evaluation unit 2805 selects / determines the region label image to be output according to the instruction of the examiner.
  • the output unit 2807 causes the user interface of the display unit 2820 to display an area label image group whose evaluation result is a true value.
  • FIG. 42 shows an example of the interface.
  • the interface displays the input image UI4210 and the region label images UI4220, UI4230, UI4240, and UI4250 for which the evaluation result is a true value.
  • the examiner operates an arbitrary input device (not shown) to specify an area label image with the highest accuracy among the image group (area label images UI4220 to UI4250).
  • the evaluation unit 2805 selects the area label image specified by the examiner as the area label image to be output.
  • the area label image is selected / determined as an area label image to be output. If there is no region label image for which the evaluation result is a true value, the evaluation unit 2805 determines that the region label image generated by the segmentation processing unit 2804 is not output, and generates and outputs an image without a region label. / Selection, and the process proceeds to step S4170.
  • step S4170 as in the eighth embodiment, the analysis unit 2806 performs an image analysis process on the input image using the area label image determined to be the area label image to be output by the evaluation unit 2805 and the input image. If the evaluation unit 2805 outputs an image without a region label, the analysis unit 2806 advances the process to step S4180 without performing the image analysis process.
  • the output unit 2807 causes the display unit 2820 to display the region label image determined as the region label image to be output and the image analysis result.
  • the output unit 2807 may display or store these on the imaging device 2810 or another device instead of displaying the region label image and the image analysis result on the display unit 2820.
  • the output unit 2807 may process the image processing device 2800 so that it can be used by the image capturing device 2810 or another device, or may transmit the data to an image management system or the like, depending on the setting or implementation form of the image processing device 2800. May be converted.
  • the output unit 2807 is not limited to a configuration that outputs both the area label image and the image analysis result, and may output only one of them.
  • step S4130 if it is determined in step S4130 that the image segmentation process cannot be performed, the output unit 2807 outputs an image without an area label and causes the display unit 2820 to display the image. Note that, instead of outputting the image without the region label, a signal indicating that the image segmentation processing has been impossible may be transmitted to the imaging device 2810.
  • step S4150 when it is determined that the image segmentation process has not been properly performed (the generated region label image is not output), the output unit 2807 outputs an image without a region label, and the display unit 2820 To be displayed. Also in this case, the output unit 2807 may transmit a signal indicating that the image segmentation processing has not been properly performed to the imaging device 2810 instead of outputting an image without an area label.
  • the output processing in step S4170 ends, a series of image processing ends.
  • the segmentation processing unit 2804 generates a plurality of region label images from an input image using a plurality of image segmentation engines each including a different learned model. Further, the evaluator 2805 selects at least one of the plurality of pieces of area information determined to be evaluated and output according to an instruction of the examiner (user). More specifically, when there are a plurality of area label images whose image evaluation indices are true values, the evaluation unit 2805 outputs the area label image with the highest accuracy according to the instruction of the examiner. Is selected / determined. Accordingly, the image processing apparatus 2800 can output a highly accurate area label image corresponding to the instruction of the examiner among the plurality of area label images generated by using the plurality of image segmentation engines.
  • the evaluation unit 2805 selects / determines the most accurate area label image as the area label image to be output in accordance with the instruction of the examiner.
  • the evaluation unit 2805 may select / determine a plurality of area label images whose evaluation results are true values as area label images to be output, in accordance with an instruction of the examiner.
  • the analysis unit 2806 performs image analysis processing on a plurality of region label images selected as region label images to be output.
  • the output unit 2807 outputs a plurality of selected region label images and analysis results of the plurality of region label images.
  • the evaluator 2805 selects an area label image to be output from a plurality of area label images for which the evaluation result is a true value, according to the instruction of the examiner.
  • the output unit 2807 causes the display unit 2820 to display all the region label images generated by the segmentation processing unit 2804, and the evaluation unit 2805 causes the plurality of region label images to be displayed according to an instruction from the examiner.
  • An area label image to be output from the image may be selected.
  • the evaluation unit 2805 may select / determine a plurality of area label images as area label images to be output according to an instruction from the examiner.
  • the evaluation unit 2805 selects / determines an image to be output according to an instruction from the examiner, for a plurality of area label images whose evaluation results by the evaluation unit 2805 are true values.
  • the evaluation unit selects / determines an area label image to be output from a plurality of area label images whose evaluation results are true values, based on a predetermined selection criterion.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the fourteenth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the fourteenth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatuses according to the eighth and fourteenth embodiments, the configuration shown in FIG. I do.
  • the evaluation unit 2805 evaluates a plurality of region label images generated by the segmentation processing unit 2804 using the region label image evaluation engine, and outputs the image according to the image evaluation index and a predetermined selection criterion. Select an area label image.
  • step S4160 processing other than step S4160 according to the present embodiment is the same as the processing according to the fourteenth embodiment, and a description thereof will not be repeated.
  • the process of step S4130 may be omitted after the process of step S4120, and the process may proceed to step S4140.
  • step S4160 when there are a plurality of area label images for which the evaluation result in step S4150 is a true value, the evaluation unit 2805 determines an area label image to be output among the plurality of area label images in accordance with a predetermined selection criterion. Select / judge.
  • the evaluation unit 2805 selects, for example, an area label image to which a true value has been output as an evaluation result first in time series.
  • the selection criterion is not limited to this, and may be set arbitrarily according to a desired configuration.
  • the evaluation unit 2805 may generate, for example, an area label generated by an image segmentation engine in which the combination of the shooting condition group of the input image and the shooting condition of the learning data is the closest (matched) among the area label images whose evaluation results are true values. An image may be selected / determined.
  • the evaluation unit 2805 determines that the image segmentation processing has not been properly performed, generates a region-less label image, and outputs / select.
  • the processing after step S4170 is the same as the processing after step S4170 of the fourteenth embodiment, and a description thereof will not be repeated.
  • the segmentation processing unit 2804 generates a plurality of region label images from an input image using a plurality of image segmentation engines.
  • the evaluation unit 2805 selects, based on a predetermined selection criterion, at least one of the area label images evaluated to be output or an image without an area label.
  • the output unit 2807 outputs the area label image selected by the evaluation unit 2805.
  • the image processing apparatus 2800 it is possible to prevent the output of the area label image in which the image segmentation processing has failed based on the output of the area label image evaluation engine. Further, when there are a plurality of area label images for which the image evaluation index outputted by the area label image evaluation engine is a true value, one of them can be automatically selected and displayed or output.
  • At least one of the plurality of area label images whose image evaluation index is a true value is selected and output.
  • the plurality of area label images whose image evaluation index is a true value is selected. May be output.
  • the analysis unit 2806 performs image analysis on all the region label images output from the evaluation unit 2805.
  • the output unit 2807 may cause the display unit 2820 to display all of the area label images and the corresponding analysis results output from the evaluation unit 2805, or may output them to another device.
  • the segmentation processing unit divides a three-dimensional input image into a plurality of two-dimensional images (two-dimensional image group).
  • a two-dimensional image group is input to an image segmentation engine, and a segmentation processing unit combines the output images from the image segmentation engine to generate a three-dimensional region label image.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
  • the acquisition unit 2801 acquires a three-dimensional image composed of a structurally continuous two-dimensional image group.
  • the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a group of OCT B-scan images (tomographic images).
  • the segmentation processing unit 2804 performs a segmentation process on a three-dimensional image as an input image by using the image segmentation engine according to the present embodiment, and generates a plurality of two-dimensional region label images.
  • a pair group of input data and output data, which are teacher data of the image segmentation engine according to the present embodiment, is configured by a two-dimensional image group.
  • the segmentation processing unit 2804 divides the acquired three-dimensional image into a plurality of two-dimensional images, and inputs the two-dimensional images to the image segmentation engine. Accordingly, the segmentation processing unit 2804 can generate a plurality of two-dimensional region label images. Further, the segmentation processing unit 2804 generates a three-dimensional region label image by arranging and combining a plurality of two-dimensional region label images in an arrangement of the two-dimensional image before division.
  • the evaluation unit 2805 determines whether or not the three-dimensional region label image is a likely region label image by using the region label image evaluation engine. When the evaluation result is a true value, the evaluation unit 2805 determines and outputs the three-dimensional area label image as an area label image to be output. On the other hand, when the evaluation result is a false value, the evaluation unit 2805 generates and outputs a three-dimensional region label-less image.
  • the region label image evaluation engine includes a learned model
  • a three-dimensional region label image and an image evaluation index can be used as teacher data of the learned model.
  • the evaluation unit 2805 may evaluate each of the two-dimensional region label images before the combination.
  • the analysis unit 2806 performs an image analysis process on the three-dimensional region label image determined as a likely region label image by the evaluation unit 2805. Note that the analysis unit 2806 may perform image analysis processing on each of the two-dimensional region label images before the combination. If the evaluation unit 2805 outputs a three-dimensional region label-less image, the analysis unit 2806 does not perform image analysis.
  • the output unit 2807 outputs a three-dimensional region label image and an analysis result.
  • the display mode of the three-dimensional region label image may be arbitrary.
  • step S2910 the obtaining unit 2801 obtains a three-dimensional image.
  • the process of step S2930 may be omitted after the process of step S2920, and the process may proceed to step S2940.
  • step S2930 if the processing availability determination unit 2803 determines that the input image can be handled by the image segmentation engine, the process proceeds to step S2940.
  • step S2940 the segmentation processing unit 2804 divides the obtained three-dimensional image into a plurality of two-dimensional images.
  • the segmentation processing unit 2804 inputs each of the divided two-dimensional images to the image segmentation engine, and generates a plurality of two-dimensional region label images.
  • the segmentation processing unit 2804 combines the generated two-dimensional region label images based on the acquired three-dimensional image, and generates a three-dimensional region label image.
  • the processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, and a description thereof will not be repeated.
  • the segmentation processing unit 2804 divides an input image into a plurality of images having dimensions lower than the dimensions of the input image, and inputs the divided images to the segmentation engine. More specifically, the segmentation processing unit 2804 divides a three-dimensional input image into a plurality of two-dimensional images and inputs the divided two-dimensional images to an image segmentation engine. The segmentation processing unit 2804 combines a plurality of two-dimensional region label images output from the image segmentation engine to generate a three-dimensional region label image.
  • the segmentation processing unit 2804 can perform the image segmentation processing on the three-dimensional image using the image segmentation engine including the trained model trained using the teacher data of the two-dimensional image. it can.
  • the segmentation processing unit 2804 divides a three-dimensional input image into a plurality of two-dimensional images and performs image segmentation processing.
  • the target for performing the processing related to the division is not limited to the three-dimensional input image.
  • the segmentation processing unit 2804 may divide the two-dimensional input image into a plurality of one-dimensional images and perform the image segmentation processing.
  • the segmentation processing unit 2804 may divide the four-dimensional input image into a plurality of three-dimensional images or a plurality of two-dimensional images, and perform the image segmentation processing.
  • the segmentation processing unit divides the three-dimensional input image into a plurality of two-dimensional images, and performs the image segmentation processing on the two-dimensional images in parallel by the plurality of image segmentation engines. Then, the segmentation processing unit generates a three-dimensional region label image by combining the output images from the respective image segmentation engines.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the sixteenth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the sixteenth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatuses according to the eighth and sixteenth embodiments, the configuration illustrated in FIG. I do.
  • the segmentation processing unit 2804 performs image segmentation processing on a three-dimensional image as an input image using a plurality of image segmentation engines similar to those in the sixteenth embodiment, and generates a three-dimensional region label image.
  • the plurality of image segmentation engines used by the segmentation processing unit 2804 may be mounted so as to be distributed to two or more devices via a circuit or a network, or may be mounted on a single device. It may be.
  • the segmentation processing unit 2804 divides the obtained three-dimensional image into a plurality of two-dimensional images, as in the sixteenth embodiment.
  • a segmentation processing unit 2804 performs image segmentation processing on a plurality of two-dimensional images by sharing (in parallel) using a plurality of image segmentation engines to generate a plurality of two-dimensional region label images.
  • the segmentation processing unit 2804 combines a plurality of two-dimensional region label images output from the plurality of image segmentation engines based on a three-dimensional image to be processed, and generates a three-dimensional region label image. More specifically, the segmentation processing unit 2804 generates a three-dimensional region label image by arranging and combining a plurality of two-dimensional region label images in an arrangement of the two-dimensional image before division.
  • steps S2910 to S2930 and steps S2950 to S2970 according to the present embodiment are the same as those of the sixteenth embodiment, and a description thereof will be omitted.
  • the process of step S2930 may be omitted after the process of step S2920, and the process may proceed to step S2940.
  • step S2930 if the processing availability determination unit 2803 determines that the input image can be handled by the image segmentation engine, the process proceeds to step S2940.
  • step S2940 the segmentation processing unit 2804 divides the obtained three-dimensional image into a plurality of two-dimensional images.
  • the segmentation processing unit 2804 inputs each of the divided two-dimensional images to a plurality of image segmentation engines, performs image segmentation processing in parallel, and generates a plurality of two-dimensional region label images.
  • the segmentation processing unit 2804 combines the generated two-dimensional region label images based on the acquired three-dimensional image, and generates a three-dimensional region label image.
  • the segmentation processing unit 2804 includes a plurality of image segmentation engines.
  • a segmentation processing unit 2804 divides a three-dimensional input image into a plurality of two-dimensional images, and uses a plurality of image segmentation engines in parallel for the plurality of divided two-dimensional images to generate a plurality of two-dimensional region labels. Generate an image.
  • the segmentation processing unit 2804 generates a three-dimensional region label image by integrating a plurality of two-dimensional region label images.
  • the segmentation processing unit 2804 can perform the image segmentation processing on the three-dimensional image using the image segmentation engine including the trained model trained using the teacher data of the two-dimensional image. it can.
  • a three-dimensional image can be more efficiently subjected to image segmentation processing.
  • the target to be subjected to the division-related processing by the segmentation processing unit 2804 is not limited to the three-dimensional input image.
  • the segmentation processing unit 2804 may divide the two-dimensional input image into a plurality of one-dimensional images and perform the image segmentation processing.
  • the segmentation processing unit 2804 may divide the four-dimensional input image into a plurality of three-dimensional images or a plurality of two-dimensional images, and perform the image segmentation processing.
  • the teacher data of the plurality of image segmentation engines may be different teacher data depending on a processing target to be processed by each image segmentation engine.
  • the first image segmentation engine may learn with teacher data for a first shooting region
  • the second image segmentation engine may learn with teacher data for a second shooting region.
  • each image segmentation engine can perform image segmentation processing of a two-dimensional image with higher accuracy.
  • the evaluation unit 2805 can evaluate a three-dimensional region label image in parallel using a plurality of region label image evaluation engines including the learned model, similarly to the segmentation processing unit 2804. In this case, the evaluation unit 2805 evaluates a plurality of two-dimensional region label images generated by the segmentation processing unit 2804 using a plurality of region label image evaluation engines in parallel.
  • the evaluation unit 2805 may determine that the three-dimensional region label image is a likely region label image and output it. it can.
  • the teacher data of the trained model included in the area label image evaluation engine can be constituted by a two-dimensional area label image and an image evaluation index.
  • the evaluation unit 2805 determines that the three-dimensional region label image is a likely region label image and outputs it. You can also.
  • the acquisition unit acquires the input image from the image management system instead of the imaging device.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus 2800 according to the eighth embodiment, the description of the configuration illustrated in FIG.
  • FIG. 43 illustrates a schematic configuration of an image processing apparatus 2800 according to the present embodiment.
  • the image processing apparatus 2800 according to the present embodiment is connected to the image management system 4300 and the display unit 2820 via an arbitrary circuit or a network.
  • the image management system 4300 is a device and a system that receives and stores an image photographed by an arbitrary photographing device or an image processed image. Further, the image management system 4300 transmits an image in response to a request from a connected device, performs image processing on a stored image, and requests another device for a request for image processing. Can be.
  • the image management system 4300 can include, for example, an image storage and communication system (PACS).
  • PACS image storage and communication system
  • the acquisition unit 2801 can acquire an input image from the image management system 4300 connected to the image processing device 2800.
  • the output unit 2807 can output the area label image generated by the segmentation processing unit 2804 to the image management system 4300. Note that the output unit 2807 can display an area label image on the display unit 2820 as in the eighth embodiment.
  • step S2930 may be omitted after the process of step S2920, and the process may proceed to step S2940.
  • the acquiring unit 2801 acquires, as an input image, an image stored in the image management system 4300 from the image management system 4300 connected via a circuit or a network.
  • the acquisition unit 2801 may acquire an input image in response to a request from the image management system 4300.
  • a request may be issued, for example, when the image management system 4300 stores the image, before transmitting the stored image to another device, or when displaying the stored image on the display unit 2820.
  • the request is issued, for example, when a user operates the image management system 4300 to make a request for an image segmentation process, or when using an area label image for an image analysis function of the image management system 4300. May be.
  • step S2970 the output unit 2807 outputs the area label image to the image management system 4300 as an output image when the evaluation unit 2805 determines in step S2950 to output the area label image.
  • the output unit 2807 may process the output image so that it can be used by the image management system 4300 or may convert the data format of the output image depending on the settings and implementation of the image processing device 2800.
  • the output unit 2807 can also output the analysis result by the analysis unit 2806 to the image management system 4300.
  • step S2950 if the evaluation unit 2805 determines that the image segmentation process has not been properly performed, the image without the region label is output to the image management system 4300 as an output image. Also, in step S2930, when the processing availability determination unit 2803 determines that the input image cannot be subjected to the image segmentation process, the output unit 2807 outputs the image without the region label to the image management system 4300.
  • the acquisition unit 2801 acquires an input image from the image management system 4300.
  • the image processing device 2800 of the present embodiment increases the invasiveness of the photographer or the subject based on the image stored in the image management system 4300, and generates an area label image suitable for image diagnosis.
  • the output can be made without increasing labor.
  • the output area label image and the image analysis result can be stored in the image management system 4300 or displayed on a user interface provided in the image management system 4300. Further, the output region label image can be used for an image analysis function provided in the image management system 4300, or transmitted to another device connected to the image management system 4300 via the image management system 4300. .
  • the image processing device 2800, the image management system 4300, and the display unit 2820 may be connected to another device (not shown) via a circuit or a network. Although these devices are separate devices in this embodiment, a part or all of these devices may be integrally formed.
  • the correction unit uses the area label image correction engine to correct an incorrect area label value in the area label image output from the image segmentation engine.
  • the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment.
  • FIG. 44 shows a schematic configuration of an image processing device 4400 according to the present embodiment.
  • the image processing apparatus 4400 according to the present embodiment includes an acquisition unit 2801, a shooting condition acquisition unit 2802, a processability determination unit 2803, a segmentation processing unit 2804, an evaluation unit 2805, an analysis unit 2806, and an output unit 2807, and a correction unit. 4408 are provided.
  • the image processing device 4400 may be configured by a plurality of devices in which some of these components are provided.
  • the configuration of the image processing apparatus 4400 according to the present embodiment other than the correction unit 4408 is the same as the configuration of the image processing apparatus according to the eighth embodiment, the same reference numerals are used for the configuration shown in FIG. The description is omitted.
  • the image processing device 4400 may be connected to the image capturing device 2810, the display unit 2820, and other devices (not shown) via an arbitrary circuit or a network, similarly to the image processing device 2800 according to the eighth embodiment. These devices may be connected to any other device via a circuit or a network, or may be configured integrally with any other device. Although these devices are separate devices in this embodiment, a part or all of these devices may be integrally configured.
  • the correction unit 4408 includes an area label image correction engine that corrects an input area label image.
  • the region label image correction engine corrects a region label value by anatomical knowledge base processing as described above in the description of terms. Note that, for example, an area label value overwriting a continuous area of an area label value to be corrected is overwritten with an area label value having the largest number of pixels in contact with the continuous area.
  • steps S4510 to S4540 according to the present embodiment is the same as the processing in steps S2910 to S2940 in the eighth embodiment, and a description thereof will be omitted.
  • the process of step S4530 may be omitted after the process of step S4520, and the process may proceed to step S4540.
  • step S4540 when the segmentation processing unit 2804 generates an area label image, the process proceeds to step S4550.
  • step S4550 the evaluation unit 2805 evaluates the generated region label image using the region label image evaluation engine, as in the eighth embodiment. When the evaluation result is a true value, the evaluation unit 2805 determines that the area label image is an area label image to be output. On the other hand, when the evaluation result is a false value, the evaluation unit 2805 according to the present embodiment determines that the area label image is an area label image that needs to be corrected.
  • step S4560 the correction unit 4408 corrects the area label value of the area label image determined as the area label image requiring correction in step S4540 using the area label correction engine. Specifically, the correction unit 4408 inputs the region label image determined to need correction in step S4540 to the region label image correction engine.
  • the region label image correction engine corrects an erroneously set region label value of the input region label image according to anatomical knowledge base processing, and outputs the corrected region label image.
  • step S4550 If it is determined in step S4550 that the generated area label image is an area label image to be output, the correction unit 4408 proceeds with the processing without correcting the area label image.
  • step S4570 the analysis unit 2806 uses the area label image determined to be the area label image to be output in step S4540 or the area label image in which the area label has been corrected in step S4550, and Image analysis processing.
  • the content of the image analysis processing may be the same as that of the eighth embodiment, and the description is omitted.
  • the output unit 2807 causes the display unit 2820 to display the area label image determined as the area label image to be output or the area label image with the corrected area label and the image analysis result.
  • the output unit 2807 may display or store these on the imaging device 2810 or another device instead of displaying the region label image and the image analysis result on the display unit 2820.
  • the output unit 2807 may process the image processing device 4400 so that it can be used by the imaging device 2810 or another device, or may transmit the data to an image management system or the like, depending on the setting or the implementation form of the image processing device 4400. May be converted.
  • the output unit 2807 is not limited to a configuration that outputs both the area label image and the image analysis result, and may output only one of them.
  • step S4530 if it is determined in step S4530 that the image segmentation process cannot be performed, the output unit 2807 outputs an image without an area label and causes the display unit 2820 to display the image. Note that, instead of outputting the image without the region label, a signal indicating that the image segmentation processing has been impossible may be transmitted to the imaging device 2810. When the output processing in step S4580 ends, a series of image processing ends.
  • the image processing device 4400 further includes the correction unit 4408.
  • the correction unit 4408 corrects the area label image generated by the segmentation processing unit 2804 using an area label image correction engine that performs a knowledge base process using a predetermined correction method.
  • the output unit 2807 outputs the area label image corrected by the correction unit 4408.
  • the correction unit 4408 corrects the region label of the region label image determined by the evaluation unit 2805 that the image segmentation processing has not been properly performed.
  • the analysis unit 2806 performs an image analysis process on the area label image whose area label has been corrected.
  • the error of the area label image for which the image segmentation processing has failed by the area label image correction engine can be corrected and output.
  • the correction unit 4408 corrects the area label of the area label image whose evaluation result by the evaluation unit 2805 is a false value.
  • the configuration of the correction unit 4408 is not limited to this.
  • the correction unit 4408 may correct the region label for the region label image whose evaluation result by the evaluation unit 2805 is a true value.
  • the analysis unit 2806 performs an image analysis process on the input image using the corrected area label.
  • the output unit 2807 outputs the corrected area label image and the analysis result.
  • the evaluation unit 2805 determines that there is no area label so that the correction unit 4408 does not correct the area label for the area label image whose evaluation result is a false value. Images can also be generated. When the evaluation unit 2805 generates an image without an area label, the correction unit 4408 can proceed with the processing without performing the correction.
  • the image processing apparatus 2800 can output a more accurate region label image or analysis result.
  • the region information generated by the segmentation processing unit 2804 includes the region information. Not limited.
  • the region information generated by the segmentation processing unit from the input image using the image segmentation engine may be a group of numerical data such as coordinate values of pixels having each region label.
  • each of the learned models included in the image segmentation engine, the region label image evaluation engine, and the shooting location estimation engine can be provided in the image processing devices 2800 and 4400.
  • the learned model may be configured by, for example, a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may be configured by a circuit that performs a specific function such as an ASIC.
  • the learned model may be provided in a device of another server connected to the image processing devices 2800 and 4400 or the like.
  • the image processing apparatuses 2800 and 4400 can use the learned model by connecting to a server or the like having the learned model via an arbitrary network such as the Internet.
  • the server including the learned model may be, for example, a cloud server, a fog server, an edge server, or the like.
  • the processing unit 222 or the first processing unit 822 detects the retinal layer from the tomographic image using the learned model and generates a boundary image.
  • the segmentation processing unit 2804 generates an area label image corresponding to the input image by using the image segmentation engine including the learned model.
  • the information of the retinal layer detected using the learned model and the generated boundary image or region label image may be manually corrected according to an instruction from the operator.
  • the operator can change the position and label of the retinal layer by designating at least a part of the detection result of the retinal layer and the boundary image or the region label image displayed on the display units 50 and 2820.
  • the correction of the detection result and the correction of the boundary image or the region label image may be performed by the processing unit 222, the first processing unit 822, and the segmentation processing unit 2804 in accordance with the instruction of the operator. It may be performed by a component such as a correction unit different from the above.
  • the processing unit 222, the first processing unit 822, the segmentation processing unit 2804, or the correction unit corrects the structure of the retinal layer detected by the first processing unit in accordance with an instruction from the operator.
  • the correction unit or the like may be configured by a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may be configured by a circuit that performs a specific function such as an ASIC.
  • Modification 2 The data manually modified in the first modification is used for additional learning on the learned model used by the processing unit 222 or the first processing unit 822 and the learned model included in the image segmentation engine used by the segmentation processing unit 2804. You may be. Specifically, for the learned model used by the processing unit 222 or the first processing unit 822, the input tomographic image is used as input data of learning data, and the position of the retinal layer is corrected according to an instruction from the operator. Additional learning is performed using the information of (layer boundary) as output data (correct answer data). Note that a boundary image whose label has been corrected in accordance with an instruction from the operator may be used as output data. In addition, for the trained model included in the image segmentation engine, additional learning is performed using the input image as input data of learning data and the area label image whose label position has been changed according to the instruction from the operator as output data. Do.
  • Modification 3 The additional learning described in Modification 2 may be performed according to an instruction from the operator.
  • the display control unit 25 or the output unit 2807 uses the corrected detection result of the retinal layer, the region label image, and the like as the learning data when the correction according to the instruction of the operator according to the first modification is performed.
  • the presence or absence can be displayed on the display units 50 and 2820.
  • the operator can instruct whether additional learning is necessary by selecting an option displayed on the display units 50 and 2820.
  • the image processing apparatuses 20, 80, 2800, and 4400 can determine whether additional learning is necessary or not according to an instruction from the operator.
  • the learned model can be provided in a device such as a server.
  • the image processing apparatuses 20, 80, 2800, and 4400 respond to the instruction of the operator to perform the additional learning by using the input image and the detection result or the region label image or the like in which the above correction has been performed. Can be transmitted and stored in the server or the like as a pair of learning data.
  • the image processing apparatuses 20, 80, 2800, and 4400 can determine whether to transmit learning data for additional learning to a device such as a server including a learned model in accordance with an instruction from the operator. .
  • Modification 4 In the various embodiments and the modified examples described above, the configuration in which the processing of detecting the retinal layer and the processing of generating the region label image and the like are described for the still image.
  • the process of detecting the retinal layer and the process of generating the region label image and the like according to the above-described embodiment and the modification may be repeatedly performed on the moving image.
  • an ophthalmologic apparatus generates and displays a preview image (moving image) for positioning of the apparatus or the like before performing actual imaging. Therefore, for example, for at least one frame of the moving image of the tomographic image as the preview image, the processing of detecting the retinal layer and the processing of generating the region label image and the like according to the embodiment and the modification may be repeatedly executed. .
  • the display control unit 25 or the output unit 2807 can cause the display units 50 and 2820 to display the retinal layer, the region label image, and the like detected for the preview image.
  • the image processing apparatuses 20, 80, 2800, and 4400 operate the OCT so that the retinal layer detected in the preview image and the labeled area of the retinal layer are located at predetermined positions in the display area of the tomographic image.
  • the device can be controlled. More specifically, in the image processing apparatuses 20, 80, 2800, and 4400, the retinal layer detected in the preview image and the labeled area of the retinal layer are located at predetermined positions in the display area of the tomographic image.
  • the coherence gate position as follows. The adjustment of the coherence gate position may be performed, for example, by driving the coherence gate stage 14 by the drive control unit 23 or the like.
  • the adjustment of the coherence gate position may be manually performed according to an instruction from the operator.
  • the operator adjusts the adjustment amount of the coherence gate position based on the retinal layer or the region label image detected for the preview image displayed on the display unit 50, 2820 by the image processing device 20, 80, 2800, 4400. Can be entered.
  • the moving image to which the detection processing of the retinal layer and the generation processing of the region label image and the like according to the above-described embodiment and the modified example are applicable is not limited to a live moving image, and may be, for example, a moving image stored (saved) in a storage unit. It may be an image. Also, during various adjustments of the coherence gate position and the like, there is a possibility that the imaging target such as the retina of the eye to be inspected has not been imaged well yet. For this reason, since there is a large difference between the medical image input to the learned model and the medical image used as the learning data, there is a possibility that the detection result of the retinal layer, the region label image and the like cannot be obtained with high accuracy.
  • the evaluation value such as the image quality evaluation of the tomographic image (B-scan) exceeds the threshold value
  • the detection process of the retinal layer using the learned model and the generation process of the region label image are automatically started. You may.
  • the button for instructing the image segmentation process using the learned model is changed to a state (active state) that can be specified by the examiner. May be configured.
  • the learned models used in the above-described various embodiments and modifications may be generated and prepared for each type of disease or each abnormal part.
  • the image processing apparatuses 20, 80, 2800, and 4400 select a learned model to be used for processing according to an input (instruction) of a disease type, an abnormal part, or the like of the eye to be inspected from the operator. can do.
  • the trained model prepared for each type of disease or abnormal region is not limited to a trained model used for detecting a retinal layer or generating an area label image, and may be, for example, an image evaluation engine or an analysis engine. It may be a learned model used in an engine or the like.
  • the image processing apparatuses 20, 80, 2800, and 4400 may identify a disease type and an abnormal part of the subject's eye from the image using a separately prepared learned model.
  • the image processing apparatuses 20, 80, 2800, and 4400 automatically generate the learned model used in the above processing based on the type of the disease and the abnormal part identified using the separately prepared learned model. Can be selected.
  • the trained model for identifying the type of disease or abnormal part of the eye to be examined is a learning model in which a tomographic image or a fundus image is used as input data, and the type of disease or abnormal part in these images is used as output data. Learning may be performed using pairs.
  • a tomographic image, a fundus image, or the like may be used alone as input data, or a combination thereof may be used as input data.
  • a hostility generation network GAN: General Adversary Network
  • VAE Variational auto-encoder
  • a DCGAN Deep ⁇ Convolutional ⁇ GAN including a generator obtained by learning generation of a tomographic image and a classifier obtained by learning identification of a new tomographic image generated by the generator and a real frontal fundus image.
  • a machine learning model can be used as a machine learning model.
  • the discriminator encodes the input tomographic image into a latent variable by encoding, and the generator generates a new tomographic image based on the latent variable. Thereafter, a difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part.
  • VAE digital versatile image
  • an input tomographic image is encoded by an encoder to be a latent variable, and the latent variable is decoded by a decoder to generate a new tomographic image. Thereafter, a difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part.
  • a tomographic image has been described as an example of the input data, a fundus image, a front image of the anterior eye, and the like may be used.
  • the processing unit 222, the first processing unit 822, and the segmentation processing unit 2804 use the learned model to detect the region of the subject's eye, May be subjected to predetermined image processing. For example, consider a case where at least two of the vitreous, retinal, and choroidal regions are detected. In this case, when image processing such as contrast adjustment is performed on at least two detected areas, adjustments suitable for each area can be performed by using different image processing parameters. By displaying an image adjusted appropriately for each region, the operator can more appropriately diagnose a disease or the like for each region. Note that, for a configuration using different image processing parameters for each of the detected regions, for example, regarding the region of the subject's eye detected by the second processing unit 823 that detects the region of the subject's eye without using the learned model The same may be applied.
  • the display control unit 25 or the output unit 2807 in the various embodiments and modifications described above may display analysis results such as a desired layer thickness and various blood vessel densities on the report screen of the display screen.
  • the value (distribution) of a parameter relating to a site of interest including at least one of a blood vessel wall, a blood vessel inner wall boundary, a blood vessel outer boundary, a ganglion cell, a corneal region, a corner region, and Schlemm's canal may be displayed as an analysis result.
  • the artifact is, for example, a false image region caused by light absorption by a blood vessel region or the like, a projection artifact, a band-like artifact in a front image generated in a main scanning direction of measurement light due to a state (movement, blink, etc.) of an eye to be inspected, or the like.
  • the artifact may be anything as long as it is an image failure area that randomly appears on a medical image of a predetermined part of the subject at each imaging, for example.
  • the display control unit 25 or the output unit 2807 causes the display units 50 and 2820 to display parameter values (distribution) regarding an area including at least one of the various artifacts (defect areas) as an analysis result. You may. Further, the values (distributions) of parameters relating to a region including at least one of abnormal sites such as drusen, new blood vessels, vitiligo (hard vitiligo), and pseudo drusen may be displayed as analysis results.
  • the analysis result may be displayed as an analysis map, a sector indicating a statistical value corresponding to each divided region, or the like.
  • the analysis result may be generated using a learned model (analysis result generation engine, a learned model for generating the analysis result) obtained by learning the analysis result of the medical image as learning data.
  • the trained model is a learning model using learning data including a medical image and an analysis result of the medical image, learning data including a medical image and an analysis result of a medical image of a type different from the medical image, and the like. May be obtained.
  • the learning data includes the detection result of the retinal layer by the processing unit 222 or the first processing unit 822 and / or the second processing unit 823, the region label image generated by the segmentation processing unit 2804, and the learning data. It may include the analysis result of the medical image.
  • the image processing apparatus generates an analysis result of the tomographic image from a result obtained by executing the first detection process using the learned model for generating the analysis result. It can serve as an example.
  • the learned model is obtained by learning using learning data including input data in which a plurality of medical images of different types of a predetermined part are set, such as a luminance front image and a motion contrast front image. Is also good.
  • the luminance front image corresponds to the luminance En-Face image
  • the motion contrast front image corresponds to the OCTA En-Face image.
  • an analysis result obtained using a high-quality image generated using a learned model for improving image quality may be displayed.
  • the input data included in the learning data may be a high-quality image generated using a trained model for high image quality, or a set of a low-quality image and a high-quality image. Is also good.
  • the learning data may be an image obtained by manually or automatically correcting at least a part of an image whose image quality has been improved using the learned model.
  • the learning data includes, for example, at least an analysis value (for example, an average value or a median value) obtained by analyzing the analysis area, a table including the analysis value, an analysis map, and the position of the analysis area such as a sector in the image.
  • Information including one may be data obtained by labeling input data as correct answer data (for supervised learning). Note that, in accordance with an instruction from the operator, an analysis result obtained using the learned model for generating the analysis result may be displayed.
  • the display control unit 25 and the output unit 2807 in the above-described embodiments and modified examples may display various diagnosis results such as glaucoma and age-related macular degeneration on the report screen of the display screen.
  • diagnosis results such as glaucoma and age-related macular degeneration
  • the display control unit 25 and the output unit 2807 in the above-described embodiments and modified examples may display various diagnosis results such as glaucoma and age-related macular degeneration on the report screen of the display screen.
  • diagnosis result the position of the specified abnormal part or the like may be displayed on the image, or the state or the like of the abnormal part may be displayed by characters or the like.
  • a classification result for example, a Curtin classification
  • an abnormal part or the like may be displayed as a diagnosis result.
  • the diagnosis result may be generated using a learned model (diagnosis result generation engine, a learned model for generating a diagnosis result) obtained by learning a diagnosis result of a medical image as learning data.
  • the learned model is obtained by learning using learning data including a medical image and a diagnosis result of the medical image, and learning data including a medical image and a diagnosis result of a medical image of a different type from the medical image. It may be obtained.
  • the learning data includes the detection result of the retinal layer by the processing unit 222 or the first processing unit 822 and / or the second processing unit 823, the region label image generated by the segmentation processing unit 2804, and the learning data.
  • a diagnosis result of the medical image for example, the image processing apparatus generates a diagnosis result of a tomographic image from a result obtained by executing the first detection processing using a learned model for generating a diagnosis result. It can serve as an example.
  • the input data included in the learning data may be a high-quality image generated using a learned model for improving the image quality, or a set of a low-quality image and a high-quality image. Is also good.
  • the learning data may be an image obtained by manually or automatically correcting at least a part of an image whose image quality has been improved using the learned model.
  • the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal part), the position of the lesion in the image, the position of the lesion with respect to the attention area, the findings (interpretation findings, etc.), Data that includes at least one of the following: information including at least one of grounds for negating the diagnosis name (negative medical support information) and the like (supervised learning). You may. In addition, according to the instruction from the examiner, a configuration may be adopted in which a diagnosis result obtained using the learned model for generating a diagnosis result is displayed.
  • the display control unit 25 and the output unit 2807 on the report screen of the display screen, recognize the object (for example, the object detection result such as the noted part, the artifact, and the abnormal part) as described above (object detection).
  • Results) and the segmentation results may be displayed.
  • a rectangular frame or the like may be superimposed and displayed around the object on the image.
  • a color or the like may be superimposed and displayed on an object in an image.
  • the object recognition result and the segmentation result may be generated using a learned model obtained by learning learning data obtained by labeling a medical image with information indicating the object recognition and the segmentation as correct data. .
  • the above-described generation of the analysis result and the generation of the diagnosis result may be obtained by using the above-described object recognition result and the segmentation result.
  • a process of generating an analysis result and a process of generating a diagnosis result may be performed on a region of interest obtained by the processing of object recognition and segmentation.
  • the learned model for generating a diagnosis result may be a learned model obtained by learning using learning data including input data in which a plurality of different types of medical images of a predetermined part of the subject are set.
  • input data included in the learning data for example, input data in which a front image of a motion contrast of a fundus and a luminance front image (or a luminance tomographic image) are set can be considered.
  • input data included in the learning data for example, input data in which a tomographic image of a fundus (B-scan image) and a color fundus image (or a fluorescent fundus image) are set may be considered.
  • the plurality of different types of medical images may be any medical images obtained by different modalities, different optical systems, different principles, or the like.
  • the learned model for generating a diagnosis result may be a learned model obtained by learning using learning data including input data in which a plurality of medical images of different parts of the subject are set.
  • input data included in the learning data for example, input data in which a tomographic image of the fundus (B-scan image) and a tomographic image of the anterior segment (B-scan image) are set can be considered.
  • input data included in the learning data for example, input data in which a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic papilla of the fundus are set. Is also conceivable.
  • the input data included in the learning data may be different parts of the subject and a plurality of different types of medical images.
  • the input data included in the learning data may be, for example, input data that sets a tomographic image of the anterior ocular segment and a color fundus image.
  • the above-described learned model may be a learned model obtained by learning using learning data including input data in which a plurality of medical images of a predetermined part of the subject with different imaging angles of view are set.
  • the input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a predetermined part into a plurality of regions, such as a panoramic image.
  • the input data included in the learning data may be input data in which a plurality of medical images at different dates and times of a predetermined part of the subject are set.
  • the display screen on which at least one of the analysis result, the diagnosis result, the object recognition result, and the segmentation result is displayed is not limited to the report screen.
  • a display screen includes, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). May be displayed.
  • a shooting confirmation screen a display screen for follow-up observation
  • a preview screen for various adjustments before shooting a display screen on which various live moving images are displayed. May be displayed.
  • the change in the display of the low-quality image and the high-quality image described in the seventh embodiment and the like may be, for example, the change in the display of the analysis result of the low-quality image and the analysis result of the high-quality image.
  • the various learned models described above can be obtained by machine learning using learning data.
  • the machine learning includes, for example, deep learning (Deep @ Learning) including a multi-layer neural network.
  • deep learning Deep @ Learning
  • a convolutional neural network CNN: Convolutional Neural Network
  • at least a part of the multi-layer neural network may use a technology related to an auto encoder (self-encoder).
  • a technology related to back propagation error back propagation method
  • the machine learning is not limited to the deep learning, but may be any learning using a model capable of extracting (representing) a feature amount of learning data such as an image by learning.
  • the machine learning model may be, for example, a capsule network (Capsule @ Network; CapsNet).
  • a capsule network Capsule @ Network; CapsNet
  • each unit is configured to output a scalar value, for example, spatial information about a spatial positional relationship (relative position) between features in an image is obtained. It is configured to be reduced. Thereby, for example, it is possible to perform learning such that the influence of local distortion or parallel movement of the image is reduced.
  • each unit is configured to output spatial information as a vector, so that, for example, spatial information is held.
  • learning can be performed in which a spatial positional relationship between features in an image is considered.
  • the high-quality image engine (learned model for high-quality image) may be a learned model obtained by additionally learning learning data including at least one high-quality image generated by the high-quality image engine. Good. At this time, whether or not to use the high-quality image as learning data for additional learning may be configured to be selectable by an instruction from the examiner.
  • the preview screens in the various embodiments and modifications described above may be configured so that the above-described learned model for improving image quality is used for at least one frame of a live moving image.
  • the learned model corresponding to each live moving image may be used.
  • the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of imaging. For this reason, for example, failure in re-imaging can be reduced, so that the accuracy and efficiency of diagnosis can be improved.
  • the plurality of live moving images may be, for example, a moving image of the anterior segment for alignment in the XYZ directions and a front moving image of the fundus for focus adjustment of the fundus observation optical system and OCT focus adjustment.
  • the plurality of live video images may be, for example, tomographic video images of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length).
  • the moving image to which the learned model can be applied is not limited to a live moving image, and may be, for example, a moving image stored (saved) in a storage unit.
  • a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus stored (saved) in the storage unit may be displayed on the display screen.
  • a reference frame may be selected based on the condition that the vitreous body exists on the frame as much as possible.
  • each frame is a tomographic image (B-scan image) in the XZ direction.
  • a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen.
  • a configuration may be adopted in which high-quality images (high-quality frames) sequentially generated using a learned model for improving image quality are continuously displayed for at least one frame of a moving image.
  • the same method may be applied to the method of positioning in the X direction and the method of positioning in the Z direction (depth direction), or all different methods may be used. May be applied.
  • the alignment in the same direction may be performed a plurality of times by different methods. For example, after the rough alignment is performed, the precise alignment may be performed.
  • a positioning method for example, there is positioning (coarse in the Z direction) using a retinal layer boundary obtained by performing a segmentation process on a tomographic image (B-scan image).
  • an alignment method for example, there is an alignment (precise in the X and Z directions) using correlation information (similarity) between a plurality of regions obtained by dividing a tomographic image and a reference image.
  • a positioning method for example, positioning (in the X direction) using a one-dimensional projection image generated for each tomographic image (B scan image), and using a two-dimensional front image (in the X direction) There is alignment, etc.
  • the configuration may be such that, after coarse positioning is performed in pixel units, precise positioning is performed in subpixel units.
  • a different learned model for improving image quality is prepared for each shooting mode having a different scanning pattern or the like, and a learned model for improving image quality corresponding to the selected shooting mode is selected. Is also good. Further, one learned model for improving image quality obtained by learning learning data including various medical images obtained in different imaging modes may be used.
  • a learned model obtained by learning for each imaging region may be selectively used. Specifically, a first learned model obtained using learning data including a first imaging part (lung, eye to be examined, etc.) and a learning including a second imaging part different from the first imaging part A plurality of learned models including the second learned model obtained using the data can be prepared. Then, the control unit 200 may include a selection unit that selects any one of the plurality of learned models. At this time, the control unit 200 may include a control unit that executes additional learning on the selected learned model. The control means searches for data in which an imaging part corresponding to the selected learned model and an imaging image of the imaging part are paired in accordance with an instruction from the examiner, and retrieves the obtained data as learning data.
  • the imaging part corresponding to the selected learned model may be obtained from the information in the header of the data or manually input by the examiner.
  • the data search may be performed via a network from a server or the like of an external facility such as a hospital or a laboratory.
  • the selection unit and the control unit may be configured by a software module executed by a processor such as a CPU or an MPU of the control unit 200. Further, the selection unit and the control unit may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like.
  • the validity of the learning data for additional learning may be detected by confirming the matching by digital signature or hashing. Thereby, the learning data for additional learning can be protected. At this time, if the validity of the learning data for additional learning cannot be detected as a result of checking the consistency by digital signature or hashing, a warning to that effect is issued, and additional learning using the learning data is performed. Make it not exist.
  • the server may be in any form, such as a cloud server, a fog server, an edge server, etc., regardless of the installation location.
  • the instruction from the examiner may be an instruction by voice or the like in addition to a manual instruction (for example, an instruction using a user interface or the like).
  • a machine learning model including a speech recognition model speech recognition engine, learned model for speech recognition
  • the manual instruction may be an instruction by character input using a keyboard, a touch panel, or the like.
  • a machine learning model including a character recognition model a character recognition engine, a learned model for character recognition
  • the instruction from the examiner may be an instruction by a gesture or the like.
  • a machine learning model including a gesture recognition model gesture recognition engine, learned model for gesture recognition
  • the instruction from the examiner may be the result of detection of the examiner's line of sight on the display units 50 and 2820.
  • the gaze detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from around the display units 50 and 2820. At this time, the pupil detection from the moving image may use the above-described object recognition engine.
  • the instruction from the examiner may be an instruction based on an electroencephalogram, a weak electric signal flowing through the body, or the like.
  • the learning data character data or voice data (waveform data) indicating an instruction to display a result of processing of the various learned models as described above is used as input data, and various learned models are used. It may be learning data in which an execution instruction for actually displaying the result of the model processing on the display unit is correct data.
  • the learning data for example, character data or audio data indicating an instruction to display a high-quality image obtained by a learned model for high-quality image input is used as input data, and an instruction to execute a high-quality image display
  • the learning data may be the correct answer data as the execution instruction for changing the button 2220 to the active state as shown in FIG. 22A and FIG. 22B.
  • the learning data may be any data as long as the instruction content indicated by the character data or the voice data and the execution instruction content correspond to each other. Moreover, you may convert audio
  • machine learning includes deep learning as described above, and a recurrent neural network (RNN: Recurrent Neural Network) can be used as at least a part of the multi-layer neural network, for example.
  • RNN Recurrent Neural Network
  • an RNN that is a neural network that handles time-series information will be described with reference to FIGS. 46A and 46B.
  • LSTM Long @ short-term @ memory
  • FIGS. 47A and 47B Long @ short-term @ memory
  • FIG. 46A shows the structure of RNN which is a machine learning model.
  • RNN4620 has a loop structure to the network, the data x t 4610 at time t, and outputs the data h t 4630. Since the RNN 4620 has a loop function in the network, the state at the current time can be taken over to the next state, so that the time series information can be handled.
  • FIG. 46B shows an example of input / output of the parameter vector at time t.
  • the data x t 4610 includes N data (Params 1 to Params N).
  • the data h t 4630 output from RNN4620 includes data of N (Params1 ⁇ ParamsN) corresponding to the input data.
  • FIG. 47A shows the structure of the LSTM.
  • the information that the network takes over at the next time t is the internal state ct -1 of the network called a cell and the output data ht -1 .
  • FIG. 47B shows details of the LSTM4740.
  • FIG. 47B shows a forgetting gate network FG, an input gate network IG, and an output gate network OG, each of which is a sigmoid layer. Therefore, a vector in which each element takes a value from 0 to 1 is output.
  • the forgetting gate network FG determines how much past information is retained, and the input gate network IG determines which value to update.
  • a cell update candidate network CU is shown, and the cell update candidate network CU is an activation function tanh layer. This creates a vector of new candidate values to be added to the cell.
  • the output gate network OG selects the element of the cell candidate and selects how much information to transmit at the next time.
  • the LSTM model described above is a basic model, and is not limited to the network shown here.
  • the coupling between the networks may be changed.
  • a QRNN Quasi ⁇ Current ⁇ Neural ⁇ Network
  • the machine learning model is not limited to the neural network, and boosting, a support vector machine, or the like may be used.
  • a technology related to natural language processing for example, Sequence to Sequence
  • a dialogue engine a dialogue model, a trained model for a dialogue that responds to the examiner with an output using characters or voices may be applied.
  • the boundary image, the region label image, the high-quality image, and the like may be stored in the storage unit according to an instruction from the operator.
  • any part of the file name for example, the first part or In the last part
  • a file name including information for example, characters
  • image quality improvement image quality improvement process
  • the displayed image is generated by processing using a trained model for improving image quality.
  • a display indicating that the image is a high-quality image may be displayed together with the high-quality image.
  • the operator can easily identify from the display that the displayed high-quality image is not the image itself obtained by shooting, thereby reducing erroneous diagnosis or improving diagnosis efficiency. be able to.
  • the display indicating that the image is a high-quality image generated by the process using the learned model for improving the image quality is a display that can identify the input image and the high-quality image generated by the process. Any mode may be used.
  • the processing using the learned model for high image quality is the results generated by the processing using the type of the learned model.
  • a display indicating the presence may be displayed together with the result.
  • the display indicating that the analysis result is based on the result using the trained model for the image segmentation is displayed. It may be displayed together with the result.
  • a display screen such as a report screen may be stored in the storage unit in accordance with an instruction from the operator.
  • the report screen may be stored in the storage unit as one image in which high-quality images and the like and a display indicating that these images are images generated by processing using the learned model are arranged.
  • the display indicating that the image is a high-quality image generated by processing using the learned model for high image quality is obtained by learning the learning model for high image quality with what learning data.
  • a display indicating whether or not there is may be displayed on the display unit.
  • the display may include an explanation of the types of the input data and the correct answer data of the learning data and an arbitrary display related to the correct answer data such as the imaging part included in the input data and the correct answer data.
  • a display indicating what learning data is used by the type of the learned model is displayed on the display unit. May be done.
  • the information (for example, characters) indicating that the image is generated by the process using the learned model may be displayed or saved in a state of being superimposed on the image or the like.
  • the portion to be superimposed on the image may be any region (for example, the end of the image) that does not overlap with the region where the target region to be photographed is displayed.
  • a non-overlapping area may be determined and superimposed on the determined area.
  • the process using the learned model for improving the image quality but also the image obtained by the process using the various learned models described above, such as the image segmentation process, may be similarly processed.
  • buttons 2220 as shown in FIGS. 22A and 22B When the button 2220 as shown in FIGS. 22A and 22B is set to the active state (the high-quality processing is turned on) as an initial display screen of the report screen, an instruction from the examiner is given. May be configured to transmit a report image corresponding to a report screen including a high-quality image or the like to the server.
  • the button 2220 when the button 2220 is set to the active state by default, at the end of the examination (for example, when the photographing confirmation screen or the preview screen is changed to the report screen in response to an instruction from the examiner).
  • a report image corresponding to a report screen including a high-quality image or the like may be configured to be (automatically) transmitted to the server.
  • various settings in the default settings for example, a depth range for generating an En-Face image on the initial display screen of the report screen, presence / absence of superposition of the analysis map, whether or not the image is a high-quality image, a display screen for follow-up observation
  • the report image generated based on at least one of the settings, such as whether or not the report image may be transmitted to the server. Note that the same processing may be performed when the button 2220 indicates switching of the image segmentation processing.
  • an image for example, a high-quality image, an analysis result such as an analysis map
  • An image, an image indicating an object recognition result, an image indicating a retinal layer, and an image indicating a segmentation result may be input to a second type of trained model different from the first type.
  • a result for example, an analysis result, a diagnosis result, an object recognition result, a detection result of a retinal layer, a segmentation result
  • a result of processing of the first type of learned model (for example, an analysis result, a diagnosis result, an object recognition result, a retinal layer detection result, and a segmentation result) is used. Then, an image to be input to a first type of learned model different from the first type may be generated from an image input to the first type of learned model. At this time, the generated image is likely to be an image suitable as an image to be processed using the second type of learned model.
  • an image obtained by inputting the generated image to the second type of trained model for example, a high-quality image, an image indicating an analysis result such as an analysis map, an image indicating an object recognition result, The accuracy of the image shown and the image showing the segmentation result can be improved.
  • Similar image search using an external database stored in a server or the like may be performed using, as a search key, an analysis result, a diagnosis result, or the like obtained by processing the learned model as described above.
  • the images themselves are used as search keys.
  • a similar image search engine similar image search model, trained model for similar image search
  • Modification 13 Note that the generation processing of the motion contrast data in the above embodiment and the modification is not limited to the configuration performed based on the luminance value of the tomographic image.
  • the various processes include an interference signal acquired by the OCT device 10 or the imaging device 2810, a signal obtained by performing a Fourier transform on the interference signal, a signal obtained by performing an arbitrary process on the signal, and a tomographic image including a tomographic image based on the signals. May be applied to data. In these cases, the same effect as the above configuration can be obtained.
  • the configuration of the OCT device 10 or the imaging device 2810 is not limited to the above configuration, and a part of the configuration included in the OCT device 10 or the imaging device 2810 may be configured separately from the OCT device 10 or the imaging device 2810. .
  • the configuration of the Mach-Zehnder interferometer is used as the interference optical system of the OCT device 10 or the imaging device 2810, but the configuration of the interference optical system is not limited to this.
  • the interference optical system of the OCT apparatus 10 or the imaging apparatus 2810 may have a configuration of a Michelson interferometer.
  • a spectral domain OCT (SD-OCT) device using an SLD as a light source has been described as an OCT device, but the configuration of the OCT device according to the present invention is not limited to this.
  • the present invention can be applied to any other type of OCT device such as a wavelength-swept OCT (SS-OCT) device using a wavelength-swept light source capable of sweeping the wavelength of emitted light.
  • SS-OCT wavelength-swept OCT
  • the present invention can also be applied to a Line-OCT apparatus using line light.
  • the acquisition units 21 and 2801 acquired the interference signal acquired by the OCT device 10 or the imaging device 2810, the three-dimensional tomographic image generated by the image processing device, and the like.
  • the configuration in which the acquisition units 21 and 801 acquire these signals and images is not limited to this.
  • the acquisition units 21 and 801 may acquire these signals from a server or an imaging device connected to the control unit via a LAN, a WAN, the Internet, or the like.
  • the learned model can be provided in the image processing apparatuses 20, 80, 152, 172, and 2800.
  • the learned model can be constituted by, for example, a software module executed by a processor such as a CPU.
  • the learned model may be provided in another server or the like connected to the image processing apparatuses 20, 80, 152, 172, and 2800.
  • the image processing apparatuses 20, 80, 152, 172, and 2800 perform image quality improvement processing using the learned model by connecting to a server having the learned model via an arbitrary network such as the Internet. It can be carried out.
  • the image processed by the image processing device or the image processing method according to the various embodiments and the modified examples described above includes a medical image acquired using an arbitrary modality (imaging device, imaging method).
  • the medical image to be processed can include a medical image acquired by an arbitrary imaging device or the like, and an image created by the image processing apparatus or the image processing method according to the above-described embodiment and the modification.
  • the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject.
  • the medical image may include other parts of the subject.
  • the medical image may be a still image or a moving image, and may be a black and white image or a color image.
  • the medical image may be an image representing the structure (form) of the predetermined part or an image representing the function thereof.
  • the images representing functions include, for example, images representing blood flow dynamics (blood flow, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasonic Doppler images.
  • the predetermined site of the subject may be determined according to the imaging target, and includes organs such as the human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, and liver, head, chest, Includes any parts such as legs and arms.
  • the medical image may be a tomographic image of the subject or a front image.
  • the front image is, for example, at least a part of the fundus front image, the front image of the anterior eye part, the fundus image obtained by fluorescence imaging, and data obtained by OCT (three-dimensional OCT data) in the depth direction of the imaging target.
  • OCT three-dimensional OCT data
  • the En-Face image is an OCTA En-Face image (motion contrast front image) generated using three-dimensional OCTA data (three-dimensional motion contrast data) using data in at least a part of the range in the depth direction of the imaging target. ).
  • the three-dimensional OCT data and the three-dimensional motion contrast data are examples of three-dimensional medical image data.
  • the imaging device is a device for imaging an image used for diagnosis.
  • the imaging apparatus detects, for example, a device that obtains an image of a predetermined portion by irradiating a predetermined portion of the subject with light, radiation such as X-rays, electromagnetic waves, or ultrasonic waves, or detects radiation emitted from a subject.
  • a device for obtaining an image of a predetermined part More specifically, the imaging apparatuses according to the various embodiments and modifications described above include at least an X-ray imaging apparatus, a CT apparatus, an MRI apparatus, a PET apparatus, a SPECT apparatus, an SLO apparatus, an OCT apparatus, an OCTA apparatus, and a fundus. It includes a camera and an endoscope.
  • the OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. Further, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a wavelength sweep type OCT (SS-OCT) device. Further, the SLO device or OCT device may include a wavefront compensation SLO (AO-SLO) device using a wavefront compensation optical system, a wavefront compensation OCT (AO-OCT) device, or the like. Further, the SLO device and the OCT device may include a polarization SLO (PS-SLO) device, a polarization OCT (PS-OCT) device, and the like for visualizing information on a polarization phase difference and depolarization.
  • TD-OCT time domain OCT
  • FD-OCT Fourier domain OCT
  • SD-OCT spectral domain OCT
  • SS-OCT wavelength sweep type OCT
  • the SLO device or OCT device may include
  • the trained model for evaluating the region label image, improving the image quality, analyzing the image, and generating the diagnosis result the magnitude of the brightness value of the tomographic image, the order and inclination, position, distribution, It can be considered that continuity and the like are extracted as a part of the feature amount and are used in the estimation processing.
  • Embodiment 1 of the present disclosure relates to a medical image processing apparatus.
  • the medical image processing apparatus includes: an acquisition unit configured to acquire a tomographic image of an eye to be inspected; and a learned model obtained by learning data indicating at least one retinal layer among a plurality of retinal layers in the tomographic image of the eye to be inspected. And a first processing unit that executes a first detection process for detecting at least one retinal layer among a plurality of retinal layers in the acquired tomographic image.
  • Embodiment 2 includes the medical image processing apparatus according to Embodiment 1, and uses at least one retinal layer among the plurality of retinal layers in the acquired tomographic image without using a learned model obtained by machine learning.
  • the image processing apparatus further includes a second processing unit that executes a second detection process for detecting.
  • Embodiment 3 includes the medical image processing apparatus according to Embodiment 2, wherein the second detection processing is at least one retinal layer other than at least one retinal layer detected by executing the first detection processing. This is a process of detecting a layer.
  • Embodiment 4 includes the medical image processing apparatus according to Embodiment 2 or 3, wherein the first detection processing is processing for detecting a retinal region in the acquired tomographic image as the at least one retinal layer.
  • the second detection process is a process of detecting at least one retinal layer in a retinal region detected by executing the first detection process.
  • a fifth embodiment includes the medical image processing apparatus according to any one of the second to fourth embodiments, wherein the first detection processing is performed based on a boundary between an inner limiting membrane and a nerve fiber layer of an eye to be examined, and a photoreceptor inner segment. Outer segment junction, retinal pigment epithelium layer, and a process to detect the layer up to any of Bruch's membrane, the second detection process, at least one of the layers detected by the first detection process This is a process for detecting a retinal layer.
  • a sixth embodiment includes the medical image processing device according to any one of the second to fifth embodiments, wherein the second processing unit performs the second detection after the first detection processing by the first processing unit. The second detection process is executed.
  • a seventh embodiment includes the medical image processing device according to the second embodiment, and further includes a display control unit that controls a display unit, wherein the first detection process and the second detection process detect the same retinal layer
  • the display control unit causes the display unit to display processing results of the first detection processing and the second detection processing.
  • An eighth embodiment includes the medical image processing apparatus according to the seventh embodiment, and the display control unit causes the display unit to display a mismatched portion between the processing results of the first detection process and the second detection process.
  • a ninth embodiment includes the medical image processing apparatus according to the seventh or eighth embodiment, wherein the first detection processing and the second detection processing are performed using a photoreceptor cell from a boundary between an inner limiting membrane and a nerve fiber layer of an eye to be examined. Inner / outer segment junction, retinal pigment epithelium layer, and processing to detect the layer up to any of Bruch's membrane, the second processing unit, according to an instruction of the operator, the first detection processing And a third detection process of detecting at least one retinal layer between the layers detected by one of the second detection processes.
  • a tenth embodiment includes the medical image processing apparatus according to any one of the second to ninth embodiments, wherein the first detection process and the second detection process are performed based on imaging conditions for the acquired tomographic image. And a selecting unit for selecting at least one of the above.
  • An eleventh embodiment includes the medical image processing apparatus according to any one of the second to tenth embodiments, wherein the first processing unit is configured to execute a plurality of learned models on which machine learning has been performed using different learning data.
  • the first detection process is performed using a learned model in which machine learning has been performed using learning data corresponding to imaging conditions for the acquired tomographic image.
  • Embodiment 12 The medical image processing apparatus according to Embodiment 10 or 11, wherein the imaging conditions include at least one of an imaging region, an imaging method, an imaging region, an imaging angle of view, and an image resolution.
  • the thirteenth embodiment includes the medical image processing apparatus according to any one of the second to twelfth embodiments, and measures a shape characteristic of the eye to be inspected based on a result of the first detection processing and the second detection processing. Is done.
  • Embodiment 14 includes the medical image processing apparatus according to any one of Embodiments 1 to 13, and corrects the structure of the retinal layer detected by the first processing unit based on medical characteristics in the retinal layer. A correction unit is further provided.
  • a fifteenth embodiment includes the medical image processing apparatus according to any one of the first to fourteenth embodiments, wherein the first processing unit uses the learned model to determine in advance an input image for each imaging region using the learned model. Detects defined boundaries.
  • Embodiment 16 includes the medical image processing apparatus according to any one of Embodiments 1 to 15, wherein at least a part of the depth range in the three-dimensional tomographic image of the subject's eye is included, and the at least one detected
  • the image processing apparatus further includes a generation unit that generates a front image corresponding to the depth range determined based on the retinal layer.
  • Embodiment 17 includes the medical image processing apparatus according to Embodiment 16, wherein the generating unit corresponds to the determined depth range using three-dimensional motion contrast data corresponding to the three-dimensional tomographic image. Generate a motion contrast front image.
  • the eighteenth embodiment includes the medical image processing apparatus according to any one of the first to fifteenth embodiments, and uses the learned model for improving the image quality to convert the acquired tomographic image from the acquired tomographic image.
  • the image processing apparatus further includes a generation unit that generates a tomographic image having a higher image quality than that of the first tomographic image.
  • the first processing unit performs the first detection process on the generated tomographic image.
  • a nineteenth embodiment includes the medical image processing device according to any one of the first to eighteenth embodiments, and corrects a retinal layer information detected by the first processing unit in accordance with an instruction of an operator.
  • the modified retinal layer information is used for additional learning on the learned model used by the first processing unit.
  • the twentieth embodiment includes the medical image processing apparatus according to any one of the first to eighteenth embodiments, and uses a learned model for generating a diagnosis result to execute the first detection process and obtain a result. And a diagnostic result generation unit that generates a diagnostic result of the acquired tomographic image.
  • Embodiment 21 relates to a medical image processing method.
  • the medical image processing method includes a step of acquiring a tomographic image of the eye to be inspected, and a first step of detecting at least one retinal layer among a plurality of retinal layers of the eye to be inspected in the tomographic image using the learned model. And performing a detection process.
  • ⁇ Embodiment 22 ⁇ relates to a program.
  • the program when executed by the processor, causes the processor to execute the steps of the medical image processing method according to the twenty-first embodiment.
  • the medical image processing apparatus includes: a segmentation processing unit configured to generate region information capable of identifying an anatomical region from an input image that is a tomographic image of a predetermined part of a subject using a segmentation engine including a learned model; and And an evaluation unit that evaluates the region information by using an evaluation engine that includes a learned model or an evaluation engine that performs a knowledge base process using anatomical knowledge.
  • a further embodiment 2 includes the medical image processing apparatus according to the further embodiment 1, further comprising a photographing condition acquiring unit for acquiring photographing conditions of the input image, wherein the segmentation processing unit is configured to perform the operation based on the photographing conditions.
  • a plurality of segmentation engines including different learned models are switched and used.
  • a further embodiment 3 includes the medical image processing apparatus according to the further embodiment 2, wherein the imaging condition acquisition unit uses an imaging location estimation engine including a learned model to acquire an imaging region and an imaging region from the input image. Is estimated.
  • a further embodiment 4 includes the medical image processing apparatus according to any one of the further embodiments 1 to 3, wherein the segmentation processing unit converts the image size of the input image into an image which can be handled by the segmentation engine. Adjust the size and input to the segmentation engine.
  • a further embodiment 5 includes the medical image processing apparatus according to any one of the further embodiments 1 to 3, wherein the segmentation processing unit is configured to determine whether the image size of the input image can be handled by the segmentation engine. An image obtained by padding the input image so as to have a size is input to a segmentation engine.
  • a further embodiment 6 includes the medical image processing apparatus according to any one of the further embodiments 1 to 3, wherein the segmentation processing unit divides the input image into images of a plurality of regions, and Each segment image is input to the segmentation engine.
  • a further embodiment 7 includes the medical image processing apparatus according to any one of the further embodiments 1 to 6, wherein the evaluation unit determines whether or not to output the area information according to a result of the evaluation. to decide.
  • a further embodiment 8 includes the medical image processing apparatus according to any one of the further embodiments 1 to 7, wherein the segmentation processing unit uses a plurality of segmentation engines each including a different learned model.
  • the plurality of area information is generated from an input image, and the evaluation unit determines at least one of the plurality of area information determined to be output by evaluating the plurality of area information in accordance with a user's instruction. select.
  • a further embodiment 9 includes the medical image processing device according to any one of the further embodiments 1 to 7, wherein the segmentation processing unit uses a plurality of segmentation engines each including a different learned model, and The plurality of area information is generated from an input image, and the evaluation unit evaluates the plurality of area information based on a predetermined selection criterion, and determines at least one of the plurality of area information determined to be output. Select
  • a further embodiment 10 includes the medical image processing device according to any one of the further embodiments 1 to 9, and determines whether or not the area information can be generated from the input image using the segmentation engine. And a determining unit for performing the determination.
  • a further embodiment 11 includes the medical image processing apparatus according to any one of the further embodiments 1 to 10, wherein the segmentation processing unit converts the input image into a plurality of images having a dimension lower than the dimension of the input image.
  • the image is divided into images, and the divided images are input to the segmentation engine.
  • a further embodiment 12 includes the medical image processing apparatus according to the further embodiment 11, wherein the segmentation processing unit processes the plurality of images in parallel using a plurality of segmentation engines.
  • a further embodiment 13 includes the medical image processing apparatus according to any one of the further embodiments 1 to 12, wherein the area information is a label image in which an area label is assigned to each pixel.
  • a further embodiment 14 includes the medical image processing apparatus according to the further embodiment 13, wherein the segmentation engine inputs a tomographic image and outputs the label image.
  • a further embodiment 15 includes the medical image processing apparatus according to the further embodiment 14, wherein the trained model of the segmentation engine receives a tomographic image including two or more layers as input data and corresponds to the tomographic image. This is a model in which learning is performed using a label image as output data.
  • a further embodiment 16 includes the medical image processing device according to any one of the further embodiments 1 to 15, wherein the input image is obtained from an imaging device, or the predetermined image of the subject is obtained from the imaging device. Obtaining part data and obtaining the input image based on the data.
  • a further embodiment 17 includes the medical image processing apparatus according to any one of the further embodiments 1 to 15, wherein the input image is obtained from an image management system, and the region information is output to the image management system. Or, the input image is obtained from the image management system, and the area information is output to the image management system.
  • a further embodiment 18 includes the medical image processing apparatus according to any one of the further embodiments 1 to 17, and further includes a correction unit that corrects the region information by anatomical knowledge-based processing.
  • a further embodiment 19 includes the medical image processing apparatus according to any one of the further embodiments 1 to 18, and performs image analysis of the input image using the area information output from the evaluation unit. An analysis unit is further provided.
  • a further embodiment 20 includes the medical image processing apparatus according to any one of the further embodiments 1 to 19, and outputs that the area information is information generated using a learned model.
  • a further embodiment 21 relates to a medical image processing method.
  • the medical image processing method includes using a segmentation engine including a learned model to generate region information capable of identifying an anatomical region from an input image that is a tomographic image of a predetermined part of a subject; Evaluating the region information using an evaluation engine that includes a completed model or a knowledge-based evaluation engine that uses anatomical knowledge.
  • a further embodiment 22 relates to a program.
  • the program when executed by the processor, causes the processor to execute the steps of the medical image processing method according to the further embodiment 21.
  • the present invention provides a program for realizing one or more functions of the above-described embodiments and modifications to a system or an apparatus via a network or a storage medium, and a computer of the system or the apparatus reads and executes the program. It is feasible.
  • a computer has one or more processors or circuits, and can include separate computers or a network of separate processors or circuits, to read and execute computer-executable instructions.
  • a processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).
  • CPU central processing unit
  • MPU microprocessing unit
  • GPU graphics processing unit
  • ASIC application specific integrated circuit
  • FPGA field programmable gateway
  • DSP digital signal processor
  • DFP data flow processor
  • NPU neural processing unit

Abstract

This image processing device comprises an acquisition unit that acquires tomographic images of an eye under examination, and a first processing unit that performs a first detection process for detecting at least one retinal layer among a plurality of retinal layers in the acquired tomographic images using a pre-trained model obtained by training with data in which at least one retinal layer among a plurality of retinal layers is shown in tomographic images of eyes under examination.

Description

医用画像処理装置、医用画像処理方法及びプログラムMedical image processing apparatus, medical image processing method, and program
　本発明は、医用画像処理装置、医用画像処理方法及びプログラムに関する。 The present invention relates to a medical image processing device, a medical image processing method, and a program.
　光干渉断層撮影法（ＯＣＴ：Ｏｐｔｉｃａｌ　Ｃｏｈｅｒｅｎｃｅ　Ｔｏｍｏｇｒａｐｈｙ）を用いた装置（ＯＣＴ装置）などの眼部の断層画像撮影装置は、網膜層内部の状態を三次元的に観察することが可能である。この断層画像撮影装置は、疾病の診断をより的確に行うのに有用であることから近年注目を集めている。 An ophthalmic tomographic imaging apparatus such as an apparatus (OCT apparatus) using optical coherence tomography (OCT: Optical Coherence Tomography) can three-dimensionally observe a state inside a retinal layer. This tomographic imaging apparatus has been attracting attention in recent years because it is useful for more accurately diagnosing diseases.
　ＯＣＴの形態として、例えば、広帯域な光源とマイケルソン干渉計を組み合わせたＴＤ－ＯＣＴ（Ｔｉｍｅ ｄｏｍａｉｎ ＯＣＴ）がある。これは、参照アームの遅延を走査することで、信号アームの後方散乱光との干渉光を計測し、深さ分解の情報を得るように構成されている。しかしながら、このようなＴＤ－ＯＣＴでは高速な画像取得は難しい。 As a form of the OCT, for example, there is a TD-OCT (Time domain OCT) that combines a broadband light source and a Michelson interferometer. This is configured to scan the delay of the reference arm, measure the interference light with the backscattered light of the signal arm, and obtain the information of the depth resolution. However, high-speed image acquisition is difficult with such TD-OCT.
　そのため、より高速に画像を取得する方法としては、広帯域光源を用い、分光器でインターフェログラムを取得する手法によるＳＤ－ＯＣＴ（Ｓｐｅｃｔｒａｌ ｄｏｍａｉｎ ＯＣＴ）が知られている。また、光源として、高速波長掃引光源を用い、単一チャネル光検出器でスペクトル干渉を計測する手法によるＳＳ－ＯＣＴ（Ｓｗｅｐｔ Ｓｏｕｒｃｅ ＯＣＴ）が知られている。 Therefore, as a method of acquiring an image at a higher speed, SD-OCT (Spectral domain OCT) using a technique of acquiring an interferogram with a spectroscope using a broadband light source is known. In addition, SS-OCT (Swept \ Source \ OCT) is known, which uses a high-speed wavelength sweep light source as a light source and measures spectral interference with a single-channel photodetector.
　ＯＣＴで撮影された断層画像が取得された場合には、神経線維層の厚みを計測できれば、緑内障などの疾病の進行度や治療後の回復具合を定量的に診断することができる。これらの層の厚みを定量的に計測するために、コンピュータを用いて断層画像から網膜の各層の境界を検出し、各層の厚みを計測する技術が、特許文献１に開示されている。 When a tomographic image taken by OCT is acquired, the progress of disease such as glaucoma and the degree of recovery after treatment can be quantitatively diagnosed if the thickness of the nerve fiber layer can be measured. In order to quantitatively measure the thicknesses of these layers, a technique of detecting boundaries of each layer of the retina from a tomographic image using a computer and measuring the thickness of each layer is disclosed in Patent Document 1.
特開２００８－７３０９９号公報JP 2008-73099 A
　しかしながら、従来の技術では以下の問題があった。疾患眼においては、層の消失、出血、及び白斑や新生血管の発生などがあるため、網膜の形状が不規則となる。そのため、画像特徴抽出の結果を、網膜の形状の規則性を利用して判断し、網膜層の境界検出を行う従来の画像処理方法では、網膜層の境界検出を自動で行う際に誤検出などが発生するという限界があった。 However, the conventional technology has the following problems. In a diseased eye, the shape of the retina becomes irregular due to loss of layers, bleeding, and the occurrence of vitiligo and new blood vessels. Therefore, in the conventional image processing method of determining the result of the image feature extraction using the regularity of the shape of the retina and detecting the boundary of the retinal layer, erroneous detection or the like occurs when automatically detecting the boundary of the retinal layer. There is a limit that occurs.
　そこで、本発明は、疾患や部位等によらず網膜層の境界検出を行うことができる医用画像処理装置、医用画像処理方法及びプログラムを提供することを目的の一つとする。 Therefore, an object of the present invention is to provide a medical image processing apparatus, a medical image processing method, and a program that can detect a boundary of a retinal layer regardless of a disease, a part, or the like.
　本発明の一実施態様による医用画像処理装置は、被検眼の断層画像を取得する取得部と、被検眼の断層画像において複数の網膜層のうち少なくとも一つの網膜層が示されたデータを学習して得た学習済モデルを用いて、前記取得された断層画像において複数の網膜層のうち少なくとも一つの網膜層を検出するための第一の検出処理を実行する第一の処理部とを備える。 The medical image processing apparatus according to an embodiment of the present invention learns an acquisition unit that acquires a tomographic image of an eye to be inspected, and data indicating at least one retinal layer among a plurality of retinal layers in the tomographic image of the eye to be inspected. A first processing unit that executes a first detection process for detecting at least one retinal layer among a plurality of retinal layers in the acquired tomographic image using the learned model obtained in the above.
　また、本発明の他の実施態様に係る医用画像処理方法は、被検眼の断層画像を取得する工程と、被検眼の断層画像において複数の網膜層のうち少なくとも一つの網膜層が示されたデータを学習して得た学習済モデルを用いて、前記取得された断層画像において複数の網膜層のうち少なくとも一つの網膜層を検出するための第一の検出処理を実行する工程とを含む。 Further, a medical image processing method according to another embodiment of the present invention includes a step of acquiring a tomographic image of the eye to be inspected, and data showing at least one retinal layer among a plurality of retinal layers in the tomographic image of the eye to be inspected. Executing a first detection process for detecting at least one retinal layer among a plurality of retinal layers in the acquired tomographic image, using a learned model obtained by learning the above.
　本発明のさらなる特徴が、添付の図面を参照して以下の例示的な実施例の説明から明らかになる。 Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings.
実施例１に係る画像処理システムの概略的な構成の一例を示す。1 illustrates an example of a schematic configuration of an image processing system according to a first embodiment. 眼部を説明するための図である。It is a figure for explaining an eye part. 断層画像を説明するための図である。It is a figure for explaining a tomographic image. 眼底画像を説明するための図である。It is a figure for explaining a fundus image. 実施例１に係る一連の処理のフローチャートである。6 is a flowchart of a series of processes according to the first embodiment. 学習画像の例を説明するための図である。It is a figure for explaining an example of a learning image. 学習画像の例を説明するための図である。It is a figure for explaining an example of a learning image. 学習画像のサイズの例を説明するための図である。It is a figure for explaining the example of the size of a learning image. 学習画像のサイズの例を説明するための図である。It is a figure for explaining the example of the size of a learning image. 学習画像のサイズの例を説明するための図である。It is a figure for explaining the example of the size of a learning image. 実施例１に係る機械学習モデルの例を説明するための図である。FIG. 4 is a diagram for describing an example of a machine learning model according to the first embodiment. 表示画面の一例を示す。4 shows an example of a display screen. 実施例２に係る画像処理システムの概略的な構成の一例を示す。9 illustrates an example of a schematic configuration of an image processing system according to a second embodiment. 実施例２に係る一連の処理のフローチャートである。9 is a flowchart of a series of processes according to the second embodiment. 実施例２に係る境界検出処理のフローチャートである。9 is a flowchart of a boundary detection process according to the second embodiment. 網膜領域の検出を説明するための図である。FIG. 3 is a diagram for explaining detection of a retinal region. 網膜領域の検出を説明するための図である。FIG. 3 is a diagram for explaining detection of a retinal region. 学習画像のサイズの例を説明するための図である。It is a figure for explaining the example of the size of a learning image. 学習画像のサイズの例を説明するための図である。It is a figure for explaining the example of the size of a learning image. 学習画像のサイズの例を説明するための図である。It is a figure for explaining the example of the size of a learning image. 実施例２に係る機械学習モデルの例を説明するための図である。FIG. 14 is a diagram for describing an example of a machine learning model according to a second embodiment. 実施例２に係る網膜層検出を説明するための図である。FIG. 9 is a diagram for explaining retinal layer detection according to a second embodiment. 実施例２に係る網膜層検出を説明するための図である。FIG. 9 is a diagram for explaining retinal layer detection according to a second embodiment. 実施例２に係る網膜層検出を説明するための図である。FIG. 9 is a diagram for explaining retinal layer detection according to a second embodiment. 実施例２に係る網膜層検出を説明するための図である。FIG. 9 is a diagram for explaining retinal layer detection according to a second embodiment. 学習済モデルにおける入力と出力画像の例を説明するための図である。It is a figure for explaining an example of an input and output picture in a learned model. 学習済モデルにおける入力と出力画像の例を説明するための図である。It is a figure for explaining an example of an input and output picture in a learned model. 学習済モデルにおける入力と出力画像の例を説明するための図である。It is a figure for explaining an example of an input and output picture in a learned model. 学習済モデルにおける入力と出力画像の例を説明するための図である。It is a figure for explaining an example of an input and output picture in a learned model. 実施例４に係る画像処理システムの概略的な構成の一例を示す。14 illustrates an example of a schematic configuration of an image processing system according to a fourth embodiment. 実施例４に係る一連の処理のフローチャートである。13 is a flowchart of a series of processes according to a fourth embodiment. 実施例４に係る一連の処理のフローチャートである。13 is a flowchart of a series of processes according to a fourth embodiment. 実施例５に係る画像処理システムの概略的な構成の一例を示す。15 shows an example of a schematic configuration of an image processing system according to Embodiment 5. 実施例５に係る一連の処理のフローチャートである。19 is a flowchart of a series of processes according to a fifth embodiment. 実施例５に係る境界検出処理のフローチャートである。19 is a flowchart of a boundary detection process according to the fifth embodiment. 網膜領域の補正処理を説明するための図である。It is a figure for explaining correction processing of a retinal area. 網膜領域の補正処理を説明するための図である。It is a figure for explaining correction processing of a retinal area. 網膜領域の補正処理を説明するための図である。It is a figure for explaining correction processing of a retinal area. 網膜領域の補正処理を説明するための図である。It is a figure for explaining correction processing of a retinal area. 実施例６に係る学習画像の例を説明するための図である。FIG. 19 is a diagram for describing an example of a learning image according to a sixth embodiment. 複数のＯＣＴＡのＥｎ－Ｆａｃｅ画像の一例を示す。5 shows an example of En-Face images of a plurality of OCTAs. 複数の輝度の断層画像の一例を示す。4 shows an example of a tomographic image having a plurality of luminances. 実施例７に係るユーザーインターフェースの一例を示す。17 illustrates an example of a user interface according to a seventh embodiment. 実施例７に係るユーザーインターフェースの一例を示す。17 illustrates an example of a user interface according to a seventh embodiment. 実施例７に係るユーザーインターフェースの一例を示す。17 illustrates an example of a user interface according to a seventh embodiment. 用語の説明に係る領域ラベル画像の一例を示す。13 shows an example of an area label image according to the explanation of terms. 用語の説明に係るニューラルネットワークの構成の一例を示す。1 shows an example of the configuration of a neural network according to the explanation of terms. 用語の説明に係るニューラルネットワークの構成の一例を示す。1 shows an example of the configuration of a neural network according to the explanation of terms. 用語の説明に係る領域ラベル画像の一例を示す。13 shows an example of an area label image according to the explanation of terms. 実施例８に係る画像処理装置の構成の一例を示す。17 shows an example of the configuration of an image processing apparatus according to an eighth embodiment. 実施例８に係る画像処理装置の処理の流れの一例を示すフローチャートである。19 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to the eighth embodiment. 実施例８に係る画像処理装置の処理の流れの一例を示すフローチャートである。19 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to the eighth embodiment. 実施例８に係る撮影装置が備えるユーザーインターフェースの一例を示す図である。FIG. 18 is a diagram illustrating an example of a user interface provided in the imaging device according to the eighth embodiment. 実施例８に係る撮影装置が備えるユーザーインターフェースの一例を示す図である。FIG. 18 is a diagram illustrating an example of a user interface provided in the imaging device according to the eighth embodiment. 実施例９に係る画像処理装置の処理の流れの一例を示すフローチャートである。19 is a flowchart illustrating an example of a flow of a process of the image processing apparatus according to the ninth embodiment. 実施例１１に係る画像処理を示す。21 shows image processing according to the eleventh embodiment. 実施例１１に係る画像処理装置の処理の流れの一例を示すフローチャートである。33 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to the eleventh embodiment. 実施例１２に係る画像処理を示す。14 shows image processing according to Embodiment 12. 実施例１３に係る画像処理装置の処理の流れの一例を示すフローチャートである。33 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to Embodiment 13. 実施例１３に係る画像処理を示す。14 shows image processing according to Embodiment 13. 実施例１３に係る画像処理装置の処理の流れの一例を示すフローチャートである。33 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to Embodiment 13. 実施例１３に係る画像処理を示す。14 shows image processing according to Embodiment 13. 実施例１４に係る画像処理装置の処理の流れの一例を示すフローチャートである。24 is a flowchart illustrating an example of the flow of a process of the image processing apparatus according to Embodiment 14. 実施例１５に係る撮影装置が備えるユーザーインターフェースの一例を示す図である。FIG. 30 is a diagram illustrating an example of a user interface provided in the imaging device according to Embodiment 15. 実施例１８に係る画像処理装置の構成の一例を示す。FIG. 47 shows an example of the configuration of the image processing apparatus according to Embodiment 18. FIG. 実施例１９に係る画像処理装置の構成の一例を示す。90 illustrates an example of the configuration of an image processing apparatus according to Embodiment 19. 実施例１９に係る画像処理装置の処理の流れの一例を示すフローチャートである。FIG. 39 is a flowchart illustrating an example of the flow of the process of the image processing apparatus according to Embodiment 19. 変形例９に係る機械学習モデルとして用いられるニューラルネットワークの構成の一例を示す。15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. 変形例９に係る機械学習モデルとして用いられるニューラルネットワークの構成の一例を示す。15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. 変形例９に係る機械学習モデルとして用いられるニューラルネットワークの構成の一例を示す。15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9. 変形例９に係る機械学習モデルとして用いられるニューラルネットワークの構成の一例を示す。15 shows an example of the configuration of a neural network used as a machine learning model according to Modification 9.
　以下、本発明を実施するための例示的な実施例を、図面を参照して詳細に説明する。 Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings.
　ただし、以下の実施例で説明する寸法、材料、形状、及び構成要素の相対的な位置等は任意であり、本発明が適用される装置の構成又は様々な条件に応じて変更できる。また、図面において、同一であるか又は機能的に類似している要素を示すために図面間で同じ参照符号を用いる。 However, dimensions, materials, shapes, relative positions of components, and the like described in the following embodiments are arbitrary, and can be changed according to the configuration of an apparatus to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate the same or functionally similar elements.
（実施例１）
　以下、図１乃至図７を参照して、本発明の実施例１に係る、眼部の断層画像を用いた画像処理装置を備える画像処理システムについて説明する。本実施例では、機械学習モデルに関する学習済モデルを用いて対象となる全ての網膜層の検出を行う。なお、以下において、機械学習モデルとは、ディープラーニング等の機械学習アルゴリズムによる学習モデルをいう。また、学習済モデルとは、任意の機械学習アルゴリズムによる機械学習モデルに対して、事前に適切な教師データを用いてトレーニングした（学習を行った）モデルである。ただし、学習済モデルは、それ以上の学習を行わないものではなく、追加の学習を行うこともできるものとする。なお、以下において、教師データとは、学習データのことをいい、入力データ及び出力データのペアで構成される。また、正解データとは、学習データ（教師データ）の出力データのことをいう。 (Example 1)
Hereinafter, an image processing system including an image processing apparatus using a tomographic image of an eye according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 7. In the present embodiment, all target retinal layers are detected using a learned model relating to a machine learning model. In the following, a machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. The trained model is a model that has been trained (learned) in advance by using appropriate teacher data with respect to a machine learning model using an arbitrary machine learning algorithm. However, it is assumed that the learned model does not perform any further learning and can perform additional learning. In the following, teacher data refers to learning data, and is composed of a pair of input data and output data. The correct data refers to output data of learning data (teacher data).
　図１は、本実施例に係る画像処理装置２０（医用画像処理装置）を備える画像処理システム１の概略的な構成の一例を示す。図１に示すように、画像処理システム１には、断層画像撮影装置の一例であるＯＣＴ装置１０、画像処理装置２０、眼底画像撮影装置３０、外部記憶装置４０、表示部５０、及び入力部６０が設けられている。 FIG. 1 shows an example of a schematic configuration of an image processing system 1 including an image processing device 20 (medical image processing device) according to the present embodiment. As shown in FIG. 1, the image processing system 1 includes an OCT apparatus 10, an image processing apparatus 20, a fundus image capturing apparatus 30, an external storage device 40, a display unit 50, and an input unit 60, which are examples of a tomographic image capturing apparatus. Is provided.
　ＯＣＴ装置１０は、被検眼の断層画像を撮影するための装置である断層画像撮影装置の一例である。ＯＣＴ装置としては、任意の種類のＯＣＴ装置を用いることができ、例えばＳＤ－ＯＣＴやＳＳ－ＯＣＴを用いることができる。 The OCT apparatus 10 is an example of a tomographic image capturing apparatus that captures a tomographic image of an eye to be inspected. Any type of OCT device can be used as the OCT device, and for example, SD-OCT or SS-OCT can be used.
　画像処理装置２０は、インターフェースを介してＯＣＴ装置１０、眼底画像撮影装置３０、外部記憶装置４０、表示部５０、及び入力部６０と接続されており、これらを制御することができる。画像処理装置２０は、ＯＣＴ装置１０、眼底画像撮影装置３０、及び外部記憶装置４０から取得する各種信号に基づいて、被検眼の断層画像やＥｎ－Ｆａｃｅ画像（正面画像）等の各種画像を生成することができる。また、画像処理装置２０は、これら画像について画像処理を施すことができる。なお、画像処理装置２０は、汎用のコンピュータによって構成されてもよいし、画像処理システム１の専用のコンピュータによって構成されてもよい。 The image processing device 20 is connected to the OCT device 10, the fundus image photographing device 30, the external storage device 40, the display unit 50, and the input unit 60 via an interface, and can control them. The image processing device 20 generates various images such as a tomographic image and an En-Face image (front image) of the subject's eye based on various signals acquired from the OCT device 10, the fundus image capturing device 30, and the external storage device 40. can do. Further, the image processing device 20 can perform image processing on these images. Note that the image processing apparatus 20 may be configured by a general-purpose computer, or may be configured by a computer dedicated to the image processing system 1.
　眼底画像撮影装置３０は、被検眼の眼底画像を撮影するための装置であり、当該装置としては、例えば、眼底カメラやＳＬＯ（Ｓｃａｎｎｉｎｇ　Ｌａｓｅｒ　Ｏｐｈｔｈａｌｍｏｓｃｏｐｅ）等を用いることができる。なお、ＯＣＴ装置１０と眼底画像撮影装置３０の装置構成は、一体型でもよいし別体型でもよい。 The fundus image photographing device 30 is a device for photographing a fundus image of the eye to be inspected. As the device, for example, a fundus camera, an SLO (Scanning Laser Ophthalmoscope), or the like can be used. In addition, the device configuration of the OCT device 10 and the fundus image photographing device 30 may be an integrated type or a separate type.
　外部記憶装置４０は、被検眼に関する情報（患者の氏名、年齢、性別等）と、撮影した各種画像データ、撮影パラメータ、画像解析パラメータ、及び操作者によって設定されたパラメータをそれぞれ関連付けて保持している。外部記憶装置４０は、任意の記憶装置によって構成されてよく、例えば、光学ディスクやメモリ等の記憶媒体によって構成されてよい。 The external storage device 40 stores information relating to the subject's eye (patient's name, age, gender, etc.) in association with various types of captured image data, imaging parameters, image analysis parameters, and parameters set by the operator. I have. The external storage device 40 may be configured by an arbitrary storage device, for example, may be configured by a storage medium such as an optical disk or a memory.
　表示部５０は、任意のディスプレイによって構成され、画像処理装置２０による制御に従い、被検眼に関する情報や各種画像を表示することができる。 The display unit 50 is configured by an arbitrary display, and can display information about the eye to be inspected and various images under the control of the image processing device 20.
　入力部６０は、例えば、マウス、キーボード、又はタッチ操作画面などであり、操作者は、入力部６０を介して、画像処理装置２０やＯＣＴ装置１０、眼底画像撮影装置３０への指示を画像処理装置２０に入力することができる。なお、入力部６０をタッチ操作画面とする場合には、入力部６０を表示部５０と一体として構成することができる。 The input unit 60 is, for example, a mouse, a keyboard, a touch operation screen, or the like. The operator performs image processing on the instruction to the image processing device 20, the OCT device 10, and the fundus image capturing device 30 via the input unit 60. It can be input to the device 20. When the input unit 60 is a touch operation screen, the input unit 60 can be configured integrally with the display unit 50.
　なお、これら構成要素は、図１では別体として示されているが、これら構成要素の一部又は全部を一体として構成してもよい。 Although these components are shown separately in FIG. 1, some or all of these components may be integrally formed.
　次にＯＣＴ装置１０について説明する。ＯＣＴ装置１０には、光源１１、ガルバノミラー１２、フォーカスレンズステージ１３、コヒーレンスゲートステージ１４、ディテクタ１５、及び内部固視灯１６が設けられている。なお、ＯＣＴ装置１０は既知の装置であるため詳細な説明は省略し、ここでは、画像処理装置２０からの指示により行われる断層画像の撮影について説明を行う。 Next, the OCT apparatus 10 will be described. The OCT apparatus 10 includes a light source 11, a galvanometer mirror 12, a focus lens stage 13, a coherence gate stage 14, a detector 15, and an internal fixation lamp 16. Since the OCT apparatus 10 is a known apparatus, a detailed description thereof will be omitted, and here, a description will be given of tomographic image capturing performed in accordance with an instruction from the image processing apparatus 20.
　画像処理装置２０から撮影の指示が伝えられると、光源１１が光を出射する。光源１１からの光は不図示の分割部を用いて測定光と参照光に分割される。ＯＣＴ装置１０では、測定光を被検体（被検眼）に照射し、被検体からの戻り光と、参照光との干渉光を検出することで、被検体の断層情報を含む干渉信号を生成することができる。 (4) When a photographing instruction is transmitted from the image processing device 20, the light source 11 emits light. The light from the light source 11 is split into measurement light and reference light using a splitting unit (not shown). The OCT apparatus 10 generates an interference signal including tomographic information of the subject by irradiating the subject (eye to be examined) with measurement light and detecting interference light between the return light from the subject and the reference light. be able to.
　ガルバノミラー１２は、測定光を被検眼の眼底において走査するために用いられ、ガルバノミラー１２による測定光の走査範囲により、ＯＣＴ撮影による眼底の撮影範囲を規定することができる。画像処理装置２０は、ガルバノミラー１２の駆動範囲及び速度を制御することで、眼底における平面方向の撮影範囲及び走査線数（平面方向の走査速度）を規定することができる。図１では、説明を簡略化するため、ガルバノミラー１２を１つのユニットとして示したが、ガルバノミラー１２は、実際にはＸスキャン用のミラーとＹスキャン用の２枚のミラーで構成され、眼底上における所望の範囲を測定光で走査できる。なお、測定光を走査するための走査部の構成はガルバノミラーに限られず、他の任意の偏向ミラーを用いることができる。また、走査部として、例えば、ＭＥＭＳミラーなどの１枚で二次元方向に測定光を走査することができる偏向ミラーを用いてもよい。 The galvanomirror 12 is used for scanning the measurement light on the fundus of the eye to be inspected, and the scanning range of the measurement light by the galvanomirror 12 can define an imaging range of the fundus by the OCT imaging. The image processing device 20 controls the driving range and the speed of the galvanomirror 12 so that the imaging range in the planar direction and the number of scanning lines (scanning speed in the planar direction) on the fundus can be defined. In FIG. 1, the galvanometer mirror 12 is shown as one unit for simplicity of description, but the galvanometer mirror 12 is actually composed of two mirrors for X scan and Y scan, and The desired range above can be scanned with the measuring light. The configuration of the scanning unit for scanning the measurement light is not limited to the galvanomirror, and any other deflecting mirror can be used. Further, as the scanning unit, for example, a deflecting mirror that can scan the measurement light in two-dimensional directions with one sheet such as a MEMS mirror may be used.
　フォーカスレンズステージ１３には不図示のフォーカスレンズが設けられている。フォーカスレンズステージ１３を移動させることで、フォーカスレンズを測定光の光軸に沿って移動させることができる。このため、フォーカスレンズによって、被検眼の前眼部を介し、眼底の網膜層に測定光をフォーカスすることができる。眼底を照射した測定光は各網膜層で反射・散乱して戻り光として、光路を戻る。 The focus lens stage 13 is provided with a focus lens (not shown). By moving the focus lens stage 13, the focus lens can be moved along the optical axis of the measurement light. Therefore, the measurement light can be focused on the retinal layer of the fundus through the anterior segment of the subject's eye by the focus lens. The measurement light illuminating the fundus is reflected and scattered by each retinal layer and returns to the optical path as return light.
　コヒーレンスゲートステージ１４は、被検眼の眼軸長の相違等に対応するため、参照光又は測定光の光路の長さを調整するために用いられる。本実施例では、コヒーレンスゲートステージ１４は、ミラーが設けられたステージによって構成され、参照光の光路において光軸方向に移動することで参照光の光路長を測定光の光路長に対応させることができる。ここで、コヒーレンスゲートは、ＯＣＴにおける測定光と参照光の光学距離が等しい位置を表す。コヒーレンスゲートステージ１４は、画像処理装置２０により制御されることができる。画像処理装置２０は、コヒーレンスゲートステージ１４によりコヒーレンスゲートの位置を制御することによって、被検眼の深さ方向の撮影範囲を制御することができ、網膜層側の撮影、又は網膜層より深部側の撮影等を制御することができる。 The coherence gate stage 14 is used to adjust the length of the optical path of the reference light or the measurement light in order to cope with a difference in the axial length of the eye to be examined. In the present embodiment, the coherence gate stage 14 is configured by a stage provided with a mirror, and can move the optical path length of the reference light to correspond to the optical path length of the measurement light by moving in the optical axis direction in the optical path of the reference light. it can. Here, the coherence gate represents a position where the optical distance between the measurement light and the reference light in OCT is equal. The coherence gate stage 14 can be controlled by the image processing device 20. The image processing device 20 can control the imaging range in the depth direction of the subject's eye by controlling the position of the coherence gate by the coherence gate stage 14, and can perform imaging on the retinal layer side or imaging on the deeper side than the retinal layer. Shooting and the like can be controlled.
　ディテクタ１５は、不図示の干渉部において生じた、被検眼からの測定光の戻り光と参照光との干渉光を検出し、干渉信号を生成する。画像処理装置２０は、ディテクタ１５からの干渉信号を取得し、干渉信号に対してフーリエ変換等を行うことで被検眼の断層画像を生成することができる。 The １５ detector 15 detects an interference light between the reference light and the return light of the measurement light from the subject's eye, which is generated in an interference unit (not shown), and generates an interference signal. The image processing device 20 can generate a tomographic image of the subject's eye by acquiring an interference signal from the detector 15 and performing a Fourier transform or the like on the interference signal.
　内部固視灯１６には、表示部１６１、及びレンズ１６２が設けられている。本実施例では、表示部１６１の一例として複数の発光ダイオード（ＬＤ）がマトリックス状に配置されたものを用いる。発光ダイオードの点灯位置は、画像処理装置２０の制御により撮影したい部位に応じて変更される。表示部１６１からの光は、レンズ１６２を介し、被検眼に導かれる。表示部１６１から出射される光は、例えば５２０ｎｍの波長を有し、画像処理装置２０による制御により所望のパターンで表示される。 表示 The internal fixation lamp 16 is provided with a display unit 161 and a lens 162. In this embodiment, an example in which a plurality of light emitting diodes (LDs) are arranged in a matrix is used as an example of the display unit 161. The lighting position of the light emitting diode is changed according to a part to be photographed under the control of the image processing device 20. Light from the display unit 161 is guided to the subject's eye via the lens 162. The light emitted from the display unit 161 has a wavelength of, for example, 520 nm, and is displayed in a desired pattern under the control of the image processing device 20.
　なお、ＯＣＴ装置１０には、画像処理装置２０による制御に基づいて、各構成要素の駆動を制御するＯＣＴ装置１０用の駆動制御部が設けられてもよい。 The OCT device 10 may be provided with a drive control unit for the OCT device 10 that controls the driving of each component based on the control of the image processing device 20.
　次に、図２Ａ乃至図２Ｃを参照して、画像処理システム１で取得する眼の構造と画像について説明する。図２Ａは眼球の模式図である。図２Ａには、角膜Ｃ、水晶体ＣＬ、硝子体Ｖ、黄斑部Ｍ（黄斑の中心部は中心窩を表す）、及び視神経乳頭部Ｄが表されている。本実施例では、主に、硝子体Ｖ、黄斑部Ｍ、視神経乳頭部Ｄを含む網膜の後極部を撮影する場合について説明を行う。なお、以下では説明をしないが、ＯＣＴ装置１０は、角膜や水晶体等の前眼部を撮影することも可能である。 Next, an eye structure and an image acquired by the image processing system 1 will be described with reference to FIGS. 2A to 2C. FIG. 2A is a schematic diagram of an eyeball. FIG. 2A shows the cornea C, the lens CL, the vitreous V, the macula M (the central portion of the macula represents the fovea), and the optic disc D. In the present embodiment, a case will be mainly described in which the rear pole of the retina including the vitreous body V, the macula M, and the optic disc D is imaged. Although not described below, the OCT apparatus 10 can also photograph the anterior eye such as the cornea and the crystalline lens.
　図２Ｂは、ＯＣＴ装置１０を用いて網膜を撮影することで取得した断層画像の一例を示す。図２Ｂにおいて、ＡＳは一回のＡスキャンにより取得される画像単位を示す。ここで、Ａスキャンとは、ＯＣＴ装置１０の上記一連の動作により、被検眼の一点における深さ方向の断層情報を取得することをいう。また、Ａスキャンを任意の横断方向（主走査方向）において複数回行うことで被検眼の当該横断方向と深さ方向の二次元の断層情報を取得することをＢスキャンという。Ａスキャンによって取得されたＡスキャン画像を複数集めることで、１つのＢスキャン画像を構成することができる。以下、このＢスキャン画像のことを、断層画像と呼ぶ。 FIG. 2B shows an example of a tomographic image obtained by imaging the retina using the OCT apparatus 10. In FIG. 2B, AS indicates an image unit obtained by one A-scan. Here, the A-scan refers to acquiring the tomographic information in the depth direction at one point of the subject's eye by the above-described series of operations of the OCT apparatus 10. In addition, acquiring two-dimensional tomographic information in the transverse direction and the depth direction of the subject's eye by performing the A scan a plurality of times in an arbitrary transverse direction (main scanning direction) is referred to as a B scan. By collecting a plurality of A-scan images acquired by the A-scan, one B-scan image can be formed. Hereinafter, this B-scan image is referred to as a tomographic image.
　図２Ｂには、血管Ｖｅ、硝子体Ｖ、黄斑部Ｍ、及び視神経乳頭部Ｄが表されている。また、境界線Ｌ１は内境界膜（ＩＬＭ）と神経線維層（ＮＦＬ）との境界、境界線Ｌ２は神経線維層と神経節細胞層（ＧＣＬ）との境界、境界線Ｌ３は視細胞内節外節接合部（ＩＳＯＳ）を表す。さらに、境界線Ｌ４は網膜色素上皮層（ＲＰＥ）、境界線Ｌ５はブルッフ膜（ＢＭ）、境界線Ｌ６は脈絡膜を表す。断層画像において、横軸（ＯＣＴの主走査方向）をｘ軸とし、縦軸（深さ方向）をｚ軸とする。 FIG. 2B shows blood vessel Ve, vitreous body V, macula M, and optic disc D. The boundary line L1 is a boundary between the inner limiting membrane (ILM) and the nerve fiber layer (NFL), the boundary line L2 is a boundary between the nerve fiber layer and the ganglion cell layer (GCL), and the boundary line L3 is a photoreceptor inner segment. Represents the outer joint (ISOS). Further, the boundary line L4 represents the retinal pigment epithelium layer (RPE), the boundary line L5 represents the Bruch's membrane (BM), and the boundary line L6 represents the choroid. In the tomographic image, the horizontal axis (OCT main scanning direction) is the x-axis, and the vertical axis (depth direction) is the z-axis.
　図２Ｃは、眼底画像撮影装置３０を用いて被検眼の眼底を撮影することで取得した眼底画像の一例を示す。図２Ｃには、黄斑部Ｍ、及び視神経乳頭部Ｄが表されており、網膜の血管が太い曲線で表されている。眼底画像において、横軸（ＯＣＴの主走査方向）をｘ軸とし、縦軸（ＯＣＴの副走査方向）をｙ軸とする。 FIG. 2C shows an example of a fundus image acquired by photographing the fundus of the eye to be examined using the fundus image photographing apparatus 30. FIG. 2C shows the macula M and the optic papilla D, and the blood vessels of the retina are represented by thick curves. In the fundus image, the horizontal axis (OCT main scanning direction) is the x-axis, and the vertical axis (OCT sub-scanning direction) is the y-axis.
　次に、画像処理装置２０について説明する。画像処理装置２０には、取得部２１、画像処理部２２、駆動制御部２３、記憶部２４、及び表示制御部２５が設けられている。 Next, the image processing device 20 will be described. The image processing device 20 includes an acquisition unit 21, an image processing unit 22, a drive control unit 23, a storage unit 24, and a display control unit 25.
　取得部２１は、ＯＣＴ装置１０から被検眼の干渉信号のデータを取得することができる。なお、取得部２１が取得する干渉信号のデータは、アナログ信号でもデジタル信号でもよい。取得部２１がアナログ信号を取得する場合には、画像処理装置２０でアナログ信号をデジタル信号に変換することができる。また、取得部２１は、画像処理部２２で生成された断層データや断層画像及びＥｎ－Ｆａｃｅ画像等の各種画像を取得することができる。ここで、断層データとは、被検体の断層に関する情報を含むデータであり、ＯＣＴによる干渉信号に基づくデータ、及びこれに高速フーリエ変換（ＦＦＴ：Ｆａｓｔ　Ｆｏｕｒｉｅｒ　Ｔｒａｎｓｆｏｒｍ）や任意の信号処理を行ったデータを含むものをいう。 The acquisition unit 21 can acquire the data of the interference signal of the subject's eye from the OCT apparatus 10. The data of the interference signal acquired by the acquisition unit 21 may be an analog signal or a digital signal. When the acquisition unit 21 acquires an analog signal, the image processing device 20 can convert the analog signal into a digital signal. Further, the acquisition unit 21 can acquire various images such as tomographic data, tomographic images, and En-Face images generated by the image processing unit 22. Here, the tomographic data is data including information on a tomographic image of a subject, and is data based on an interference signal by OCT and data obtained by performing fast Fourier transform (FFT: Fast @ Fourier @ Transform) or arbitrary signal processing on the data. Including
　さらに、取得部２１は、画像処理すべき断層画像の撮影条件群（例えば、撮影日時、撮影部位名、撮影領域、撮影画角、撮影方式、画像の解像度や階調、画像の画素サイズ、画像フィルタ、及び画像のデータ形式に関する情報など）を取得する。なお、撮影条件群については、例示したものに限られない。また、撮影条件群は、例示したもの全てを含む必要はなく、これらのうちの一部を含んでもよい。 Further, the acquiring unit 21 may obtain a group of imaging conditions of a tomographic image to be subjected to image processing (for example, imaging date and time, imaging part name, imaging area, imaging angle of view, imaging method, image resolution and gradation, image pixel size, image size, Filter, and information on the data format of the image). Note that the photographing condition group is not limited to the illustrated one. Further, the photographing condition group does not need to include all of the illustrated ones, and may include some of them.
　また、取得部２１は、眼底画像撮影装置３０で取得した眼底情報を含むデータ等を取得することができる。さらに、取得部２１は、被検者識別番号等の被検眼を同定するための情報を入力部６０等から取得することができる。取得部２１は、取得した各種データや画像を記憶部２４に記憶させることができる。 The acquisition unit 21 can also acquire data including fundus information acquired by the fundus image photographing device 30 and the like. Further, the acquiring unit 21 can acquire information for identifying the subject's eye such as the subject identification number from the input unit 60 or the like. The acquisition unit 21 can cause the storage unit 24 to store the acquired various data and images.
　画像処理部２２は、取得部２１で取得されたデータや記憶部２４に記憶されたデータから、断層画像やＥｎ－Ｆａｃｅ画像等を生成し、生成又は取得した画像に画像処理を施すことができる。このため、画像処理部２２は、Ｅｎ－Ｆａｃｅ画像や後述するモーションコントラスト正面画像を生成する生成部の一例として機能することができる。画像処理部２２には、断層画像生成部２２１及び処理部２２２（第一の処理部）が設けられている。 The image processing unit 22 generates a tomographic image, an En-Face image, and the like from the data acquired by the acquiring unit 21 and the data stored in the storage unit 24, and can perform image processing on the generated or acquired image. . Therefore, the image processing unit 22 can function as an example of a generation unit that generates an En-Face image or a motion contrast front image described later. The image processing unit 22 includes a tomographic image generation unit 221 and a processing unit 222 (first processing unit).
　断層画像生成部２２１は、取得部２１で取得された干渉信号に対してフーリエ変換等の処理を施して断層データを生成し、断層データに基づいて断層画像を生成することができる。なお、断層画像の生成方法としては既知の任意の方法を採用してよく、詳細な説明は省略する。 The tomographic image generating unit 221 can generate tomographic data by performing processing such as Fourier transform on the interference signal acquired by the acquiring unit 21 and generate a tomographic image based on the tomographic data. Note that any known method may be used as a method for generating a tomographic image, and a detailed description thereof will be omitted.
　処理部２２２は、ディープラーニング等の機械学習アルゴリズムによる機械学習モデルに関する学習済モデルを含むことができる。具体的な機械学習モデルに関しては後述する。処理部２２２は、学習済モデルを用いて、断層画像において被検眼の網膜層を検出するための検出処理を実行し、各網膜層を検出する。 The processing unit 222 can include a learned model related to a machine learning model based on a machine learning algorithm such as deep learning. A specific machine learning model will be described later. The processing unit 222 executes a detection process for detecting the retinal layer of the eye to be inspected in the tomographic image using the learned model, and detects each retinal layer.
　駆動制御部２３は、画像処理装置２０に接続されている、ＯＣＴ装置１０や眼底画像撮影装置３０の各構成要素の駆動を制御することができる。記憶部２４は、取得部２１で取得された断層データ、及び画像処理部２２で生成・処理された断層画像等の各種画像やデータ等を記憶することができる。また、記憶部２４は、プロセッサーによって実行されることで画像処理装置２０の各構成要素の機能を果たすためのプログラム等を記憶することもできる。 The drive control unit 23 can control the driving of each component of the OCT device 10 and the fundus image capturing device 30 connected to the image processing device 20. The storage unit 24 can store the tomographic data acquired by the acquiring unit 21 and various images and data such as tomographic images generated and processed by the image processing unit 22. Further, the storage unit 24 can also store a program or the like for performing the function of each component of the image processing device 20 by being executed by the processor.
　表示制御部２５は、取得部２１で取得された各種情報や画像処理部２２で生成・処理された断層画像、及び操作者によって入力された情報等の表示部５０における表示を制御することができる。 The display control unit 25 can control display on the display unit 50 of various information acquired by the acquisition unit 21, tomographic images generated and processed by the image processing unit 22, and information input by an operator. .
　画像処理装置２０の記憶部２４以外の各構成要素は、ＣＰＵ（Ｃｅｎｔｒａｌ　Ｐｒｏｃｅｓｓｉｎｇ　Ｕｎｉｔ）やＭＰＵ（Ｍｉｃｒｏ　Ｐｒｏｃｅｓｓｉｎｇ　Ｕｎｉｔ）等のプロセッサーによって実行されるソフトウェアモジュールにより構成されてよい。なお、プロセッサーは、例えば、ＧＰＵ（Ｇｒａｐｈｉｃａｌ　Ｐｒｏｃｅｓｓｉｎｇ　Ｕｎｉｔ）やＦＰＧＡ（Ｆｉｅｌｄ－Ｐｒｏｇｒａｍｍａｂｌｅ　Ｇａｔｅ　Ａｒｒａｙ）等であってもよい。また、当該各構成要素は、ＡＳＩＣ等の特定の機能を果たす回路等によって構成されてもよい。記憶部２４は、例えば、光学ディスクやメモリ等の任意の記憶媒体によって構成されてよい。 Each component other than the storage unit 24 of the image processing apparatus 20 may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The processor may be, for example, a GPU (Graphical Processing Unit) or an FPGA (Field-Programmable Gate Array). In addition, each of the components may be configured by a circuit or the like that performs a specific function such as an ASIC. The storage unit 24 may be configured by an arbitrary storage medium such as an optical disk and a memory.
　次に、図３を参照して、本実施例に係る一連の処理について説明する。図３は、本実施例に係る一連の処理のフローチャートである。本実施例に係る一連の処理が開始されると、処理はステップＳ３０１に移行する。 Next, a series of processes according to the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart of a series of processes according to the present embodiment. When a series of processes according to the present embodiment is started, the process proceeds to step S301.
　ステップＳ３０１では、取得部２１が、被検眼を同定する情報の一例である被検者識別番号を入力部６０等の画像処理装置２０の外部から取得する。取得部２１は、被検者識別番号に基づいて、外部記憶装置４０が保持している当該被検眼に関する情報を取得して記憶部２４に記憶する。 In step S301, the acquiring unit 21 acquires the subject identification number, which is an example of information for identifying the subject's eye, from outside the image processing device 20 such as the input unit 60. The acquiring unit 21 acquires information on the subject's eye held by the external storage device 40 based on the subject identification number and stores the information in the storage unit 24.
　ステップＳ３０２では、駆動制御部２３がＯＣＴ装置１０を制御して被検眼をスキャンすることで撮影を行い、取得部２１がＯＣＴ装置１０から被検眼の断層情報を含む干渉信号を取得する。被検眼のスキャンは、操作者によるスキャン開始の指示に応じて、駆動制御部２３がＯＣＴ装置１０を制御し、光源１１やガルバノミラー１２等を動作させることで行われる。 In step S302, the drive control unit 23 controls the OCT apparatus 10 to scan the subject's eye to perform imaging, and the acquisition unit 21 acquires from the OCT apparatus 10 an interference signal including tomographic information of the subject's eye. The scanning of the subject's eye is performed by the drive control unit 23 controlling the OCT device 10 and operating the light source 11, the galvanomirror 12, and the like in response to an instruction to start scanning by the operator.
　ガルバノミラー１２は、水平方向用のＸスキャナと垂直方向用のＹスキャナを含む。そのため、駆動制御部２３は、これらのスキャナの向きをそれぞれ変更することで、装置座標系における水平方向（Ｘ）及び垂直方向（Ｙ）のそれぞれの方向に測定光を走査することができる。なお、駆動制御部２３は、これらのスキャナの向きを同時に変更させることで、水平方向と垂直方向とを合成した方向にも測定光を走査することができる。そのため、駆動制御部２３は、眼底平面上の任意の方向に測定光を走査することができる。 The galvanometer mirror 12 includes a horizontal X scanner and a vertical Y scanner. Therefore, the drive control unit 23 can scan the measuring light in each of the horizontal direction (X) and the vertical direction (Y) in the apparatus coordinate system by changing the directions of these scanners. The drive control unit 23 can scan the measurement light in a direction obtained by combining the horizontal direction and the vertical direction by simultaneously changing the directions of the scanners. Therefore, the drive control unit 23 can scan the measurement light in any direction on the fundus plane.
　駆動制御部２３は、撮影を行うにあたり各種撮影パラメータの調整を行う。具体的には、駆動制御部２３は、内部固視灯１６で表示するパターンの位置、ガルバノミラー１２によるスキャン範囲やスキャンパターン、コヒーレンスゲート位置、及びフォーカスを少なくとも設定する。 The drive control unit 23 adjusts various shooting parameters when shooting. Specifically, the drive control unit 23 sets at least the position of the pattern displayed by the internal fixation lamp 16, the scan range and scan pattern by the galvanomirror 12, the coherence gate position, and the focus.
　駆動制御部２３は、表示部１６１の発光ダイオードを制御して、被検眼の黄斑部中心や視神経乳頭の撮影を行うように内部固視灯１６で表示するパターンの位置を制御する。また、駆動制御部２３は、ガルバノミラー１２のスキャンパターンとして、三次元ボリュームを撮影するラスタスキャンや放射状スキャン、クロススキャンなどのスキャンパターンを設定する。なお、どのスキャンパターンを選択したとしても、一つのライン上を繰り返し複数枚（繰り返し回数は２枚以上）撮影する。本実施例においては、スキャンパターンはクロススキャン、同一箇所を１５０枚繰り返し撮影する場合について説明する。これら撮影パラメータの調整終了後、操作者による撮影開始の指示に応じて、駆動制御部２３がＯＣＴ装置１０を制御して被検眼の撮影を行う。なお、本実施例に係る繰り返し回数は一例であり、所望の構成に応じて任意の回数に設定されてよい。 The drive control unit 23 controls the light emitting diode of the display unit 161 to control the position of the pattern displayed by the internal fixation lamp 16 so as to image the center of the macula of the subject's eye and the optic disc. In addition, the drive control unit 23 sets a scan pattern such as a raster scan, a radial scan, or a cross scan for capturing a three-dimensional volume as a scan pattern of the galvanometer mirror 12. Regardless of which scan pattern is selected, one line is repeatedly photographed (the number of repetitions is two or more). In the present embodiment, a case will be described where the scan pattern is a cross scan and the same portion is repeatedly photographed 150 times. After the adjustment of these imaging parameters is completed, the drive control unit 23 controls the OCT apparatus 10 to perform imaging of the subject's eye in response to an instruction to start imaging by the operator. The number of repetitions according to the present embodiment is an example, and may be set to an arbitrary number according to a desired configuration.
　本開示においては詳細な説明を省略するが、ＯＣＴ装置１０は、加算平均用に同じ箇所を撮影するために、被検眼のトラッキングを行うことができる。これにより、ＯＣＴ装置１０は、固視微動の影響を少なくして被検眼のスキャンを行うことができる。 を Although detailed description is omitted in the present disclosure, the OCT apparatus 10 can perform tracking of the subject's eye in order to photograph the same portion for averaging. Thereby, the OCT apparatus 10 can scan the subject's eye while reducing the influence of the fixation tremor.
　ステップＳ３０３では、断層画像生成部２２１が、取得部２１によって取得された干渉信号に基づいて断層画像の生成を行う。断層画像生成部２２１は、それぞれの干渉信号に対して、一般的な再構成処理を行うことで、断層画像を生成することができる。 In step S303, the tomographic image generation unit 221 generates a tomographic image based on the interference signal acquired by the acquisition unit 21. The tomographic image generation unit 221 can generate a tomographic image by performing general reconstruction processing on each interference signal.
　まず、断層画像生成部２２１は、干渉信号から固定パターンノイズ除去を行う。固定パターンノイズ除去は、取得した複数のＡスキャンの信号を平均することで固定パターンノイズを抽出し、これを入力した干渉信号から減算することで行われる。その後、断層画像生成部２２１は、有限区間で干渉信号をフーリエ変換した場合にトレードオフの関係となる深さ分解能とダイナミックレンジを最適化するために、所望の窓関数処理を行う。断層画像生成部２２１は、窓関数処理を行った干渉信号に対して高速フーリエ変換（ＦＦＴ）処理を行うことによって断層データを生成する。 First, the tomographic image generation unit 221 removes fixed pattern noise from the interference signal. The fixed pattern noise removal is performed by averaging the plurality of acquired A-scan signals to extract fixed pattern noise, and subtracting this from the input interference signal. Thereafter, the tomographic image generation unit 221 performs a desired window function process in order to optimize the depth resolution and the dynamic range that are in a trade-off relationship when the interference signal is Fourier-transformed in a finite section. The tomographic image generation unit 221 generates tomographic data by performing fast Fourier transform (FFT) processing on the interference signal that has been subjected to the window function processing.
　断層画像生成部２２１は、生成した断層データに基づいて断層画像の各画素値を求め、断層画像を生成する。なお、断層画像の生成方法はこれに限られず、既知の任意の方法で行われてよい。 The tomographic image generation unit 221 obtains each pixel value of the tomographic image based on the generated tomographic data, and generates a tomographic image. The method of generating a tomographic image is not limited to this, and may be performed by any known method.
　ステップＳ３０４では、画像処理部２２の処理部２２２が網膜層の検出処理を行う。図４Ａ及び図４Ｂを参照して、処理部２２２の処理について説明する。 In step S304, the processing unit 222 of the image processing unit 22 performs a retinal layer detection process. The processing of the processing unit 222 will be described with reference to FIGS. 4A and 4B.
　処理部２２２は、ＯＣＴ装置１０を用いて取得した複数の断層画像において網膜層の境界を検出する。処理部２２２は、予め機械学習が行われた機械学習モデルに関する学習済モデルを用いて各網膜層を検出する。 The processing unit 222 detects a retinal layer boundary in a plurality of tomographic images acquired using the OCT apparatus 10. The processing unit 222 detects each retinal layer using a learned model related to a machine learning model on which machine learning has been performed in advance.
　ここで、図４Ａ乃至図６を参照して、本実施例に係る機械学習アルゴリズムについて説明する。本実施例に係る機械学習モデルの学習データ（教師データ）は、１つ以上の入力データと出力データとのペア群で構成される。具体的には、入力データとして、ＯＣＴにより取得された断層画像４０１が挙げられ、出力データとして、当該断層画像について網膜層の境界が特定された境界画像４０２が挙げられる。本実施例では、境界画像４０２として、ＩＬＭとＮＦＬとの境界４０３、ＮＦＬとＧＣＬとの境界４０４、ＩＳＯＳ４０５、ＲＰＥ４０６、及びＢＭ４０７が示された画像を用いる。なお、図示はしないが、その他の境界として、外網状層（ＯＰＬ）と外顆粒層（ＯＮＬ）との境界、内網状層（ＩＰＬ）と内顆粒層（ＩＮＬ）との境界、ＩＮＬとＯＰＬとの境界、ＧＣＬとＩＰＬとの境界等が示された画像を用いてもよい。 Here, the machine learning algorithm according to the present embodiment will be described with reference to FIGS. 4A to 6. The learning data (teacher data) of the machine learning model according to the present embodiment includes one or more pairs of input data and output data. Specifically, the input data includes a tomographic image 401 obtained by OCT, and the output data includes a boundary image 402 in which a boundary of a retinal layer is specified for the tomographic image. In this embodiment, an image showing a boundary 403 between the ILM and the NFL, a boundary 404 between the NFL and the GCL, the ISSO 405, the RPE 406, and the BM 407 is used as the boundary image 402. Although not shown, other boundaries include a boundary between the outer plexiform layer (OPL) and the outer granular layer (ONL), a boundary between the inner plexiform layer (IPL) and the inner granular layer (INL), and a boundary between INL and OPL. , The boundary between the GCL and the IPL, or the like may be used.
　なお、出力データとして用いられる境界画像４０２は、医師等により断層画像において境界が示された画像であってもよいし、ルールベースの境界検出処理により境界が検出された画像であってもよい。ただし、適切に境界検出が行われていない境界画像を教師データの出力データとして用いて機械学習を行うと、当該教師データを用いて学習した学習済モデルを用いて得た画像も適切に境界検出が行われていない境界画像となってしまう可能性がある。そのため、そのような境界画像を含むペアを教師データから取り除くことで、学習済モデルを用いて適切でない境界画像が生成される可能性を低減させることができる。ここで、ルールベースの処理とは既知の規則性を利用した処理をいい、ルールベースの境界検出とは、例えば網膜の形状の規則性等の既知の規則性を利用した境界検出処理をいう。 The boundary image 402 used as output data may be an image in which a boundary is indicated in a tomographic image by a doctor or the like, or may be an image in which a boundary has been detected by a rule-based boundary detection process. However, if machine learning is performed using boundary images for which boundary detection has not been performed properly as output data of teacher data, images obtained using a trained model trained using the teacher data will also be properly detected. There is a possibility that the boundary image will not be obtained. Therefore, by removing the pair including such a boundary image from the teacher data, it is possible to reduce the possibility that an inappropriate boundary image is generated using the learned model. Here, the rule-based processing refers to processing using known regularity, and the rule-based boundary detection refers to boundary detection processing using known regularity such as, for example, the retina shape regularity.
　また、図４Ａ及び図４Ｂにおいては、網膜のＸＹ面内におけるある一つのＸＺ断面の例を示しているが、断面はこれに限らない。図示しないが、ＸＹ面内における任意の複数のＸＺ断面についての断層画像及び境界画像を事前に学習しておき、ラスタスキャンやラジアルスキャン等、異なる様々なスキャンパターンで撮影された断面に対して対応できるようにしておくことができる。例えば、ラスタスキャンで三次元的に網膜を撮影した断層画像等のデータを用いる場合には、隣接する複数の断層画像間の位置合わせをしたボリュームデータを教師データに用いることができる。この場合には、１つのボリュームデータ（三次元の断層画像）とこれに対応する１つの三次元の境界データ（三次元の境界画像）とから、任意の角度のペア画像群を生成することが可能である。また、機械学習モデルは、実際に様々なスキャンパターンで撮影した画像を教師データとして用いて学習してもよい。 4A and 4B show an example of one XZ cross section in the XY plane of the retina, but the cross section is not limited to this. Although not shown, a tomographic image and a boundary image of an arbitrary plurality of XZ sections in the XY plane are learned in advance, and corresponding to sections taken in various different scan patterns such as a raster scan and a radial scan. Can be made available. For example, when using data such as a tomographic image obtained by three-dimensionally photographing the retina by raster scanning, volume data obtained by aligning a plurality of adjacent tomographic images can be used as teacher data. In this case, a pair image group of an arbitrary angle can be generated from one volume data (three-dimensional tomographic image) and one corresponding three-dimensional boundary data (three-dimensional boundary image). It is possible. Further, the machine learning model may learn using images actually captured with various scan patterns as teacher data.
　次に、学習時の画像について説明する。機械学習モデルの教師データを構成する、断層画像４０１と境界画像４０２とのペア群を構成する画像群を、位置関係が対応する一定の画像サイズの矩形領域画像によって作成する。当該画像の作成について、図５Ａ乃至図５Ｃを参照して説明する。 Next, the image at the time of learning will be described. An image group forming a pair group of a tomographic image 401 and a boundary image 402, which forms teacher data of a machine learning model, is created by a rectangular area image having a fixed image size corresponding to a positional relationship. The creation of the image will be described with reference to FIGS. 5A to 5C.
　まず、教師データを構成するペア群の１つを、断層画像４０１と境界画像４０２とした場合について説明する。この場合には、図５Ａに示すように、断層画像４０１の全体である矩形領域画像５０１を入力データ、境界画像４０２の全体である矩形領域画像５０２を出力データとして、ペアを構成する。なお、図５Ａに示す例では各画像の全体により入力データと出力データのペアを構成しているが、ペアはこれに限らない。 First, a case will be described in which one of the group of pairs forming the teacher data is a tomographic image 401 and a boundary image 402. In this case, as shown in FIG. 5A, a pair is formed by using a rectangular area image 501 that is the entire tomographic image 401 as input data and a rectangular area image 502 that is the entire boundary image 402 as output data. In the example shown in FIG. 5A, a pair of input data and output data is formed by the entirety of each image, but the pair is not limited to this.
　例えば、図５Ｂに示すように、断層画像４０１のうちの矩形領域画像５１１を入力データ、境界画像４０２における対応する撮影領域である矩形領域画像５１３を出力データとして、ペアを構成してもよい。矩形領域画像５１１，５１３の矩形領域は、Ａスキャン単位を基本としている。Ａスキャン単位とは、１本のＡスキャン単位でもよいし、数本のＡスキャン単位でもよい。 For example, as shown in FIG. 5B, a pair may be formed by using a rectangular area image 511 of the tomographic image 401 as input data and a rectangular area image 513 as a corresponding imaging area in the boundary image 402 as output data. The rectangular areas of the rectangular area images 511 and 513 are based on A-scan units. The A scan unit may be one A scan unit or several A scan units.
　なお、図５ＢではＡスキャン単位を基本としているが、画像に対して深さ方向の全てを領域とするのではなく、上下に矩形領域外の部分を設けてもよい。すなわち、矩形領域の横方向のサイズはＡスキャン数本分、矩形領域の深さ方向のサイズは、画像の深さ方向のサイズよりも小さく設定してもよい。 In addition, although FIG. 5B is based on the unit of A-scan, the whole area in the depth direction may not be the area of the image, and a part outside the rectangular area may be provided vertically. That is, the size of the rectangular area in the horizontal direction may be set to several A scans, and the size of the rectangular area in the depth direction may be set smaller than the size of the image in the depth direction.
　また、図５Ｃに示すように、断層画像４０１のうちの矩形領域画像５２１を入力データ、境界画像４０２における対応する撮影領域である矩形領域画像５２３を出力データとして、ペアを構成してもよい。 5C, as shown in FIG. 5C, a pair may be formed by using the rectangular area image 521 of the tomographic image 401 as input data and the rectangular area image 523 which is a corresponding imaging area in the boundary image 402 as output data.
　なお、学習時には、スキャン範囲（撮影画角）、スキャン密度（Ａスキャン数）を正規化して画像サイズを揃えて、学習時の矩形領域サイズを一定に揃えることができる。また、図５Ａ乃至図５Ｃに示した矩形領域画像は、それぞれ別々に学習する際の矩形領域サイズの一例である。 At the time of learning, the scan range (angle of view) and the scan density (the number of A-scans) are normalized to make the image size uniform, so that the rectangular area size at the time of learning can be made uniform. The rectangular area images shown in FIGS. 5A to 5C are examples of the rectangular area size when learning is performed separately.
　矩形領域の数は、図５Ａに示す例では１つ、図５Ｂ及び図５Ｃに示す例では複数設定可能である。例えば、図５Ｂに示す例において、断層画像４０１のうちの矩形領域画像５１２を入力データ、境界画像４０２における対応する撮影領域である矩形領域画像５１４を出力データとしてペアを構成することもできる。また、例えば、図５Ｃに示す例において、断層画像４０１のうちの矩形領域画像５２２を入力データ、境界画像４０２における対応する撮影領域である矩形領域画像５２４を出力データとしてペアを構成することもできる。このように、１枚ずつの断層画像及び境界画像のペアから、互いに異なる矩形領域画像のペアを作成できる。なお、元となる断層画像及び境界画像において、領域の位置を異なる座標に変えながら多数の矩形領域画像のペアを作成することで、教師データを構成するペア群を充実させることができる。 5) The number of rectangular areas can be set to one in the example shown in FIG. 5A, and a plurality can be set in the examples shown in FIGS. 5B and 5C. For example, in the example illustrated in FIG. 5B, a pair may be configured using the rectangular area image 512 of the tomographic image 401 as input data and the rectangular area image 514 that is a corresponding imaging area in the boundary image 402 as output data. Further, for example, in the example shown in FIG. 5C, a pair can be formed by using the rectangular area image 522 of the tomographic image 401 as input data and the rectangular area image 524 as the corresponding imaging area in the boundary image 402 as output data. . Thus, a pair of mutually different rectangular area images can be created from a pair of a tomographic image and a boundary image one by one. In the original tomographic image and the boundary image, by creating a large number of pairs of rectangular area images while changing the position of the area to different coordinates, it is possible to enrich the group of pairs forming the teacher data.
　図５Ｂ及び図５Ｃに示す例では、離散的に矩形領域を示しているが、実際には、元となる断層画像及び境界画像を、隙間なく連続する一定の画像サイズの矩形領域画像群に分割することができる。また、元となる断層画像及び境界画像について、互いに対応する、ランダムな位置の矩形領域画像群に分割してもよい。このように、矩形領域（又は、短冊領域）として、より小さな領域の画像を入力データ及び出力データのペアとして選択することで、もともとのペアを構成する断層画像４０１及び境界画像４０２から多くのペアデータを生成できる。そのため、機械学習モデルのトレーニングにかかる時間を短縮することができる。一方で、完成した機械学習モデルの学習済モデルでは、実行する画像セグメンテーション処理の時間が長くなる傾向にある。ここで、画像セグメンテーション処理とは、画像内の領域や境界を識別したり、区別したりする処理をいう。 In the examples shown in FIGS. 5B and 5C, the rectangular areas are discretely shown. However, in practice, the original tomographic image and the boundary image are divided into a group of rectangular area images having a constant image size and without gaps. can do. Alternatively, the original tomographic image and the boundary image may be divided into a group of rectangular area images at random positions corresponding to each other. As described above, by selecting an image of a smaller area as a rectangular area (or a strip area) as a pair of input data and output data, many pairs are obtained from the tomographic image 401 and the boundary image 402 constituting the original pair. Can generate data. Therefore, the time required for training the machine learning model can be reduced. On the other hand, in the trained model of the completed machine learning model, the time of the executed image segmentation process tends to be long. Here, the image segmentation processing refers to processing for identifying or distinguishing an area or a boundary in an image.
　次に、本実施例に係る機械学習モデルの一例として、入力された断層画像に対して、画像セグメンテーション処理を行う畳み込みニューラルネットワーク（ＣＮＮ）に関して、図６を参照して説明する。図６は、処理部２２２における機械学習モデルの構成６０１の一例を示している。なお、本実施例に係る機械学習モデルとしては、例えば、ＦＣＮ（Ｆｕｌｌｙ　Ｃｏｎｖｏｌｕｔｉｏｎａｌ　Ｎｅｔｗｏｒｋ）、又はＳｅｇＮｅｔ等を用いることもできる。また、所望の構成に応じて領域単位で物体認識を行う機械学習モデルを用いてもよい。物体認識を行う機械学習モデルとしては、例えば、ＲＣＮＮ（Ｒｅｇｉｏｎ　ＣＮＮ）、ｆａｓｔＲＣＮＮ、又はｆａｓｔｅｒＲＣＮＮを用いることができる。さらに、領域単位で物体認識を行う機械学習モデルとして、ＹＯＬＯ（Ｙｏｕ　Ｌｏｏｋ　Ｏｎｌｙ　Ｏｎｃｅ）、又はＳＳＤ（Ｓｉｎｇｌｅ　Ｓｈｏｔ　ＭｕｌｔｉＢｏｘ　Ｄｅｔｅｃｔｏｒ）を用いることもできる。 Next, as an example of the machine learning model according to the present embodiment, a convolutional neural network (CNN) that performs image segmentation processing on an input tomographic image will be described with reference to FIG. FIG. 6 illustrates an example of a configuration 601 of a machine learning model in the processing unit 222. In addition, as the machine learning model according to the present embodiment, for example, FCN (Fully Convolutional Network), SegNet, or the like can be used. Further, a machine learning model that performs object recognition on a region basis according to a desired configuration may be used. As a machine learning model for performing object recognition, for example, RCNN (Region @ CNN), fastRCNN, or fastRCNN can be used. Furthermore, YOLO (You Look Only Once) or SSD (Single Shot MultiBox Detector) can be used as a machine learning model for performing object recognition in units of regions.
　図６に示す機械学習モデルは、入力値群を加工して出力する処理を担う複数の層群によって構成される。なお、当該機械学習モデルの構成６０１に含まれる層の種類としては、畳み込み（Ｃｏｎｖｏｌｕｔｉｏｎ）層、ダウンサンプリング（Ｄｏｗｎｓａｍｐｌｉｎｇ）層、アップサンプリング（Ｕｐｓａｍｐｌｉｎｇ）層、及び合成（Ｍｅｒｇｅｒ）層がある。 機械 The machine learning model shown in FIG. 6 is composed of a plurality of layer groups responsible for processing the input value group and outputting it. The types of layers included in the configuration 601 of the machine learning model include a convolution layer, a downsampling (Downsampling) layer, an upsampling (Upsampling) layer, and a synthesis (Merger) layer.
　畳み込み層は、設定されたフィルタのカーネルサイズ、フィルタの数、ストライドの値、ダイレーションの値等のパラメータに従い、入力値群に対して畳み込み処理を行う層である。なお、入力される画像の次元数に応じて、フィルタのカーネルサイズの次元数も変更してもよい。 The convolution layer is a layer that performs a convolution process on an input value group according to parameters such as a set filter kernel size, the number of filters, a stride value, and a dilation value. The number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image.
　ダウンサンプリング層は、入力値群を間引いたり、合成したりすることによって、出力値群の数を入力値群の数よりも少なくする処理を行う層である。具体的には、このような処理として、例えば、Ｍａｘ　Ｐｏｏｌｉｎｇ処理がある。 (4) The downsampling layer is a layer that performs processing to reduce the number of output value groups from the number of input value groups by thinning out or combining input value groups. Specifically, for example, there is a Max @ Pooling process.
　アップサンプリング層は、入力値群を複製したり、入力値群から補間した値を追加したりすることによって、出力値群の数を入力値群の数よりも多くする処理を行う層である。具体的には、このような処理として、例えば、線形補間処理がある。 (4) The upsampling layer is a layer that performs processing for increasing the number of output value groups beyond the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. Specifically, as such processing, for example, there is a linear interpolation processing.
　合成層は、ある層の出力値群や画像を構成する画素値群といった値群を、複数のソースから入力し、それらを連結したり、加算したりして合成する処理を行う層である。 The composition layer is a layer that performs a process of inputting a value group such as an output value group of a certain layer or a pixel value group constituting an image from a plurality of sources, concatenating them, and adding them to combine them.
　なお、図６に示す構成６０１に含まれる畳み込み層群に設定されるパラメータとして、例えば、フィルタのカーネルサイズを幅３画素、高さ３画素、フィルタの数を６４とすることで、一定の精度の画像セグメンテーション処理が可能である。ただし、ニューラルネットワークを構成する層群やノード群に対するパラメータの設定が異なると、教師データからトレーニングされた傾向を出力データに再現可能な程度が異なる場合があるので注意が必要である。つまり、多くの場合、実施する際の形態に応じて適切なパラメータは異なるので、必要に応じて好ましい値に変更することができる。 As parameters set in the convolutional layer group included in the configuration 601 shown in FIG. 6, for example, by setting the kernel size of the filter to 3 pixels in width, 3 pixels in height, and 64 to the number of filters, a certain accuracy can be obtained. Image segmentation processing is possible. However, it should be noted that if the parameter setting for the layer group or the node group forming the neural network is different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may be different. In other words, in many cases, appropriate parameters are different depending on the mode of implementation, and can be changed to preferable values as needed.
　また、上述したようなパラメータを変更するという方法だけでなく、ＣＮＮの構成を変更することによって、ＣＮＮがより良い特性を得られる場合がある。より良い特性とは、例えば、画像セグメンテーション処理の精度が高かったり、画像セグメンテーション処理の時間が短かったり、機械学習モデルのトレーニングにかかる時間が短かったりする等である。 In addition to the method of changing the parameters as described above, by changing the configuration of the CNN, there is a case where the CNN can obtain better characteristics. The better characteristics include, for example, higher accuracy of the image segmentation process, shorter time for the image segmentation process, shorter time for training of the machine learning model, and the like.
　なお、本実施例で用いるＣＮＮの構成６０１は、複数のダウンサンプリング層を含む複数の階層からなるエンコーダーの機能と、複数のアップサンプリング層を含む複数の階層からなるデコーダーの機能とを有するＵ－ｎｅｔ型の機械学習モデルである。Ｕ－ｎｅｔ型の機械学習モデルでは、エンコーダーとして構成される複数の階層において曖昧にされた位置情報（空間情報）を、デコーダーとして構成される複数の階層において、同次元の階層（互いに対応する階層）で用いることができるように（例えば、スキップコネクションを用いて）構成される。 The configuration 601 of the CNN used in this embodiment has a function of an encoder having a plurality of layers including a plurality of downsampling layers and a function of a decoder including a plurality of layers including a plurality of upsampling layers. This is a net-type machine learning model. In the U-net type machine learning model, position information (spatial information) which is ambiguous in a plurality of hierarchies configured as encoders is converted into a same-dimensional hierarchy (hierarchical layers corresponding to each other) in a plurality of hierarchies configured as decoders. ) (Eg, using a skip connection).
　図示しないが、ＣＮＮの構成の変更例として、例えば、畳み込み層の後にバッチ正規化（Ｂａｔｃｈ　Ｎｏｒｍａｌｉｚａｔｉｏｎ）層や、正規化線形関数（Ｒｅｃｔｉｆｉｅｒ　Ｌｉｎｅａｒ　Ｕｎｉｔ）を用いた活性化層を組み込む等をしてもよい。 Although not shown, as a modification of the configuration of the CNN, for example, a batch normalization (Batch @ Normalization) layer or an activation layer using a normalized linear function (Rectifier @ Linear @ Unit) may be incorporated after the convolutional layer. Good.
　このような機械学習モデルの学習済モデルにデータを入力すると、機械学習モデルの設計に従ったデータが出力される。例えば、教師データを用いてトレーニングされた傾向に従って入力データに対応する可能性の高い出力データが出力される。 (4) When data is input to a trained model of such a machine learning model, data according to the design of the machine learning model is output. For example, output data having a high possibility of corresponding to the input data is output according to the tendency trained using the teacher data.
　本実施例に係る処理部２２２の学習済モデルでは、断層画像４０１が入力されると、教師データを用いてトレーニングされた傾向に従って、境界画像４０２を出力する。処理部２２２は、境界画像４０２に基づいて断層画像４０１における網膜層及びその境界を検出することができる。 In the learned model of the processing unit 222 according to the present embodiment, when the tomographic image 401 is input, the boundary image 402 is output according to the tendency trained using the teacher data. The processing unit 222 can detect a retinal layer and its boundary in the tomographic image 401 based on the boundary image 402.
　なお、図５Ｂ及び図５Ｃに示すように、画像の領域を分割して学習している場合、処理部２２２は学習済モデルを用いて、それぞれの矩形領域に対応する境界画像である矩形領域画像を得る。そのため、処理部２２２は、各矩形領域において網膜層を検出することができる。この場合、処理部２２２は、学習済モデルを用いて得た境界画像である矩形領域画像群のそれぞれを、矩形領域画像群のぞれぞれと同様の位置関係に配置して結合することで、入力された断層画像４０１に対応する境界画像４０２を生成することができる。この場合にも、処理部２２２は、生成された境界画像４０２に基づいて断層画像４０１における網膜層及びその境界を検出することができる。 As shown in FIGS. 5B and 5C, when learning is performed by dividing the image region, the processing unit 222 uses the learned model to generate a rectangular region image that is a boundary image corresponding to each rectangular region. Get. Therefore, the processing unit 222 can detect a retinal layer in each rectangular area. In this case, the processing unit 222 arranges and connects each of the rectangular area image groups, which are the boundary images obtained using the learned model, in the same positional relationship as each of the rectangular area image groups. , A boundary image 402 corresponding to the input tomographic image 401 can be generated. Also in this case, the processing unit 222 can detect the retinal layer and its boundary in the tomographic image 401 based on the generated boundary image 402.
　ステップＳ３０４において、処理部２２２が網膜層の検出処理を行うと、処理はステップＳ３０５に移行する。ステップＳ３０５では、表示制御部２５が、処理部２２２によって検出した境界と断層画像とを表示部５０に表示する。ここで、図７に表示部５０に表示する画面の一例を示す。 (4) In step S304, when the processing unit 222 performs a retinal layer detection process, the process proceeds to step S305. In step S305, the display control unit 25 displays the boundary and the tomographic image detected by the processing unit 222 on the display unit 50. Here, an example of a screen displayed on the display unit 50 is shown in FIG.
　図７には表示画面７００が示されており、表示画面７００には、ＳＬＯ画像７０１、ＳＬＯ画像７０１に重畳表示される厚みマップ７０２、Ｅｎ－Ｆａｃｅ画像７０３、断層画像７１１、及び網膜の厚みグラフ７１２が示されている。断層画像７１１には、網膜の境界７１５，７１６が重畳表示されている。 FIG. 7 shows a display screen 700. The display screen 700 includes an SLO image 701, a thickness map 702 superimposed on the SLO image 701, an En-Face image 703, a tomographic image 711, and a retinal thickness graph. 712 is shown. Retinal boundaries 715 and 716 are superimposed on the tomographic image 711.
　なお、本実施例では網膜の範囲を、内境界膜と神経線維層との境界Ｌ１～網膜色素上皮層Ｌ４としており、境界７１５，７１６はそれぞれ境界Ｌ１及び網膜色素上皮層Ｌ４に対応する。網膜の範囲はこれに限られず、例えば、内境界膜と神経線維層との境界Ｌ１～脈絡膜Ｌ６の範囲としてもよく、この場合、境界７１５，７１６はそれぞれ境界Ｌ１及び脈絡膜Ｌ６に対応することができる。 In this embodiment, the range of the retina is defined as the boundary L1 between the inner limiting membrane and the nerve fiber layer to the retinal pigment epithelium layer L4, and the boundaries 715 and 716 correspond to the boundary L1 and the retinal pigment epithelium layer L4, respectively. The range of the retina is not limited to this, and may be, for example, a range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the choroid L6. In this case, the boundaries 715 and 716 may correspond to the boundary L1 and the choroid L6, respectively. it can.
　網膜の厚みグラフ７１２は、境界７１５，７１６から求められる網膜の厚みを示すグラフである。また、厚みマップ７０２は境界７１５，７１６から求められる網膜の厚みをカラーマップで表現したものである。なお、図７では、説明のため、厚みマップ７０２に対応する色情報は示されていないが、実際には、厚みマップ７０２は、ＳＬＯ画像７０１における各座標に対応する網膜の厚みを対応するカラーマップに従って表示することができる。 The retinal thickness graph 712 is a graph showing the retinal thickness obtained from the boundaries 715 and 716. The thickness map 702 expresses the thickness of the retina obtained from the boundaries 715 and 716 by a color map. Although color information corresponding to the thickness map 702 is not shown in FIG. 7 for the sake of explanation, the thickness map 702 is actually a color map corresponding to the thickness of the retina corresponding to each coordinate in the SLO image 701. It can be displayed according to the map.
　Ｅｎ－Ｆａｃｅ画像７０３は、境界７１５，７１６の間の範囲のデータをＸＹ方向に投影して生成した正面画像である。正面画像は、光干渉を用いて得たボリュームデータ（三次元の断層画像）の少なくとも一部の深度範囲であって、２つの基準面に基づいて定められた深度範囲に対応するデータを二次元平面に投影又は積算して生成される。本実施例に係るＥｎ－Ｆａｃｅ画像７０３は、ボリュームデータのうちの、検出された網膜層に基づいて決定された深度範囲（境界７１５，７１６の間の深度範囲）に対応するデータを二次元平面に投影して生成された正面画像である。なお、２つの基準面に基づいて定められた深度範囲に対応するデータを二次元平面に投影する手法としては、例えば、当該深度範囲内のデータの代表値を二次元平面上の画素値とする手法を用いることができる。ここで、代表値は、２つの基準面に囲まれた領域の深さ方向の範囲内における画素値の平均値、中央値又は最大値などの値を含むことができる。 The En-Face image 703 is a front image generated by projecting data in a range between boundaries 715 and 716 in the XY directions. The front image is at least a part of the depth range of volume data (three-dimensional tomographic image) obtained by using optical interference, and two-dimensionally represents data corresponding to a depth range determined based on two reference planes. It is generated by projecting or integrating on a plane. The En-Face image 703 according to the present embodiment is obtained by converting data corresponding to a depth range (depth range between boundaries 715 and 716) of the volume data determined based on the detected retinal layer into a two-dimensional plane. Is a front image generated by projecting the image on the front side. As a method of projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a representative value of data within the depth range is defined as a pixel value on the two-dimensional plane. Techniques can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in a depth direction of a region surrounded by two reference planes.
　また、表示画面７００に示されるＥｎ－Ｆａｃｅ画像７０３に係る深度範囲は、境界７１５，７１６の間の深度範囲に限られない。Ｅｎ－Ｆａｃｅ画像７０３に係る深度範囲は、例えば、検出された網膜層に関する２つの層境界７１５，７１６の一方を基準として、より深い方向又はより浅い方向に所定の画素数分だけ含んだ範囲であってもよい。また、Ｅｎ－Ｆａｃｅ画像７０３に係る深度範囲は、例えば、検出された網膜層に関する２つの層境界７１５，７１６の間の範囲から、操作者の指示に応じて変更された（オフセットされた）範囲であってもよい。 The depth range of the En-Face image 703 shown on the display screen 700 is not limited to the depth range between the boundaries 715 and 716. The depth range of the En-Face image 703 is, for example, a range including a predetermined number of pixels in a deeper or shallower direction with reference to one of the two layer boundaries 715 and 716 related to the detected retinal layer. There may be. In addition, the depth range of the En-Face image 703 is, for example, a range changed (offset) from the range between the two layer boundaries 715 and 716 related to the detected retinal layer in accordance with the instruction of the operator. It may be.
　なお、表示画面７００に示される正面画像は、輝度値に基づくＥｎ－Ｆａｃｅ画像（輝度のＥｎ－Ｆａｃｅ画像）に限られない。表示画面７００に示される正面画像は、例えば、複数のボリュームデータ間のモーションコントラストデータについて、上述の深度範囲に対応するデータを二次元平面に投影又は積算して生成したモーションコントラスト正面画像であってもよい。ここで、モーションコントラストデータとは、被検眼の同一領域（同一位置）において測定光が複数回走査されるように制御して得た複数のボリュームデータ間での変化を示すデータである。このとき、ボリュームデータは、異なる位置で得た複数の断層画像により構成される。そして、異なる位置それぞれにおいて、略同一位置で得た複数の断層画像の間での変化を示すデータを得ることで、モーションコントラストデータをボリュームデータとして得ることができる。なお、モーションコントラスト正面画像は、血流の動きを測定するＯＣＴアンギオグラフィ（ＯＣＴＡ）に関するＯＣＴＡ正面画像（ＯＣＴＡのＥｎ－Ｆａｃｅ画像）とも呼ばれ、モーションコントラストデータはＯＣＴＡデータとも呼ばれる。モーションコントラストデータは、例えば、２枚の断層画像又はこれに対応する干渉信号間の脱相関値、分散値、又は最大値を最小値で割った値（最大値／最小値）として求めることができ、公知の任意の方法により求められてよい。このとき、２枚の断層画像は、例えば、被検眼の同一領域（同一位置）において測定光が複数回走査されるように制御して得ることができる。 The front image shown on the display screen 700 is not limited to the En-Face image based on the luminance value (En-Face image of luminance). The front image shown on the display screen 700 is, for example, a motion contrast front image generated by projecting or integrating data corresponding to the above-described depth range onto a two-dimensional plane with respect to motion contrast data between a plurality of volume data. Is also good. Here, the motion contrast data is data indicating a change among a plurality of volume data obtained by controlling the measurement light to scan a plurality of times in the same region (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, at each different position, by obtaining data indicating a change between a plurality of tomographic images obtained at substantially the same position, motion contrast data can be obtained as volume data. The motion contrast front image is also called an OCTA front image (OCTA En-Face image) related to OCT angiography (OCTA) for measuring blood flow movement, and the motion contrast data is also called OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value between two tomographic images or interference signals corresponding thereto (maximum value / minimum value). May be determined by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to scan a plurality of times in the same region (same position) of the eye to be inspected.
　また、ＯＣＴＡ正面画像を生成する際に用いられる三次元のＯＣＴＡデータ（ＯＣＴボリュームデータ）は、網膜層を検出するための断層画像を含むボリュームデータと共通の干渉信号の少なくとも一部を用いて生成されてもよい。この場合には、ボリュームデータ（三次元の断層画像）と三次元のＯＣＴＡデータとが互いに対応することができる。そのため、ボリュームデータに対応する三次元のモーションコントラストデータを用いて、例えば、検出された網膜層に基づいて決定された深度範囲に対応するモーションコントラスト正面画像が生成されることができる。 Also, three-dimensional OCTA data (OCT volume data) used when generating an OCTA front image is generated using at least a part of a common interference signal with volume data including a tomographic image for detecting a retinal layer. May be done. In this case, the volume data (three-dimensional tomographic image) and the three-dimensional OCTA data can correspond to each other. Therefore, using the three-dimensional motion contrast data corresponding to the volume data, for example, a motion contrast front image corresponding to a depth range determined based on the detected retinal layer can be generated.
　ここで、厚みマップ７０２、Ｅｎ－Ｆａｃｅ画像７０３、厚みグラフ７１２、及び境界７１５，７１６の表示は、処理部２２２で検出した境界や網膜層に基づいて、画像処理装置２０によって生成されることができるものの例である。なお、これらを生成する生成方法は公知の任意の方法を採用してよい。 Here, the display of the thickness map 702, the En-Face image 703, the thickness graph 712, and the boundaries 715 and 716 may be generated by the image processing device 20 based on the boundaries and the retinal layers detected by the processing unit 222. Here is an example of what can be done. In addition, as a generation method for generating these, any known method may be adopted.
　なお、表示部５０の表示画面７００には、これらに加えて患者タブ、撮影タブ、レポートタブ、及び設定タブ等を設けてもよい。この場合、図７の表示画面７００に示されている内容は、レポートタブに表示されることとなる。また、表示画面７００には、患者情報表示部、検査ソートタブ、及び検査リスト等を表示することもできる。検査リストには、眼底画像や断層画像、ＯＣＴＡ画像のサムネイルを表示してもよい。 The display screen 700 of the display unit 50 may include a patient tab, an imaging tab, a report tab, a setting tab, and the like in addition to the above. In this case, the contents shown on the display screen 700 in FIG. 7 are displayed on the report tab. Further, the display screen 700 can also display a patient information display section, an examination sort tab, an examination list, and the like. The examination list may display thumbnails of a fundus image, a tomographic image, and an OCTA image.
　次に、ステップＳ３０６において、取得部２１は、画像処理システム１による断層画像の撮影に係る一連の処理を終了するか否かの指示を外部から取得する。この指示は、入力部６０を用いて、操作者によって入力されることができる。取得部２１が、処理を終了する指示を取得した場合には、画像処理システム１は本実施例に係る一連の処理を終了する。一方、取得部２１が、処理を終了しない指示を取得した場合には、ステップＳ３０２に処理を戻して撮影を続行する。 Next, in step S306, the acquisition unit 21 acquires an instruction from the outside as to whether or not to end a series of processes relating to imaging of a tomographic image by the image processing system 1. This instruction can be input by the operator using the input unit 60. When the acquisition unit 21 acquires an instruction to end the processing, the image processing system 1 ends a series of processing according to the present embodiment. On the other hand, if the acquisition unit 21 acquires an instruction not to end the process, the process returns to step S302 to continue shooting.
　上記のように、本実施例に係る画像処理装置２０は、取得部２１と、処理部２２２（第一の処理部）とを備える。取得部２１は、被検眼の断層画像を取得する。処理部２２２は、学習済モデルを用いて、断層画像において被検眼の複数の網膜層のうち少なくとも一つの網膜層を検出するための第一の検出処理を実行する。 As described above, the image processing device 20 according to the present embodiment includes the acquisition unit 21 and the processing unit 222 (first processing unit). The acquisition unit 21 acquires a tomographic image of the subject's eye. The processing unit 222 executes a first detection process for detecting at least one retinal layer among a plurality of retinal layers of the subject's eye in the tomographic image using the learned model.
　学習済モデルを用いて画像セグメンテーション処理を行う場合、例えば、疾病眼における病変による層構造の変化についても、学習した傾向に応じて適切に境界検出を行うことができる。このため、本実施例に係る画像処理装置２０では、学習済モデルを用いて画像セグメンテーション処理を行うことで、疾患や部位等によらず境界検出を行うことができ、境界検出の精度を向上させることができる。 In the case where the image segmentation processing is performed using the learned model, for example, even for a change in the layer structure due to a lesion in the diseased eye, the boundary can be appropriately detected according to the learned tendency. For this reason, in the image processing device 20 according to the present embodiment, by performing the image segmentation process using the learned model, it is possible to perform the boundary detection regardless of the disease, the part, and the like, and to improve the accuracy of the boundary detection. be able to.
　また、画像処理装置２０は、被検眼の三次元の断層画像における少なくとも一部の深度範囲であって、検出された少なくとも一つの網膜層に基づいて決定された深度範囲に対応する正面画像を生成する画像処理部２２を更に備える。画像処理部２２は、三次元の断層画像に対応する三次元のモーションコントラストデータを用いて、決定された深度範囲に対応するモーションコントラスト正面画像を生成することができる。 The image processing device 20 generates a front image corresponding to at least a part of the depth range in the three-dimensional tomographic image of the eye to be inspected and corresponding to the depth range determined based on at least one detected retinal layer. The image processing unit 22 further includes an image processing unit 22. The image processing unit 22 can generate a motion contrast front image corresponding to the determined depth range using the three-dimensional motion contrast data corresponding to the three-dimensional tomographic image.
　なお、本実施例では、１つの学習済モデルを用いて画像セグメンテーション処理を行う構成について説明したが、複数の学習済モデルを用いて画像セグメンテーション処理を行ってもよい。 In the present embodiment, the configuration in which the image segmentation process is performed using one learned model has been described. However, the image segmentation process may be performed using a plurality of learned models.
　学習済モデルは、上述のように教師データを用いた学習の傾向に従って出力データを生成するため、特徴について似た傾向を有する教師データを用いて学習を行うことで、出力データに対する学習の傾向の再現性を高めることができる。このため、例えば、撮影部位毎に学習を行った複数の学習済モデルを用いて、対応する撮影部位の断層画像について画像セグメンテーション処理を行うことで、より精度の良い境界画像を生成することができる。この場合には、画像処理システムはより精度よく網膜層を検出することができる。また、この場合には、追加で学習モデルを増やしていくことも可能であるため、性能が徐々に向上するようなバージョンアップを行うことも期待できる。 The trained model generates output data according to the tendency of learning using teacher data as described above. Reproducibility can be improved. Therefore, for example, by performing image segmentation processing on a tomographic image of a corresponding imaging region using a plurality of learned models that have learned for each imaging region, a more accurate boundary image can be generated. . In this case, the image processing system can more accurately detect the retinal layer. In this case, it is also possible to increase the number of learning models additionally, so that it is expected that a version upgrade in which performance is gradually improved can be performed.
　さらに、処理部２２２は、断層画像における硝子体側の領域、網膜領域、及び強膜側の領域等の領域毎に学習を行った複数の学習モデルを用い、それぞれの学習モデルの出力を合わせて処理部２２２の最終的な出力を生成してもよい。この場合には、領域毎により精度の高い境界画像を生成することができるため、より精度よく網膜層を検出することができる。 Further, the processing unit 222 uses a plurality of learning models that have been trained for each region such as a vitreous body region, a retinal region, and a sclera region in the tomographic image, and processes the combined learning model outputs. A final output of the unit 222 may be generated. In this case, a more accurate boundary image can be generated for each region, so that the retinal layer can be detected more accurately.
　また、本実施例では、機械学習モデルとして画像セグメンテーション処理を行うものについて説明したが、例えば、断層画像の撮影部位を推定するような機械学習モデルを用いることもできる。 Further, in the present embodiment, the description has been given of the case where the image segmentation processing is performed as the machine learning model. However, for example, a machine learning model for estimating the imaging region of the tomographic image may be used.
　一般に、機械学習モデルの構成は、入力データである画像に対応する画像を出力する構成に限られない。例えば、入力データに対して、教師データを用いてトレーニングされた出力データの種類を出力したり、当該種類のそれぞれについて可能性を数値として出力したりするように機械学習モデルが構成されてもよい。このため、教師データを構成するペア群の入力データと出力データの形式や組み合わせは、一方が画像で他方が数値であったり、一方が複数の画像群で構成され他方が文字列であったり、双方が画像であったりする等、利用形態に適したものとすることができる。 Generally, the configuration of a machine learning model is not limited to a configuration that outputs an image corresponding to an image that is input data. For example, a machine learning model may be configured to output a type of output data trained using teacher data with respect to input data, or to output a possibility as a numerical value for each of the types. . For this reason, the format and combination of the input data and the output data of the pair group that constitutes the teacher data may be such that one is an image and the other is a numerical value, one is a plurality of image groups and the other is a character string, Both can be images, and so on, which can be suitable for the usage form.
　撮影部位を推定する機械学習モデルの教師データの例として、具体的には、ＯＣＴによって取得された断層画像と、断層画像に対応する撮影部位ラベルとのペア群によって構成された教師データが挙げられる。ここで、撮影部位ラベルは部位を表すユニークな数値や文字列である。このような教師データを用いてトレーニングされた学習済モデルに、ＯＣＴを用いて取得された断層画像を入力すると、画像に撮影されている部位の撮影部位ラベルが出力されたり、設計によっては、撮影部位ラベル毎の確率が出力されたりする。 Specifically, as an example of the teacher data of the machine learning model for estimating the imaging region, there is teacher data configured by a pair group of a tomographic image acquired by OCT and an imaging region label corresponding to the tomographic image. . Here, the imaging part label is a unique numerical value or character string representing the part. When a tomographic image acquired using OCT is input to a trained model trained using such teacher data, an imaging region label of a region photographed in the image is output. For example, the probability of each part label is output.
　処理部２２２は、このような撮影部位を推定する学習済モデルを更に用いて断層画像の撮影部位を推定し、推定された撮影部位や最も確率の高い撮影部位に応じた学習済モデルを用いて画像セグメンテーション処理を行ってもよい。このような構成では、断層画像の撮影部位に関する撮影条件を取得部２１が取得できない場合であっても、断層画像から撮影部位を推定し、撮影部位に対応する画像セグメンテーション処理を行うことで、より精度良く網膜層を検出することができる。 The processing unit 222 further estimates the imaging region of the tomographic image using such a learned model for estimating the imaging region, and uses the learned model corresponding to the estimated imaging region or the imaging region with the highest probability. Image segmentation processing may be performed. In such a configuration, even when the acquisition unit 21 cannot acquire the imaging conditions related to the imaging region of the tomographic image, the imaging region is estimated from the tomographic image, and the image segmentation process corresponding to the imaging region is performed. The retinal layer can be accurately detected.
（実施例２）
　実施例１においては、学習済モデルを用いて、断層画像から対象となる全ての網膜層を検出する画像セグメンテーション処理を行った。これに対し、実施例２では、学習済モデルによる網膜領域の検出結果に基づいて、ルールベースの画像特徴による境界検出を行う。 (Example 2)
In the first embodiment, an image segmentation process for detecting all target retinal layers from a tomographic image is performed using a trained model. On the other hand, in the second embodiment, based on the result of detection of the retinal region by the learned model, boundary detection is performed using rule-based image features.
　従来、視神経乳頭部においては、ＯＣＴ画像を用いてＣｕｐ（視神経乳頭陥凹）とＤｉｓｃ（視神経乳頭）の検出を行う際にブルッフ膜の開口端の検出を行うことが通例であるが、乳頭周囲網脈絡膜萎縮などの場合、その検出が困難な場合があった。 Conventionally, in the optic papilla, it is customary to detect the open end of the Bruch's membrane when detecting Cup (optic papilla) and Disc (optic papilla) using OCT images. In some cases, such as retinal atrophy, it is difficult to detect it.
　また、従来のルールベースの画像セグメンテーション処理では、被検眼の個体差や病変に対するロバスト性が低く初めに網膜領域を誤検出してしまうことがあった。この場合には、その後の網膜内層境界の検出が適切に行えなかった。 In addition, in the conventional rule-based image segmentation processing, robustness with respect to individual differences and lesions of an eye to be examined is low, and a retinal region may be erroneously detected first. In this case, the subsequent detection of the retinal inner layer boundary could not be performed properly.
　これに対し、機械学習モデルを用いて画像セグメンテーション処理を行うことで、境界検出の精度を向上させることができる。しかしながら、ディープラーニング等の機械学習アルゴリズムによる機械学習モデルを用いて撮像部位の認識や網膜層の境界検出を行う場合、医療画像分野であることから、正解付きの正常及び病変画像の症例数を集めるのが大変困難であるのが一般的である。さらに、学習のための正解データを作成するのにも時間を要する。 に 対 し On the other hand, by performing image segmentation processing using a machine learning model, it is possible to improve the accuracy of boundary detection. However, when performing recognition of an imaging part and detection of a retinal layer boundary using a machine learning model based on a machine learning algorithm such as deep learning, the number of cases of normal and lesion images with a correct answer is collected because of the medical image field. It is generally very difficult to do so. Furthermore, it takes time to create correct answer data for learning.
　そこで、本実施例では、学習済モデルを用いて網膜領域を検出し、検出した網膜領域に対して、画像特徴による境界検出を併用する。これにより、網膜領域の誤検出を抑制し、網膜内層境界の検出精度を向上させるとともに、機械学習の過程において、学習時には網膜層かそれ以外の正解データを作成するだけで済むため、学習を効率的に行うことができる。 Therefore, in this embodiment, the retinal region is detected using the learned model, and the detected retinal region is used together with the boundary detection based on the image feature. As a result, erroneous detection of the retinal region is suppressed, the detection accuracy of the inner layer of the retina is improved, and in the process of machine learning, only the retinal layer or other correct data is created at the time of learning. Can be done
　以下、図８乃至図１３Ｄを参照して、本実施例に係る画像処理システム８について説明する。以下、本実施例に係る画像処理システムによる画像処理について、実施例１に係る画像処理との違いを中心として説明する。なお、実施例１に係る画像処理システム１の構成及び処理と同様である本実施例による画像処理システムの構成及び処理については、同一の参照符号を用いて示し、説明を省略する。 Hereinafter, the image processing system 8 according to the present embodiment will be described with reference to FIGS. 8 to 13D. Hereinafter, image processing by the image processing system according to the present embodiment will be described focusing on differences from the image processing according to the first embodiment. Note that the configuration and processing of the image processing system according to the present embodiment that are the same as the configuration and processing of the image processing system 1 according to the first embodiment are denoted by the same reference numerals, and description thereof is omitted.
　図８は、本実施例に係る画像処理システム８の概略的な構成の一例を示す。画像処理システム８では、画像処理装置８０の画像処理部８２において、処理部２２２に代えて、第一の処理部８２２及び第二の処理部８２３が設けられている。 FIG. 8 shows an example of a schematic configuration of the image processing system 8 according to the present embodiment. In the image processing system 8, a first processing unit 822 and a second processing unit 823 are provided instead of the processing unit 222 in the image processing unit 82 of the image processing device 80.
　第一の処理部８２２は、ディープラーニング等の機械学習アルゴリズムによる機械学習モデルに関する学習済モデルを有し、学習済モデルを用いて断層画像における網膜領域を検出する。第二の処理部８２３は、第一の処理部８２２によって検出された網膜領域について、画像特徴抽出の結果をルールベースで判断して網膜層の境界検出を行う。 The first processing unit 822 has a trained model related to a machine learning model by a machine learning algorithm such as deep learning, and detects a retinal region in a tomographic image using the trained model. The second processing unit 823 determines the result of the image feature extraction for the retinal region detected by the first processing unit 822 on a rule basis, and performs retinal layer boundary detection.
　次に、図９Ａ及び図９Ｂを参照して、本実施例に係る一連の処理について説明する。図９Ａは本実施例に係る一連の処理のフローチャートであり、図９Ｂは本実施例における境界検出処理のフローチャートである。なお、境界検出処理以外の処理については実施例１の処理と同様であるため、説明を省略する。ステップＳ３０３において断層画像が生成されると、処理はステップＳ９０４に移行する。 Next, a series of processes according to the present embodiment will be described with reference to FIGS. 9A and 9B. FIG. 9A is a flowchart of a series of processes according to the present embodiment, and FIG. 9B is a flowchart of a boundary detection process in the present embodiment. Note that the processing other than the boundary detection processing is the same as the processing of the first embodiment, and thus the description is omitted. When a tomographic image is generated in step S303, the process proceeds to step S904.
　ステップＳ９０４における境界検出処理が開始されると、処理はステップＳ９４１に移行する。ステップＳ９４１では、第一の処理部８２２が、第一の境界検出処理として、学習済モデルを用いて断層画像における網膜領域を検出する。 When the boundary detection processing in step S904 is started, the processing shifts to step S941. In step S941, the first processing unit 822 detects a retinal region in a tomographic image using a learned model as a first boundary detection process.
　ここで、図１０Ａ乃至図１２を参照して、本実施例に係る機械学習モデルについて説明する。本実施例に係る機械学習モデルの学習データ（教師データ）は、１つ以上の入力データと出力データとのペア群で構成される。教師データの例として、図１０Ａに示すＯＣＴの撮影によって取得された断層画像１００１と、図１０Ｂに示す断層画像１００１から任意の層にラベルを与えたラベル画像１００２とのペア群によって構成されている教師データ等が挙げられる。 Here, the machine learning model according to the present embodiment will be described with reference to FIGS. 10A to 12. The learning data (teacher data) of the machine learning model according to the present embodiment includes one or more pairs of input data and output data. As an example of the teacher data, a pair group of a tomographic image 1001 obtained by the OCT imaging illustrated in FIG. 10A and a label image 1002 in which a label is assigned to an arbitrary layer from the tomographic image 1001 illustrated in FIG. 10B is configured. Teacher data and the like.
　ここで、ラベル画像とは画素毎にラベル付けがなされた画像（アノテーションして得た画像）であり、本実施例では、画素毎に当該画素に現れている（撮影されている）像に関するラベルが与えられた画像をいう。ラベル画像１００２においては、ラベルの例として網膜よりも浅層側（硝子体側）のラベル１００３、網膜内層のラベル１００４、及び網膜よりも深層側（脈絡膜側）のラベル１００５が与えられている。本実施例における第一の処理部８２２は、このようなラベル画像に基づいて網膜内層を検出する。なお、本実施例では網膜の範囲（網膜内層の範囲）を内境界膜と神経線維層との境界Ｌ１～網膜色素上皮層Ｌ４としたが、これに限らない。例えば、網膜の範囲を内境界膜と神経線維層との境界Ｌ１～視細胞内節外節接合部Ｌ３の範囲、内境界膜と神経線維層との境界Ｌ１～ブルッフ膜Ｌ５の範囲、又は内境界膜と神経線維層との境界Ｌ１～脈絡膜Ｌ６の範囲等と定義してもよい。 Here, the label image is an image that is labeled for each pixel (an image obtained by annotation), and in the present embodiment, a label related to an image appearing (photographed) at the pixel for each pixel. Refers to the image given. In the label image 1002, as examples of labels, a label 1003 on the shallower side (vitreous body side) than the retina, a label 1004 on the inner layer of the retina, and a label 1005 on the deeper side (choroidal side) than the retina are given. The first processing unit 822 in the present embodiment detects the retinal inner layer based on such a label image. In this embodiment, the range of the retina (the range of the inner layer of the retina) is defined as the boundary L1 between the inner limiting membrane and the nerve fiber layer to the retinal pigment epithelium layer L4, but is not limited thereto. For example, the range of the retina is defined as the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the range of the junction between the photoreceptor inner and outer segments L3, the range from the boundary L1 between the inner limiting membrane and the nerve fiber layer to the Bruch's membrane L5, or It may be defined as a range from the boundary L1 between the limiting membrane and the nerve fiber layer to the choroid L6.
　さらに、図１０Ａ及び図１０Ｂにおいては、網膜のＸＹ面内においてある一つのＸＺ断面の例を示しているが、断面はこれに限らない。図示しないが、ＸＹ面内における任意の複数のＸＺ断面を事前に学習しておき、ラスタスキャンやラジアルスキャン等、異なる様々なスキャンパターンで撮影された断面に対して対応できるようにしておくことができる。例えば、ラスタスキャンで三次元的に網膜を撮影した断層画像等のデータを用いる場合には、隣接する複数の断層画像間の位置合わせをしたボリュームデータを教師データに用いることができる。この場合には、１つのボリュームデータとこれに対応する１つの三次元ラベルデータ（三次元のラベル画像）とから、任意の角度のペア画像群を生成することが可能である。また、機械学習モデルは、実際に様々なスキャンパターンで撮影した画像を教師画像として用いて学習してもよい。 10A and 10B show an example of one XZ cross section in the XY plane of the retina, but the cross section is not limited to this. Although not shown, arbitrary plural XZ sections in the XY plane may be learned in advance so that sections corresponding to various different scan patterns such as a raster scan and a radial scan can be handled. it can. For example, when using data such as a tomographic image obtained by three-dimensionally photographing the retina by raster scanning, volume data obtained by aligning a plurality of adjacent tomographic images can be used as teacher data. In this case, it is possible to generate a pair image group of an arbitrary angle from one volume data and one corresponding three-dimensional label data (three-dimensional label image). Further, the machine learning model may learn using images actually captured with various scan patterns as teacher images.
　次に、学習時の画像について説明する。機械学習モデルの教師データを構成する、断層画像１００１とラベル画像１００２とのペア群を構成する画像群を、位置関係が対応する一定の画素サイズの矩形領域画像によって作成する。当該画像の作成について、図１１Ａ乃至図１１Ｃを参照して説明する。 Next, the image at the time of learning will be described. An image group forming a pair group of a tomographic image 1001 and a label image 1002, which constitutes teacher data of a machine learning model, is created by a rectangular area image having a fixed pixel size corresponding to a positional relationship. The creation of the image will be described with reference to FIGS. 11A to 11C.
　まず、教師データを構成するペア群の１つを、断層画像１００１とラベル画像１００２とした場合について説明する。この場合には、図１１Ａに示すように、断層画像１００１の全体である矩形領域画像１１０１を入力データ、ラベル画像１００２の全体である矩形領域画像１１０２を出力データとして、ペアを構成する。なお、図１１Ａに示す例では各画像の全体により入力データと出力データのペアを構成しているが、ペアはこれに限らない。 First, a case will be described in which one of the group of pairs forming the teacher data is a tomographic image 1001 and a label image 1002. In this case, as shown in FIG. 11A, a pair is formed using the rectangular area image 1101 that is the entire tomographic image 1001 as input data and the rectangular area image 1102 that is the entire label image 1002 as output data. In the example shown in FIG. 11A, a pair of input data and output data is formed by the entire image, but the pair is not limited to this.
　例えば、図１１Ｂに示すように、断層画像１００１のうちの矩形領域画像１１１１を入力データ、ラベル画像１００２における対応する撮影領域である矩形領域画像１１１３を出力データとして、ペアを構成してもよい。矩形領域画像１１１１，１１１３は、Ａスキャン単位を基本としている。Ａスキャン単位とは、１本のＡスキャン単位でもよいし、数本のＡスキャン単位でもよい。 For example, as shown in FIG. 11B, a pair may be formed using a rectangular area image 1111 of the tomographic image 1001 as input data and a rectangular area image 1113 which is a corresponding imaging area in the label image 1002 as output data. The rectangular area images 1111 and 1113 are based on A-scan units. The A scan unit may be one A scan unit or several A scan units.
　なお、図１１ＢではＡスキャン単位を基本としているが、画像に対して深さ方向の全てを領域とするのではなく、上下に矩形領域外の部分を設けてもよい。すなわち、矩形領域の横方向のサイズはＡスキャン数本分、矩形領域の深さ方向のサイズは、画像の深さ方向のサイズよりも小さく設定してもよい。 Note that although FIG. 11B is based on the unit of A-scan, the whole area in the depth direction of the image may not be the entire area, but a part outside the rectangular area may be provided at the top and bottom. That is, the size of the rectangular area in the horizontal direction may be set to several A scans, and the size of the rectangular area in the depth direction may be set smaller than the size of the image in the depth direction.
　また、図１１Ｃに示すように、断層画像１００１のうちの矩形領域画像１１２１を入力データ、ラベル画像１００２における対応する撮影領域である矩形領域画像１１２３を出力データとして、ペアを構成してもよい。この場合、矩形領域のサイズは、１つの矩形領域内に複数のラベルを含むサイズとする。 11C, as shown in FIG. 11C, a pair may be formed using the rectangular area image 1121 of the tomographic image 1001 as input data and the rectangular area image 1123 which is a corresponding imaging area in the label image 1002 as output data. In this case, the size of the rectangular area is a size that includes a plurality of labels in one rectangular area.
　なお、学習時には、スキャン範囲（撮影画角）、スキャン密度（Ａスキャン数）を正規化して画像サイズを揃えて、学習時の矩形領域サイズを一定に揃えることができる。また、図１１Ａ乃至図１１Ｃに示した矩形領域画像は、それぞれ別々に学習する際の矩形領域サイズの一例である。 At the time of learning, the scan range (angle of view) and the scan density (the number of A-scans) are normalized to make the image size uniform, so that the rectangular area size at the time of learning can be made uniform. The rectangular area images shown in FIGS. 11A to 11C are examples of the rectangular area size when learning is performed separately.
　矩形領域の数は、図１１Ａに示す例では１つ、図１１Ｂ及び図１１Ｃに示す例では複数設定可能である。例えば、図１１Ｂに示す例において、断層画像１００１のうちの矩形領域画像１１１２を入力データ、ラベル画像１００２における対応する撮影領域である矩形領域画像１１１４を出力データとしてペアを構成することもできる。また、例えば、図１１Ｃに示す例において、断層画像１００１のうちの矩形領域画像１１２２を入力データ、ラベル画像１００２における対応する撮影領域である矩形領域画像１１２４を出力データとしてペアを構成することもできる。このように、１枚ずつの断層画像及びラベル画像のペアから、互いに異なる矩形領域画像のペアを作成できる。なお、元となる断層画像及びラベル画像において、領域の位置を異なる座標に変えながら多数の矩形領域画像のペアを作成することで、教師データを構成するペア群を充実させることができる。 11) The number of rectangular areas can be set to one in the example shown in FIG. 11A, and a plurality can be set in the examples shown in FIGS. 11B and 11C. For example, in the example shown in FIG. 11B, a pair can be formed by using a rectangular area image 1112 of the tomographic image 1001 as input data and a rectangular area image 1114 as a corresponding imaging area in the label image 1002 as output data. Further, for example, in the example illustrated in FIG. 11C, a pair can be formed using the rectangular area image 1122 of the tomographic image 1001 as input data and the rectangular area image 1124 that is the corresponding imaging area in the label image 1002 as output data. . Thus, a pair of mutually different rectangular area images can be created from a pair of a tomographic image and a label image one by one. In the original tomographic image and label image, by creating a large number of pairs of rectangular area images while changing the position of the area to different coordinates, it is possible to enrich the group of pairs forming the teacher data.
　図１１Ｂ及び図１１Ｃに示す例では、離散的に矩形領域を示しているが、実際には、元となる断層画像及びラベル画像を、隙間なく連続する一定の画像サイズの矩形領域画像群に分割することができる。また、元となる断層画像及びラベル画像について、互いに対応する、ランダムな位置の矩形領域画像群に分割してもよい。このように、矩形領域（又は、短冊領域）として、より小さな領域の画像を入力データ及び出力データのペアとして選択することで、もともとのペアを構成する断層画像１００１及びラベル画像１００２から多くのペアデータを生成できる。そのため、機械学習モデルのトレーニングにかかる時間を短縮することができる。一方で、完成した機械学習モデルの学習済モデルでは、実行する画像セグメンテーション処理の時間が長くなる傾向にある。 In the examples shown in FIGS. 11B and 11C, rectangular areas are discretely shown. However, in reality, the original tomographic image and label image are divided into a group of rectangular area images having a constant image size and without gaps. can do. Further, the original tomographic image and label image may be divided into a group of rectangular area images at random positions corresponding to each other. As described above, by selecting an image of a smaller area as a rectangular area (or a strip area) as a pair of input data and output data, many pairs are obtained from the tomographic image 1001 and the label image 1002 that constitute the original pair. Can generate data. Therefore, the time required for training the machine learning model can be reduced. On the other hand, in the trained model of the completed machine learning model, the time of the executed image segmentation process tends to be long.
　次に、本実施例に係る機械学習モデルの一例として、入力された断層画像に対して、セグメンテーション処理を行う畳み込みニューラルネットワーク（ＣＮＮ）の構成を、図１２を参照して説明する。図１２は、第一の処理部８２２における機械学習モデルの構成１２０１の一例を示している。なお、本実施例に係る機械学習モデルとしては、実施例１と同様に、例えば、ＦＣＮ、又はＳｅｇＮｅｔ等を用いることもできる。また、所望の構成に応じて、実施例１で述べたような領域単位で物体認識を行う機械学習モデルを用いてもよい。 Next, as an example of a machine learning model according to the present embodiment, a configuration of a convolutional neural network (CNN) that performs a segmentation process on an input tomographic image will be described with reference to FIG. FIG. 12 illustrates an example of a configuration 1201 of a machine learning model in the first processing unit 822. Note that, as in the first embodiment, for example, FCN, SegNet, or the like can be used as the machine learning model according to the present embodiment. Further, according to a desired configuration, a machine learning model that performs object recognition in units of regions as described in the first embodiment may be used.
　図１２に示す機械学習モデルは、図６に示す実施例１に係る機械学習モデルの例と同様に、入力値群を加工して出力する処理を担う、複数の層群によって構成される。当該機械学習モデルの構成１２０１に含まれる層の種類としては、畳み込み層、ダウンサンプリング層、アップサンプリング層、及び合成層がある。なお、これら層の構成や、ＣＮＮの構成の変形例については、実施例１に係る機械学習モデルと同様であるため詳細な説明については省略する。なお、本実施例で用いるＣＮＮの構成１２０１は、実施例１で述べたＣＮＮの構成６０１と同様に、Ｕ－ｎｅｔ型の機械学習モデルである。 機械 The machine learning model shown in FIG. 12 is configured by a plurality of layer groups that are responsible for processing of processing the input value group and outputting the same as in the example of the machine learning model according to the first embodiment shown in FIG. The types of layers included in the configuration 1201 of the machine learning model include a convolution layer, a downsampling layer, an upsampling layer, and a composite layer. Note that the configurations of these layers and the modifications of the configuration of the CNN are the same as those of the machine learning model according to the first embodiment, and thus detailed descriptions thereof will be omitted. The CNN configuration 1201 used in the present embodiment is a U-net type machine learning model, like the CNN configuration 601 described in the first embodiment.
　本実施例に係る第一の処理部８２２の学習済モデルでは、断層画像１００１が入力されると、教師データを用いてトレーニングされた傾向に従って、ラベル画像１００２を出力する。第一の処理部８２２は、ラベル画像１００２に基づいて断層画像１００１における網膜領域を検出することができる。 In the learned model of the first processing unit 822 according to the present embodiment, when the tomographic image 1001 is input, a label image 1002 is output according to the tendency trained using the teacher data. The first processing unit 822 can detect a retinal region in the tomographic image 1001 based on the label image 1002.
　なお、図１１Ｂ及び図１１Ｃに示すように、画像の領域を分割して学習している場合、第一の処理部８２２は学習済モデルを用いて、それぞれの矩形領域に対応するラベル画像である矩形領域画像を得る。そのため、第一の処理部８２２は、各矩形領域において網膜層を検出することができる。この場合、第一の処理部８２２は、学習済モデルを用いて得たラベル画像である矩形領域画像群のそれぞれを、矩形領域画像群のぞれぞれと同様の位置関係に配置して結合する。これにより、第一の処理部８２２は、入力された断層画像１００１に対応するラベル画像１００２を生成することができる。この場合も、第一の処理部８２２は、生成されたラベル画像１００２に基づいて断層画像１００１における網膜領域を検出することができる。 As shown in FIGS. 11B and 11C, when learning is performed by dividing the image region, the first processing unit 822 uses the learned model to generate a label image corresponding to each rectangular region. Obtain a rectangular area image. Therefore, the first processing unit 822 can detect a retinal layer in each rectangular area. In this case, the first processing unit 822 arranges and combines each of the rectangular area image groups, which are the label images obtained using the learned model, in the same positional relationship as each of the rectangular area image groups. I do. Thereby, the first processing unit 822 can generate the label image 1002 corresponding to the input tomographic image 1001. Also in this case, the first processing unit 822 can detect a retinal region in the tomographic image 1001 based on the generated label image 1002.
　ステップＳ９４１において、第一の処理部８２２によって網膜領域が検出されると、処理はステップＳ９４２に移行する。ステップＳ９４２では、第二の処理部８２３が、第二の検出処理として、図１０Ａに示す断層画像１００１において第一の処理部８２２が検出した網膜領域に基づいて、ルールベースの画像セグメンテーション処理により網膜内層での残りの境界を検出する。 In step S941, when the retinal region is detected by the first processing unit 822, the process proceeds to step S942. In step S942, the second processing unit 823 performs, as a second detection process, a retinal region by a rule-based image segmentation process based on the retinal region detected by the first processing unit 822 in the tomographic image 1001 illustrated in FIG. 10A. Detect remaining boundaries in inner layers.
　図１３Ａ乃至図１３Ｄを参照して、第二の処理部８２３による第二の境界検出処理について説明する。図１３Ａは入力となる断層画像の一例である断層画像１００１を示す。図１３Ｂは第一の処理部８２２が出力したラベル画像１００２であり、網膜領域のラベル１００４とそれ以外に対応するラベル１００３，１００５を付与した画像である。本実施例に係る第二の処理部８２３は、ラベル画像１００２におけるラベル１００４で示される網膜領域の範囲を層検出の対象領域とする。 The second boundary detection processing by the second processing unit 823 will be described with reference to FIGS. 13A to 13D. FIG. 13A shows a tomographic image 1001 which is an example of an input tomographic image. FIG. 13B is a label image 1002 output by the first processing unit 822, which is an image to which labels 1004 of the retina region and labels 1003 and 1005 corresponding to the other are added. The second processing unit 823 according to the present embodiment sets the range of the retinal region indicated by the label 1004 in the label image 1002 as the target region for layer detection.
　第二の処理部８２３は、ラベル画像１００２におけるラベル１００４で示される網膜領域内の輪郭を検出することで、対象となる境界を検出することができる。図１３Ｃは第二の処理部８２３が、処理を行ったエッジ強調処理画像１３０３を示す。当該第二の処理部８２３による処理について、以下で説明する。なお、図１３Ｃ及び図１３Ｄに示すように、視神経乳頭部については、網膜層が途切れるため、第二の処理部８２３による境界検出を行わないこととする。 The second processing unit 823 can detect the target boundary by detecting the contour in the retinal region indicated by the label 1004 in the label image 1002. FIG. 13C shows an edge-enhanced image 1303 that has been processed by the second processing unit 823. The processing by the second processing unit 823 will be described below. As shown in FIGS. 13C and 13D, the retinal layer of the optic disc is interrupted, so that the second processing unit 823 does not perform the boundary detection.
　第二の処理部８２３は、処理の対象とする断層画像１００１において、ラベル１００４に対応する領域に対して、ノイズ除去とエッジ強調処理を行う。第二の処理部８２３は、ノイズ除去処理としては、例えばメディアンフィルタやガウシアンフィルタを適用する。また、第二の処理部８２３は、エッジ強調処理としては、ＳｏｂｅｌフィルタやＨｅｓｓｉａｎフィルタを適用する。 The second processing unit 823 performs noise removal and edge enhancement on the area corresponding to the label 1004 in the tomographic image 1001 to be processed. The second processing unit 823 applies, for example, a median filter or a Gaussian filter as the noise removal processing. Further, the second processing unit 823 applies a Sobel filter or a Hessian filter as the edge enhancement processing.
　ここで、二次元のＨｅｓｓｉａｎフィルタを用いた、二次元断層画像に対するエッジ強調処理について説明する。Ｈｅｓｓｉａｎフィルタは、ヘッセ行列の２つの固有値（λ１、λ２）の関係に基づいて、二次元濃淡分布の二次局所構造を強調することができる。そのため、本実施例では、ヘッセ行列の固有値と固有ベクトル（ｅ１、ｅ２）の関係を用いて、二次元の線構造を強調する。被検眼についての二次元断層画像における線構造は網膜層の構造に相当するため、当該Ｈｅｓｓｉａｎフィルタの適用により、網膜層の構造を強調することができる。 Here, edge enhancement processing on a two-dimensional tomographic image using a two-dimensional Hessian filter will be described. The Hessian filter can emphasize a secondary local structure of a two-dimensional grayscale distribution based on a relationship between two eigenvalues (λ 1 , λ 2 ) of the Hessian matrix. Therefore, in this embodiment, the two-dimensional line structure is emphasized using the relationship between the eigenvalues of the Hessian matrix and the eigenvectors (e 1 , e 2 ). Since the line structure in the two-dimensional tomographic image of the eye to be examined corresponds to the structure of the retinal layer, the structure of the retinal layer can be emphasized by applying the Hessian filter.
　なお、厚みの異なる網膜層を検出するには、ヘッセ行列を計算する際に行うガウス関数による平滑化の解像度を変更すればよい。また、二次元のＨｅｓｓｉａｎフィルタを適用する際には、画像のＸＺの物理サイズを合わせるようにデータを変形した後に適用することができる。一般的なＯＣＴの場合、ＸＹ方向とＺ方向の物理サイズが異なる。そのため、画素毎の網膜層の物理サイズを合わせてフィルタを適用する。なお、ＸＹ方向とＺ方向の物理サイズは、ＯＣＴ装置１０の設計／構成から把握できるため、当該物理サイズに基づいて、断層画像のデータを変形させることができる。また、物理サイズを正規化しない場合には、ガウス関数による平滑化の解像度を変更することでも近似的に対応できる。 In order to detect retinal layers having different thicknesses, the resolution of the smoothing by the Gaussian function performed when calculating the Hessian matrix may be changed. When a two-dimensional Hessian filter is applied, it can be applied after data is deformed to match the physical size of XZ of the image. In the case of general OCT, the physical sizes in the XY direction and the Z direction are different. Therefore, the filter is applied by matching the physical size of the retinal layer for each pixel. Since the physical size in the XY direction and the Z direction can be grasped from the design / configuration of the OCT apparatus 10, the data of the tomographic image can be deformed based on the physical size. In the case where the physical size is not normalized, it can be approximately handled by changing the resolution of smoothing by the Gaussian function.
　上記では、二次元断層画像での処理について説明したが、Ｈｅｓｓｉａｎフィルタを適用する対象はこれに限らない。断層画像を撮影した際のデータ構造がラスタスキャンによる三次元断層画像である場合、三次元のＨｅｓｓｉａｎフィルタを適用することも可能である。この場合、画像処理部８２によって、隣接する断層画像間においてＸＺ方向の位置合わせ処理を行った後に、第二の処理部８２３がヘッセ行列の３つの固有値（λ１、λ２、λ３）の関係に基づいて、三次元濃淡分布の二次局所構造を強調することができる。そのため、ヘッセ行列の固有値と固有ベクトル（ｅ１、ｅ２、ｅ３）の関係を用いて三次元の層構造を強調することで、三次元的にエッジを強調することも可能である。 In the above, the processing using the two-dimensional tomographic image has been described, but the target to which the Hessian filter is applied is not limited to this. When the data structure at the time of capturing the tomographic image is a three-dimensional tomographic image obtained by raster scanning, a three-dimensional Hessian filter can be applied. In this case, after performing the alignment process in the XZ direction between the adjacent tomographic images by the image processing unit 82, the second processing unit 823 outputs the three eigenvalues (λ 1 , λ 2 , λ 3 ) of the Hessian matrix. Based on the relation, the secondary local structure of the three-dimensional grayscale distribution can be emphasized. Therefore, by emphasizing the three-dimensional layer structure using the relationship between the eigenvalues of the Hessian matrix and the eigenvectors (e 1 , e 2 , e 3 ), it is also possible to emphasize the edges three-dimensionally.
　エッジ強調処理画像１３０３においては、エッジを強調した部分が白線１３０４として現れる。なお、断層画像１００１における、ラベル１００４に対応しない領域については、エッジ検出されない領域として扱うことができる。また、ここでは、Ｈｅｓｓｉａｎフィルタを用いてエッジ強調処理を行う構成について説明したが、エッジ強調処理の処理方法はこれに限られず、既存の任意の方法によって行われてよい。 In the edge-enhanced image 1303, a portion where the edge is enhanced appears as a white line 1304. Note that an area that does not correspond to the label 1004 in the tomographic image 1001 can be treated as an area where no edge is detected. Further, here, the configuration in which the edge enhancement processing is performed using the Hessian filter has been described, but the processing method of the edge enhancement processing is not limited to this, and may be performed by any existing method.
　図１３Ｄは、第二の処理部８２３が、ラベル画像１００２とエッジ強調処理画像１３０３を用いて検出した網膜層の境界を示す境界画像１３０５を示す。境界画像１３０５においては、黒線１３０６が境界の例を示す。第二の処理部８２３が、ラベル画像１００２とエッジ強調処理画像１３０３から網膜層の境界を検出する処理について、以下で説明する。 FIG. 13D shows a boundary image 1305 indicating the boundary of the retinal layer detected by the second processing unit 823 using the label image 1002 and the edge enhancement image 1303. In the boundary image 1305, a black line 1306 shows an example of the boundary. The process in which the second processing unit 823 detects the boundary of the retinal layer from the label image 1002 and the edge enhanced image 1303 will be described below.
　第二の処理部８２３は、エッジ強調処理画像１３０３からエッジ強調された境界を検出する。本実施例では、第一の処理部８２２が既にＩＬＭとＮＦＬとの境界とＲＰＥについて検出しているので、第二の処理部８２３は、続けて、ＩＳＯＳ、ＮＦＬとＧＣＬ境界を検出する。なお、図示しないが、その他の境界として、外網状層（ＯＰＬ）と外顆粒層（ＯＮＬ）との境界、内網状層（ＩＰＬ）と内顆粒層（ＩＮＬ）との境界、ＩＮＬとＯＰＬとの境界、ＧＣＬとＩＰＬとの境界等を検出してもよい。 The second processing unit 823 detects the edge-enhanced boundary from the edge-enhanced image 1303. In this embodiment, since the first processing unit 822 has already detected the boundary between the ILM and the NFL and the RPE, the second processing unit 823 subsequently detects the boundary between the ISSO, the NFL, and the GCL. Although not shown, other boundaries include a boundary between the outer plexiform layer (OPL) and the outer granular layer (ONL), a boundary between the inner plexiform layer (IPL) and the inner granular layer (INL), and a boundary between the INL and OPL. A boundary, a boundary between GCL and IPL, or the like may be detected.
　境界の検出方法としては、各Ａスキャンにおいてエッジ強度が強い箇所を境界候補として複数検出し、隣接するＡスキャンにおいて境界候補同士の連続性を基に、点（エッジ強度が強い箇所）を線としてつなげる処理を行う。また、第二の処理部８２３は、点を線としてつなげた場合に、線の滑らかさを評価することで、外れ値を除去することができる。より具体的には、例えば、つなげた点同士のＺ方向の位置を比較し、所定の閾値よりもＺ方向の位置の差が大きい場合には、新しくつなげられた点を外れ値として判断し、つなげる処理から除外することができる。また、外れ値を除去した場合、除外した点のＡスキャン位置に隣接するＡスキャンにおける境界候補を線としてつなげてもよい。なお、外れ値の除去方法はこれに限られず、既存の任意の方法によって行われてよい。 As a method for detecting a boundary, a plurality of points having high edge strength in each A scan are detected as boundary candidates, and a point (a point having high edge intensity) is converted into a line based on continuity between boundary candidates in adjacent A scans. Perform connection processing. Further, when the points are connected as a line, the second processing unit 823 can remove outliers by evaluating the smoothness of the line. More specifically, for example, the positions of the connected points in the Z direction are compared, and when the difference between the positions in the Z direction is larger than a predetermined threshold, the newly connected point is determined as an outlier, It can be excluded from the connecting process. When an outlier is removed, a boundary candidate in the A-scan adjacent to the A-scan position of the excluded point may be connected as a line. The method of removing outliers is not limited to this, and may be performed by any existing method.
　第二の処理部８２３は、点をつなげて形成した各線について、網膜層の境界のＺ方向の上下の距離や位置関係に基づいて、対応する境界を決定する。なお、各Ａスキャンにおいて外れ値を除去した結果として検出された境界がない場合には、周囲の境界から補間で求めてもよい。また、周囲の境界からエッジを頼りに水平方向（Ｘ又はＹ方向）に境界候補を探索していき、周囲の境界から探索した境界候補を基にして再度、境界を決定するようにしてもよい。 The second processing unit 823 determines a corresponding boundary of each line formed by connecting points based on the vertical distance and the positional relationship in the Z direction of the boundary of the retinal layer. If there is no boundary detected as a result of removing outliers in each A scan, the boundary may be obtained by interpolation from surrounding boundaries. Alternatively, a boundary candidate may be searched in the horizontal direction (X or Y direction) from the surrounding boundary by relying on the edge, and the boundary may be determined again based on the boundary candidate searched from the surrounding boundary. .
　その後、第二の処理部８２３は、検出した境界に対して、境界の形状を滑らかに補正する処理を実行する。例えば、ＳｎａｋｅｓやＬｅｖｅｌ　Ｓｅｔ法等の動的輪郭モデル等により、画像特徴と形状特徴とを用いて境界の形状を滑らかにしてもよい。また、境界形状の座標値を信号による時系列データとみなして、Ｓａｖｉｔｚｋｙ－Ｇｏｌａｙフィルタや、単純移動平均、加重移動平均、指数移動平均等の平滑化処理で形状を滑らかにしてもよい。 Then, the second processing unit 823 executes a process for smoothly correcting the shape of the detected boundary. For example, the shape of the boundary may be smoothed using an image feature and a shape feature using a dynamic contour model such as Snakes or Level @ Set method. Alternatively, the coordinate values of the boundary shape may be regarded as time-series data based on signals, and the shape may be smoothed by a Savitzky-Golay filter or a smoothing process such as a simple moving average, a weighted moving average, or an exponential moving average.
　このような処理により、第二の処理部８２３は、第一の処理部８２２が検出した網膜領域内の網膜層を検出することができる。なお、前述した第二の処理部８２３による網膜層の検出処理は一例であり、既存の任意のセグメンテーション処理を用いて網膜層を検出することもできる。第二の処理部８２３が網膜層を検出すると、処理はステップＳ３０５に移行する。以降の処理は実施例１と同様であるため説明を省略する。 に よ り Through such processing, the second processing unit 823 can detect a retinal layer in the retinal region detected by the first processing unit 822. Note that the above-described retinal layer detection processing by the second processing unit 823 is an example, and the retinal layer can be detected using any existing segmentation processing. When the second processing unit 823 detects a retinal layer, the process proceeds to step S305. Subsequent processing is the same as in the first embodiment, and a description thereof will not be repeated.
　上記のように、本実施例に係る画像処理装置８０は、取得部２１と、第一の処理部８２２と、第二の処理部８２３とを備える。取得部２１は、被検眼の断層画像を取得する。第一の処理部８２２は、学習済モデルを用いて、断層画像において被検眼の複数の網膜層のうち少なくとも一つの網膜層を検出するための第一の検出処理を実行する。第二の処理部８２３は、学習済モデルを用いずに、断層画像において複数の網膜層のうち少なくとも一つの網膜層を検出するための第二の検出処理を実行する。 As described above, the image processing device 80 according to the present embodiment includes the acquisition unit 21, the first processing unit 822, and the second processing unit 823. The acquisition unit 21 acquires a tomographic image of the subject's eye. The first processing unit 822 executes a first detection process for detecting at least one retinal layer among a plurality of retinal layers of the subject's eye in the tomographic image using the learned model. The second processing unit 823 performs a second detection process for detecting at least one retinal layer among a plurality of retinal layers in the tomographic image without using the learned model.
　具体的には、第二の処理部８２３は、第二の検出処理によって、第一の検出処理により検出された少なくとも一つの網膜層以外の少なくとも一つの網膜層を検出する。特に本実施例では、第一の検出処理は、被検眼の内境界膜と神経線維層との境界から視細胞内節外節接合部、網膜色素上皮層、及びブルッフ膜のいずれかまでの層を検出する処理である。また、第二の検出処理は、第一の検出処理の後に行われ、第一の検出処理により検出した層に含まれる、すなわち検出した層の間の少なくとも一つの網膜層を検出する処理である。 Specifically, the second processing unit 823 detects at least one retinal layer other than the at least one retinal layer detected by the first detection processing by the second detection processing. In particular, in the present embodiment, the first detection processing is performed from the boundary between the inner limiting membrane and the nerve fiber layer of the eye to be examined to the junction between the photoreceptor inner and outer nodes, the retinal pigment epithelium layer, and any of the Bruch's membranes. This is the process of detecting. Further, the second detection process is performed after the first detection process, and is a process for detecting at least one retinal layer included in the layers detected by the first detection process, that is, between the detected layers. .
　本実施例に係る画像処理装置８０では、疾患や部位等によらず境界検出を行うことができる。また、機械学習モデルが出力した領域に対して、画像特徴による境界検出を併用することで境界検出の精度を向上させることができる。さらに、機械学習の過程において、学習時には網膜層かそれ以外の正解データを作成するだけで済むため、学習も効率的に行うことができる。 In the image processing apparatus 80 according to the present embodiment, it is possible to perform the boundary detection regardless of a disease or a part. In addition, the accuracy of boundary detection can be improved by using boundary detection based on image features together with the region output by the machine learning model. Furthermore, in the process of machine learning, at the time of learning, only the retinal layer or other correct data need be created, so that learning can be performed efficiently.
　また、機械学習モデルによって検出する境界が多くなると、出力されるラベル画像や境界画像において誤検出が生じる可能性が増加する場合がある。これに対して、本実施例による機械学習モデルを用いた網膜領域の検出では、検出するべき境界が少ないため、出力されるラベル画像や境界画像における誤検出を抑制することができる。 Also, when the number of boundaries detected by the machine learning model increases, the possibility of erroneous detection occurring in the output label image or boundary image may increase. On the other hand, in the detection of the retinal region using the machine learning model according to the present embodiment, since there are few boundaries to be detected, erroneous detection in the output label image or boundary image can be suppressed.
　なお、本実施例においても、実施例１と同様に、第一の処理部８２２において、複数の機械学習モデルを用いて網膜領域の検出を行うように構成されてもよい。この場合には、網膜領域の検出の精度を向上させることができる。また、追加で学習モデルを増やしていくことも可能であるため、性能が徐々に向上するようなバージョンアップを行うことも期待できる。 In this embodiment, as in the first embodiment, the first processing unit 822 may be configured to detect a retinal region using a plurality of machine learning models. In this case, the accuracy of detecting the retinal region can be improved. Further, since it is possible to increase the number of learning models additionally, it is expected that a version upgrade in which performance is gradually improved can be performed.
（実施例２の変形例）
　上述の実施例２では、ルールベースに基づく第二の処理部８２３による網膜の境界の検出に先立って、前段階として学習済モデルを用いた第一の処理部８２２による網膜領域の検出を行う例を示した。しかしながら、第一の処理部８２２による処理及び第二の処理部８２３による処理の順序はこれに限らない。例えば、第一の処理部８２２による網膜領域の検出に非常に時間がかかる場合、当該網膜領域の検出処理を第二の処理部８２３でルールベースにより、先に実行させることも可能である。 (Modification of Embodiment 2)
In the second embodiment described above, prior to the detection of the retinal boundary by the second processing unit 823 based on the rule base, the first processing unit 822 detects the retinal region using the learned model as a previous step. showed that. However, the order of the processing by the first processing unit 822 and the processing by the second processing unit 823 is not limited to this. For example, when it takes a very long time for the first processing unit 822 to detect a retinal region, the second processing unit 823 may first execute the retinal region detection processing based on a rule base.
　このような場合、第二の処理部８２３は、実施例２で示した第二の処理部８２３による方法と同様の方法を用いてＩＬＭとＮＦＬとの境界と、ＲＰＥ又はＩＳＯＳとを最初に検出する。これは、これら境界が、網膜において輝度値が高い場所であり網膜の浅層部と深層部に位置している境界だからである。ＩＬＭとＮＦＬとの境界と、ＲＰＥ又はＩＳＯＳを検出する場合、他の境界よりも特徴が出やすいため、ノイズ処理を何度か行ったボケの大きな画像に基づいてこれら境界を検出してもよい。この場合には、大局的な特徴だけを検出することができるので、その他の境界の誤検出を防ぐことができる。また、断層画像に対して、動的閾値による二値化処理を行って網膜領域を限定し、その中からＩＬＭとＮＦＬとの境界と、ＲＰＥ又はＩＳＯＳとを検出するようにしてもよい。なお、第二の処理部８２３は、ＩＬＭとＮＦＬとの境界と、ＢＭとを検出してもよい。 In such a case, the second processing unit 823 first detects the boundary between the ILM and the NFL and the RPE or ISSO using the same method as the method by the second processing unit 823 described in the second embodiment. I do. This is because these boundaries are places where the luminance value is high in the retina, and are boundaries located at the shallow layer and the deep layer of the retina. When detecting the boundary between the ILM and the NFL and the RPE or the ISOS, since the characteristics are more likely to appear than the other boundaries, these boundaries may be detected based on a large blurred image subjected to noise processing several times. . In this case, since only global features can be detected, erroneous detection of other boundaries can be prevented. In addition, the tomographic image may be binarized by a dynamic threshold to limit the retinal region, and the boundary between the ILM and the NFL and the RPE or ISSO may be detected from the retinal region. Note that the second processing unit 823 may detect the boundary between the ILM and the NFL and the BM.
　ただし、このようなルールベースで第一の検出対象である網膜領域を検出する場合、前述のように、被検眼の個体差や病変に対するロバスト性が低く初めに網膜領域を誤検出してしまうことがある。この場合には、その後の網膜内層境界の検出が適切に行えないことがある。 However, when detecting the retinal region that is the first detection target based on such a rule base, as described above, robustness to individual differences and lesions of the eye to be examined is low, and the retinal region is erroneously detected first. There is. In this case, the subsequent detection of the retinal inner layer boundary may not be performed properly.
　その対策として本変形例では、画像処理部８２が、網膜領域の誤検出をチェックするパラメータ、例えば、網膜領域境界の不連続性や局所曲率又は局所領域における境界座標の分散等を所定の閾値と比較する。これらパラメータが所定の閾値を超える場合には、画像処理部８２が、第二の処理部８２３における網膜領域の検出が誤検出であると判断する。そして、画像処理部８２により、第二の処理部８２３による網膜領域の検出が誤検出と判断された場合に、第一の処理部８２２が網膜領域の検出を行うように構成される。 As a countermeasure, in this modification, the image processing unit 82 sets a parameter for checking erroneous detection of the retinal region, such as discontinuity of the retinal region boundary, local curvature, or dispersion of boundary coordinates in the local region, with a predetermined threshold. Compare. If these parameters exceed a predetermined threshold, the image processing unit 82 determines that the detection of the retinal region in the second processing unit 823 is an erroneous detection. When the image processing unit 82 determines that the detection of the retinal region by the second processing unit 823 is erroneous detection, the first processing unit 822 is configured to detect the retinal region.
　本変形例に従えば、第一の処理部８２２による網膜領域の検出に非常に時間がかかる場合であっても、多数の検査を行う検者（操作者）の実質的な処理待ち時間を軽減しつつ被検眼の個体差や病変に対するロバスト性を確保することができる。 According to this modification, even if it takes a very long time for the first processing unit 822 to detect the retinal region, the substantial processing wait time of the examiner (operator) who performs a large number of examinations is reduced. In addition, robustness to individual differences and lesions of the eye to be examined can be ensured.
　また、本変形例では、第二の処理部８２３の処理を第一の処理部８２２の処理より先に行う構成について述べたが、これらの処理は同時に開始されてもよい。この場合、画像処理部８２により、第二の処理部８２３による網膜領域の検出が誤検出と判断された場合に、第一の処理部８２２による網膜領域の検出を待って、第二の処理部８２３が網膜内層の境界検出を行う。なお、第二の処理部８２３による網膜領域の検出が適切に行われた場合には、第一の処理部８２２による処理を中断したり、第一の処理部８２２による処理結果を破棄したりすることができる。 Also, in the present modification, the configuration in which the processing of the second processing unit 823 is performed prior to the processing of the first processing unit 822 has been described, but these processings may be started simultaneously. In this case, if the image processing unit 82 determines that the detection of the retinal region by the second processing unit 823 is erroneous detection, the image processing unit 82 waits for the detection of the retinal region by the first processing unit 822 and 823 performs boundary detection of the inner layer of the retina. If the detection of the retinal region by the second processing unit 823 is performed appropriately, the processing by the first processing unit 822 is interrupted, or the processing result by the first processing unit 822 is discarded. be able to.
　また、第一の処理部８２２及び第二の処理部８２３が網膜領域（同一の網膜層）を検出する場合、表示制御部２５が、第一の処理部８２２及び第二の処理部８２３による検出処理の処理結果を表示部５０に表示させてもよい。この場合には、表示部５０に表示された処理結果に対する操作者の指示に応じて、第一の処理部８２２及び第二の処理部８２３による検出処理の処理結果のいずれかについて第二の処理部８２３が網膜内層の境界検出を行ってもよい。この場合、第二の処理部８２３による網膜領域の検出処理を第二の検出処理、第二の処理部８２３によるその後の網膜内層の境界の検出処理を第三の検出処理として定義してもよい。 When the first processing unit 822 and the second processing unit 823 detect a retinal region (the same retinal layer), the display control unit 25 performs the detection by the first processing unit 822 and the second processing unit 823. The processing result of the processing may be displayed on the display unit 50. In this case, in response to the operator's instruction for the processing result displayed on the display unit 50, the second processing is performed on one of the processing results of the detection processing by the first processing unit 822 and the second processing unit 823. The unit 823 may detect the boundary of the inner layer of the retina. In this case, the detection processing of the retinal region by the second processing unit 823 may be defined as a second detection processing, and the subsequent processing of detecting the boundary of the retinal inner layer by the second processing unit 823 may be defined as a third detection processing. .
（実施例３）
　実施例２においては、学習済モデルを用いて網膜領域を検出し、検出された網膜領域に対して網膜内層の境界を検出する例について示した。これに対して、本実施例では、学習済モデルを用いて検出する領域として、網膜領域に限らず、入力データである画像の撮影部位に関して特徴的な領域を検出する。 (Example 3)
In the second embodiment, an example has been described in which the retinal region is detected using the learned model, and the boundary of the retinal inner layer is detected with respect to the detected retinal region. On the other hand, in the present embodiment, as a region to be detected using the learned model, not only the retinal region but also a characteristic region with respect to the imaged portion of the image as input data is detected.
　以下、本実施例に係る画像処理システムによる画像処理について、実施例２による画像処理との違いを中心として説明する。なお、本実施例に係る画像処理システムの構成及び処理手順は、実施例２に係る画像処理システム８の構成及び処理手順と同様であるため、同一の参照符号を用いて示し、説明を省略する。 Hereinafter, the image processing by the image processing system according to the present embodiment will be described focusing on the difference from the image processing according to the second embodiment. Note that the configuration and the processing procedure of the image processing system according to the present embodiment are the same as the configuration and the processing procedure of the image processing system 8 according to the second embodiment. .
　本実施例に係る第一の処理部８２２が学習済モデルを用いて検出する領域に関して、図１４Ａ乃至図１４Ｄを参照して説明する。図１４Ａ乃至図１４Ｄでは、被検眼の各部位を撮影した画像と第一の処理部８２２が処理した処理結果のラベル画像の例を示す。 {Regions detected by the first processing unit 822 according to the present embodiment using the learned model will be described with reference to FIGS. 14A to 14D. 14A to 14D show an example of an image of each part of the eye to be inspected and a label image of a processing result processed by the first processing unit 822.
　図１４Ａは、黄斑部を撮影した場合の断層画像１４０１及び学習済モデルを用いて得た黄斑部におけるラベル画像１４０２を示す。ラベル画像１４０２には、硝子体ラベル１４０３、ＩＬＭからＩＳＯＳの範囲のラベル１４０４、ＩＳＯＳからＲＰＥの範囲のラベル１４０５、脈絡膜ラベル１４０６、及び強膜ラベル１４０７が示されている。 FIG. 14A shows a tomographic image 1401 when the macula is photographed, and a label image 1402 in the macula obtained using the trained model. The label image 1402 shows a vitreous label 1403, a label 1404 ranging from ILM to ISSO, a label 1405 ranging from ISSO to RPE, a choroid label 1406, and a sclera label 1407.
　黄斑部に関しては、例えば、出血や新生血管発生などによる網膜全体の厚み、視力に関する視細胞の欠損、又は病的近視による脈絡膜の菲薄化等を捉えるため、形態変化が現れやすい領域毎にラベルを設定して機械学習モデルに事前に学習させる。学習データとしては、黄斑部の断層画像を入力データとし、例えば、硝子体ラベル、ＩＬＭからＩＳＯＳの範囲のラベル、ＩＳＯＳからＲＰＥの範囲のラベル、脈絡膜ラベル、及び強膜ラベルが付されたラベル画像を出力データとする。これにより、第一の処理部８２２は、学習済モデルに黄斑部の断層画像を入力することで、上述した形態変化が現れやすい領域毎にラベルを示すラベル画像を取得し、それらラベルの単位で領域を検出することができる。 For the macula, for example, to capture the thickness of the entire retina due to bleeding or neovascularization, loss of visual cells related to visual acuity, or thinning of the choroid due to pathological myopia, labels should be placed for each area where morphological changes are likely Set and let the machine learning model learn in advance. As learning data, a tomographic image of the macula is used as input data, and, for example, a label image with a vitreous label, a label ranging from ILM to ISSO, a label ranging from ISSO to RPE, a choroid label, and a sclera label Is output data. Accordingly, the first processing unit 822 inputs a tomographic image of the macula into the learned model, thereby acquiring a label image indicating a label for each region where the above-described morphological change is likely to appear, and in units of those labels. The area can be detected.
　図１４Ｂは、視神経乳頭部を撮影した場合の断層画像１４１１及び学習済モデルを用いて得た視神経乳頭部におけるラベル画像１４１２を示す。ラベル画像１４１２には、硝子体ラベル１４１３、ＩＬＭからＮＦＬとＧＣＬ境界の範囲のラベル１４１４、及びＮＦＬとＧＣＬの境界からＧＣＬとＩＰＬの境界の範囲のラベル１４１５が示されている。さらに、ラベル画像１４１２には、ＧＣＬとＩＰＬの境界からＩＳＯＳの範囲のラベル１４１６、及びＩＳＯＳからＲＰＥの範囲のラベル１４１７が示されている。また更に、ラベル画像１４１２には、ＲＰＥから深層の範囲のラベル１４１８、及び篩状板の範囲のラベル１４１９が示されている。この場合の学習データとしては、視神経乳頭部の断層画像を入力データとし、例えば、硝子体ラベル、ＩＬＭからＮＦＬとＧＣＬ境界の範囲のラベル、ＮＦＬとＧＣＬの境界からＧＣＬとＩＰＬの境界の範囲のラベル、ＧＣＬとＩＰＬの境界からＩＳＯＳの範囲のラベル、ＩＳＯＳからＲＰＥの範囲のラベル、ＲＰＥから深層の範囲のラベル、及び篩状板の範囲のラベルが付されたラベル画像を出力データとする。 FIG. 14B shows a tomographic image 1411 when the optic papilla is imaged and a label image 1412 in the optic papilla obtained using the trained model. The label image 1412 shows a vitreous label 1413, a label 1414 in the range from the ILM to the boundary between NFL and GCL, and a label 1415 in a range from the boundary between NFL and GCL to the boundary between GCL and IPL. Further, in the label image 1412, a label 1416 in a range from the boundary between the GCL and the IPL to the ISOS and a label 1417 in a range from the IOS to the RPE are shown. Still further, the label image 1412 shows a label 1418 in the range from the RPE to the deep layer and a label 1419 in the range of the sieve plate. As the learning data in this case, a tomographic image of the optic papilla is used as input data, for example, a vitreous label, a label in the range from the ILM to the NFL and the GCL boundary, and a range from the boundary between the NFL and the GCL to the boundary between the GCL and the IPL. Label images to which labels, labels ranging from ICL to IPL from the boundary of GCL and IPL, labels ranging from ISSO to RPE, labels ranging from RPE to deep layers, and labels ranging from cribrosa are used as output data.
　図１４Ｃの正面画像１４２１は、視神経乳頭部撮影時において、ＸＹ面内の正面方向から見た画像であり、眼底画像撮影装置３０を用いて撮影された画像である。また、図１４Ｃのラベル画像１４２２は、視神経乳頭部の正面画像１４２１に対して学習済モデルを用いて得たラベル画像である。ラベル画像１４２２には、視神経乳頭の周辺部のラベル１４２３、Ｄｉｓｃラベル１４２４、及びＣｕｐラベル１４２５が示されている。 The front image 1421 in FIG. 14C is an image viewed from the front in the XY plane at the time of imaging the optic papilla, and is an image captured using the fundus image capturing apparatus 30. The label image 1422 in FIG. 14C is a label image obtained by using the learned model for the front image 1421 of the optic papilla. The label image 1422 shows a label 1423, a Disc label 1424, and a Cup label 1425 on the periphery of the optic disc.
　視神経乳頭部においては、緑内障によって、神経節細胞の消失や、ＲＰＥの端部（ＲＰＥ－ｔｉｐ）又はブルッフ膜開口端（ＢＭＯ）、篩状板、及びＣｕｐとＤｉｓｃ等の形態変化が現れやすい。そのため、これらの領域毎にラベルを設定して機械学習モデルに事前に学習させる。学習データとしては、視神経乳頭部の正面画像を入力データとし、例えば、視神経乳頭の周辺部のラベル、Ｄｉｓｃラベル、及びＣｕｐラベルが付されたラベル画像を出力データとする。これにより、第一の処理部８２２は、学習済モデルに視神経乳頭部の画像を入力することで、上述した形態変化が現れやすい領域毎にラベルを示すラベル画像を取得し、それらラベルの単位で領域を検出することができる。 In the optic disc, glaucoma tends to cause ganglion cell loss and morphological changes such as the end of the RPE (RPE-tip) or the open end of the Bruch's membrane (BMO), the lamina cribrosa, and the Cup and Disc. Therefore, a label is set for each of these areas, and the machine learning model is trained in advance. As learning data, a front image of the optic papilla is used as input data, and, for example, a label image with a peripheral label, a Disc label, and a Cup label attached to the optic papilla is used as output data. Thereby, the first processing unit 822 obtains a label image indicating a label for each region where the above-mentioned morphological change is likely to appear by inputting an image of the optic disc to the learned model, and in units of those labels. The area can be detected.
　図１４Ｄは、前眼部を撮影した場合の断層画像１４３１及び学習済モデルを用いて得た前眼部撮影におけるラベル画像１４３２が示されている。ラベル画像１４３２には、角膜ラベル１４３３、前房ラベル１４３４、虹彩ラベル１４３５、及び水晶体ラベル１４３６が示されている。前眼部の断層画像１４３１に関しては、後眼部画像とは異なり、前述したような主要な領域を機械学習モデルに事前に学習させる。学習データとしては、前眼部の断層画像を入力データとし、例えば、角膜ラベル、前房ラベル、虹彩ラベル、及び水晶体ラベルが付されたラベル画像を出力データとする。これにより、第一の処理部８２２は、学習済モデルに前眼部の画像を入力することで、上述した形態変化が現れやすい領域毎にラベルを示すラベル画像を取得し、それらラベルの単位で領域を検出することができる。 FIG. 14D shows a tomographic image 1431 when the anterior segment is photographed and a label image 1432 in the anterior segment photographing obtained using the trained model. The label image 1432 shows a corneal label 1433, an anterior chamber label 1434, an iris label 1435, and a lens label 1436. Regarding the tomographic image 1431 of the anterior eye, unlike the posterior eye image, the main region as described above is learned in advance by the machine learning model. As the learning data, a tomographic image of the anterior segment is used as input data, and for example, a label image to which a corneal label, an anterior chamber label, an iris label, and a lens label are attached is used as output data. Thereby, the first processing unit 822 obtains a label image indicating a label for each region where the above-mentioned morphological change is likely to appear by inputting the image of the anterior segment into the learned model, and in units of those labels. The area can be detected.
　本実施例に係る第二の処理部８２３は、図１４Ａ、図１４Ｂ、及び図１４Ｄに示す断層画像１４０１，１４１１，１４３１において、第一の処理部８２２が検出した領域に基づいて残りの境界を検出する。また、第二の処理部８２３は、検出した境界や境界に挟まれた層領域、又は第一の処理部８２２が検出した領域の厚さの計測を実施してもよい。 The second processing unit 823 according to the present embodiment determines the remaining boundary in the tomographic images 1401, 1411, and 1431 shown in FIGS. 14A, 14B, and 14D based on the region detected by the first processing unit 822. To detect. Further, the second processing unit 823 may measure the thickness of the detected boundary or a layer region sandwiched between the boundaries, or the thickness of the region detected by the first processing unit 822.
　また、第二の処理部８２３は、図１４Ｃの正面画像に対してはラベルにより分類された各領域に対して計測を行い各領域の高さ、幅、面積やＣｕｐ/Ｄｉｓｃ比を算出することができる。なお、これらの計測や比率の算出については、既存の任意の方法を用いてよい。 In addition, the second processing unit 823 performs measurement on each area classified by the label with respect to the front image of FIG. 14C and calculates the height, width, area, and Cup / Disc ratio of each area. Can be. In addition, about these measurement and calculation of a ratio, you may use the existing arbitrary methods.
　このように本実施例に係る第二の処理部８２３は、第一の処理部８２２が検出した領域に対応する画像処理アルゴリズムを実行するとともに、領域毎に画像処理アルゴリズムの適用する際のルールを変更することができる。ここでいうルールとは、例えば、検出すべき境界の種類等を含む。例えば、第二の処理部８２３は、図１４Ａに示す黄斑部の断層画像１４０１について、ラベル１４０４で示されるＩＬＭからＩＳＯＳの範囲において実施例１と同様に、追加の境界検出を行うようにしてもよい。なお、境界検出方法は実施例１における境界検出方法と同様であってよい。また、第二の処理部８２３は、黄斑部の断層画像１４０１について、ＩＬＭとＮＦＬの境界、ＯＰＬとＯＮＬの境界、ＩＰＬとＩＮＬの境界、ＩＮＬとＯＰＬの境界、ＧＣＬとＩＰＬの境界を検出するように画像処理アルゴリズム及びルールを適用してもよい。このように、第二の処理部８２３では、領域毎に、所望の構成に応じて任意の画像処理アルゴリズム及びルールが設定されてよい。 As described above, the second processing unit 823 according to the present embodiment executes the image processing algorithm corresponding to the region detected by the first processing unit 822, and sets a rule when the image processing algorithm is applied to each region. Can be changed. Here, the rule includes, for example, the type of boundary to be detected. For example, the second processing unit 823 may perform additional boundary detection on the tomographic image 1401 of the macula shown in FIG. 14A in the range from the ILM to the ISSO indicated by the label 1404, as in the first embodiment. Good. Note that the boundary detection method may be the same as the boundary detection method in the first embodiment. Further, the second processing unit 823 detects a boundary between the ILM and the NFL, a boundary between the OPL and the ONL, a boundary between the IPL and the INL, a boundary between the INL and the OPL, and a boundary between the GCL and the IPL for the tomographic image 1401 of the macula. The image processing algorithm and rules may be applied as described above. Thus, in the second processing unit 823, an arbitrary image processing algorithm and rule may be set for each region according to a desired configuration.
　上記のように、本実施例に係る画像処理装置８０では、第一の処理部８２２は、入力された画像について撮影部位毎に予め定められた境界を検出する。このため、本実施例では、学習済モデルを用いて検出する領域は網膜領域に限らず、撮影部位に関して特徴的な領域であるため、疾患などのバリエーションなどにも対応できる。なお、第一の処理部８２２は、入力された画像が網膜の断層画像である場合には、網膜層の少なくとも一つを検出する第一の検出処理として当該撮影部位毎に予め定められた境界を検出することができる。また、第一の処理部８２２は、入力された画像が網膜の断層画像以外の画像である場合には、第一の検出処理とは異なる処理として当該撮影部位毎に予め定められた境界を検出することができる。なお、実施例１と同様に、断層画像の領域毎に学習を行った複数の学習モデルのそれぞれの出力を合わせて第一の処理部８２２の最終的な出力を生成してもよい。 As described above, in the image processing device 80 according to the present embodiment, the first processing unit 822 detects a predetermined boundary of the input image for each imaging region. For this reason, in the present embodiment, the region detected using the learned model is not limited to the retinal region, but is a characteristic region with respect to the imaged region, and thus can cope with variations such as diseases. Note that when the input image is a retinal tomographic image, the first processing unit 822 performs a predetermined boundary for each imaging region as first detection processing for detecting at least one of the retinal layers. Can be detected. When the input image is an image other than the retinal tomographic image, the first processing unit 822 detects a predetermined boundary for each imaging part as processing different from the first detection processing. can do. Note that, as in the first embodiment, the final output of the first processing unit 822 may be generated by combining the outputs of a plurality of learning models that have learned for each region of the tomographic image.
　また、本実施例に係る画像処理装置８０では、第一の検出処理又は第二の検出処理の結果に基づいて、被検眼に関する各領域の高さ（厚さ）、幅、面積やＣｕｐ/Ｄｉｓｃ比等の所定の形状特徴が計測されることができる。 Further, in the image processing apparatus 80 according to the present embodiment, the height (thickness), width, area, Cup / Disc, and the like of each region relating to the subject's eye are determined based on the result of the first detection process or the second detection process. Certain shape features, such as ratios, can be measured.
（実施例４）
　実施例３においては、学習済モデルを用いて検出する領域を網膜領域に限らず、撮影される部位における特徴的な領域を検出する例について示した。これに対して、実施例４では、画像が撮影された撮影条件に応じて、学習済モデルを用いた処理の実行の選択、さらには学習済モデルを用いて検出する領域の絞込みを行う。 (Example 4)
In the third embodiment, an example has been described in which the region detected using the learned model is not limited to the retinal region, and a characteristic region in a region to be imaged is detected. On the other hand, in the fourth embodiment, execution of processing using the learned model is selected according to the imaging conditions under which the image is captured, and further, the area to be detected is narrowed down using the learned model.
　以下、図１５乃至図１６Ｂを参照して、本実施例に係る画像処理システム１５０による画像処理について、実施例２に係る画像処理との違いを中心として説明する。なお、実施例２に係る画像処理システム８の構成及び処理と同様である本実施例による画像処理システムの構成及び処理については、同一の参照符号を用いて示し、説明を省略する。 Hereinafter, the image processing by the image processing system 150 according to the present embodiment will be described with reference to FIGS. 15 to 16B, focusing on the differences from the image processing according to the second embodiment. Note that the configuration and processing of the image processing system according to the present embodiment that are the same as the configuration and processing of the image processing system 8 according to the second embodiment are denoted by the same reference numerals, and description thereof is omitted.
　図１５は、本実施例に係る画像処理システム１５０の概略的な構成の一例を示す。本実施例に係る画像処理システム１５０では、画像処理装置１５２の画像処理部１５２０において、断層画像生成部２２１、第一の処理部８２２、及び第二の処理部８２３に加えて、選択部１５２４が設けられている。 FIG. 15 shows an example of a schematic configuration of an image processing system 150 according to the present embodiment. In the image processing system 150 according to the present embodiment, in the image processing unit 1520 of the image processing device 152, in addition to the tomographic image generation unit 221, the first processing unit 822, and the second processing unit 823, the selection unit 1524 Is provided.
　選択部１５２４は、取得部２１が取得した撮影条件群及び第一の処理部８２２の学習済モデルに関する学習内容（教師データ）に基づいて、断層画像に対して行う画像処理を選択する。具体的には、撮影条件群と学習済モデルの学習に基づいて、第一の処理部８２２のみで網膜層を検出するか、第一の処理部８２２で網膜領域を検出し、第二の処理部８２３で網膜層を検出するか、又は第二の処理部８２３のみで網膜層を検出するかを選択する。また、選択部１５２４は、第一の処理部８２２が複数の学習済モデルを有する場合に、撮影条件群及び第一の処理部８２２の学習済モデルに関する学習内容に基づいて、第一の処理部８２２による検出処理に用いる学習済モデルを選択することができる。 The selection unit 1524 selects image processing to be performed on a tomographic image based on the imaging condition group acquired by the acquisition unit 21 and the learning content (teacher data) on the learned model of the first processing unit 822. Specifically, based on the imaging condition group and the learning of the learned model, the first processing unit 822 detects only the retinal layer, or the first processing unit 822 detects the retinal region and performs the second processing. Whether the retinal layer is detected by the unit 823 or the retinal layer is detected only by the second processing unit 823 is selected. Further, when the first processing unit 822 has a plurality of learned models, the selection unit 1524 determines whether the first processing unit 822 has the first processing unit based on the image capturing condition group and the learning content of the first processing unit 822 regarding the learned model. It is possible to select a learned model to be used for the detection process by the 822.
　次に、図１６Ａ及び図１６Ｂを参照して、本実施例に係る一連の処理について説明する。図１６Ａは、本実施例に係る一連の処理のフローチャートであり、図１６Ｂは、本実施例に係る境界検出処理のフローチャートである。なお、境界検出処理以外の処理については実施例２の処理と同様であるため、説明を省略する。ステップＳ３０３において断層画像が生成されると、処理はステップＳ１６０４に移行する。処理がステップＳ１６０４に移行すると境界検出処理が開始され、処理はステップＳ１６４１に移行する。 Next, a series of processes according to the present embodiment will be described with reference to FIGS. 16A and 16B. FIG. 16A is a flowchart of a series of processes according to the present embodiment, and FIG. 16B is a flowchart of a boundary detection process according to the present embodiment. Note that the processing other than the boundary detection processing is the same as the processing of the second embodiment, and a description thereof will not be repeated. When a tomographic image is generated in step S303, the process proceeds to step S1604. When the process proceeds to step S1604, the boundary detection process starts, and the process proceeds to step S1641.
　ステップＳ１６４１では、取得部２１が生成された断層画像についての撮影条件群を取得し、画像処理部１５２０が取得された撮影条件群から、第一の処理部８２２及び第二の処理部８２３による処理の選択を実行するのに必要な情報を取得する。例えば、これらの条件は、撮影部位、撮影方式、撮影領域、撮影画角、及び画像の解像度等を含むことができる。 In step S1641, the acquisition unit 21 acquires a group of imaging conditions for the generated tomographic image, and the image processing unit 1520 performs processing by the first processing unit 822 and the second processing unit 823 from the acquired group of imaging conditions. Get the information you need to make a selection. For example, these conditions can include an imaging region, an imaging method, an imaging region, an imaging angle of view, an image resolution, and the like.
　ステップＳ１６４２では、選択部１５２４が、ステップＳ１６４１で取得された撮影条件に基づいて、第一の処理部８２２による処理を実行するか否かの選択を行う。ここでは例として、第一の処理部８２２の学習済モデルについて、視神経乳頭部及び黄斑部のそれぞれ撮影部位毎の教師データを用いて学習が行われた視神経乳頭部用モデルと黄斑部用モデルの２つのみ用意されている場合を考える。また、この場合において、第一の処理部８２２は、広画角な画像（視神経乳頭部と黄斑部の両方が撮影されているような範囲の画像）に対応できないものとして説明する。 In step S1642, the selection unit 1524 selects whether or not to execute the processing by the first processing unit 822 based on the shooting conditions acquired in step S1641. Here, as an example, for the trained model of the first processing unit 822, a model for the optic disc and a model for the macula that have been trained using the teacher data for each imaging site of the optic disc and the macula. Consider the case where only two are prepared. Further, in this case, the first processing unit 822 will be described as not being able to handle a wide-angle image (an image in a range where both the optic papilla and the macula are photographed).
　当該例では、選択部１５２４が、撮影条件のうち、例えば撮影部位名と撮影画角の情報に基づいて、入力画像が視神経乳頭部や黄斑部を単独で撮影した画像と判定した場合には、選択部１５２４は第一の処理部８２２による処理を実行することを選択する。これにより、処理はステップＳ１６４３に移行する。一方で、選択部１５２４が、入力画像が視神経乳頭部や黄斑部以外を撮影した画像、又は視神経乳頭部及び黄斑部の両方を含む広画角な画像であると判定した場合には、選択部１５２４は第一の処理部８２２による処理を実行しないことを選択する。これにより、処理はステップＳ１６４５に移行する。 In this example, when the selecting unit 1524 determines that the input image is an image of the optic papilla or the macula alone, based on, for example, information on the imaging region name and the imaging angle of view in the imaging conditions, The selecting unit 1524 selects to execute the processing by the first processing unit 822. Thus, the process shifts to step S1643. On the other hand, when the selecting unit 1524 determines that the input image is an image obtained by capturing an image other than the optic papilla and the macula or a wide-angle image including both the optic papilla and the macula, the selecting unit 1524 Reference numeral 1524 selects that the processing by the first processing unit 822 is not executed. Accordingly, the process proceeds to step S1645.
　ステップＳ１６４３では、選択部１５２４は、ステップＳ１６４１で取得された撮影条件に基づいて、第一の処理部８２２が利用する適切な学習済モデルの選択を行う。上述の例では、選択部１５２４は、例えば撮影部位名と撮影画角の情報に基づいて、入力画像が視神経乳頭部を撮影した画像であると判定した場合には、視神経乳頭部用のモデルを選択する。同様に、選択部１５２４は、入力画像が黄斑部を撮影した画像であると判定した場合には、黄斑部用のモデルを選択する。 In step S1643, the selection unit 1524 selects an appropriate learned model used by the first processing unit 822, based on the imaging conditions acquired in step S1641. In the above example, when the input unit 1524 determines that the input image is an image obtained by imaging the optic papilla, for example, based on the information of the imaging site name and the imaging angle of view, the selection unit 1524 generates the model for the optic papilla. select. Similarly, when the selection unit 1524 determines that the input image is an image of the macula, the selection unit 1524 selects a model for the macula.
　なお、ここでは、学習済モデルが、視神経乳頭部と黄斑部を撮影した画像だけを学習してある場合の例について示したが、学習済モデルの学習内容はこれに限らない。例えば、他の部位について学習した学習済モデルや、視神経乳頭部及び黄斑部を含む広画角な画像を用いて学習が行われた学習済モデルを用いてもよい。 Here, an example in which the learned model has learned only the image obtained by capturing the optic papilla and the macula has been described, but the learning content of the learned model is not limited to this. For example, a learned model that has been learned for other parts or a learned model that has been learned using a wide-angle image including the optic papilla and the macula may be used.
　また、撮影部位ではなく撮影方式に応じた学習済モデルが別々に用意されている場合には、撮影方式に応じて処理の選択や学習済モデルの選択が行われてよい。撮影方式の例としては、ＳＤ－ＯＣＴとＳＳ－ＯＣＴの撮影方式があり、両者の撮影方式の違いにより、画質、撮影範囲、及び深さ方向の深達度等が異なる。そのため、これらの撮影方式の異なる画像に対して、適切な処理の選択や学習済モデルを選択するようにしてもよい。なお、学習時に撮影方式に関係なく一緒に学習してある場合には、撮影方式に応じて処理を変更する必要はない。また、学習済モデルが一つしかない場合には、ステップＳ１６４３における学習モデルの選択は必要ないため、この処理はスキップすることができる。 In the case where a trained model is prepared separately according to the photographing method instead of the photographing part, selection of a process or selection of a learned model may be performed according to the photographing method. Examples of the photographing method include the SD-OCT and SS-OCT photographing methods, and the image quality, the photographing range, and the depth reach in the depth direction are different depending on the difference between the two photographing methods. Therefore, selection of an appropriate process and selection of a learned model may be performed for these images having different photographing methods. If the learning is performed at the same time regardless of the shooting method, it is not necessary to change the processing according to the shooting method. If there is only one trained model, there is no need to select a learning model in step S1643, so this process can be skipped.
　ステップＳ１６４４では、第一の処理部８２２がステップＳ１６４３で選択された学習済モデルを用いて第一の境界検出処理を行う。なお、この処理に関しては、実施例１乃至３で説明したものを用いることができる。例えば、黄斑部用モデルが、黄斑部における各網膜層の画像セグメンテーション処理について学習している場合には、第一の処理部８２２は、第一の検出処理として、実施例１と同様に、検出対象となる全ての境界を検出することができる。また、例えば、視神経乳頭部用モデルが、視神経乳頭部における網膜領域を検出する処理について学習している場合には、第一の処理部８２２は、第一の検出処理として、実施例２と同様に、網膜領域を検出することができる。同様に、黄斑部用モデルが実施例３と同様に、黄斑部の特徴的な領域を検出する処理を学習している場合には、第一の処理部８２２は、第一の検出処理として、実施例３と同様に、特徴的な領域を検出することができる。なお、具体的な検出方法は、実施例１乃至３における検出方法と同様であるため説明を省略する。 In step S1644, the first processing unit 822 performs a first boundary detection process using the learned model selected in step S1643. It should be noted that the processing described in the first to third embodiments can be used for this processing. For example, when the model for the macula has learned about the image segmentation processing of each retinal layer in the macula, the first processing unit 822 performs the first detection processing as in the first embodiment. All boundaries of interest can be detected. Further, for example, when the model for the optic papilla has learned about the process of detecting the retinal region in the optic papilla, the first processing unit 822 performs the same processing as the second embodiment as the first detection process. Then, the retinal region can be detected. Similarly, when the macula model has learned the process of detecting the characteristic region of the macula, as in the third embodiment, the first processing unit 822 performs As in the third embodiment, a characteristic region can be detected. Note that a specific detection method is the same as the detection method in the first to third embodiments, and thus the description is omitted.
　ステップＳ１６４５では、選択部１５２４は、ステップＳ１６４１で取得された撮影条件に基づいて、第二の処理部８２３による処理を実行するか否かを選択する。選択部１５２４が第二の処理部８２３による処理を実行することを選択した場合には、処理はステップＳ１６４６に移行する。一方で、選択部１５２４が第二の処理部８２３による処理を実行しないことを選択した場合には、ステップＳ１６０４の処理が終了し、処理はステップＳ３０５に移行する。 In step S1645, the selection unit 1524 selects whether or not to execute the processing by the second processing unit 823 based on the shooting conditions acquired in step S1641. If the selecting unit 1524 has selected to execute the process by the second processing unit 823, the process proceeds to step S1646. On the other hand, if the selection unit 1524 selects not to execute the processing by the second processing unit 823, the processing of step S1604 ends, and the processing moves to step S305.
　ここで、ステップＳ１６４５における選択部１５２４の選択処理の例について説明する。第二の処理部８２３による処理を実行する場合とは、例えば、実施例２及び３で説明したように、第一の処理部８２２が検出した領域に基づいて、第二の処理部８２３が境界を検出する場合である。 Here, an example of the selection processing of the selection unit 1524 in step S1645 will be described. The case where the processing by the second processing unit 823 is executed means that, for example, as described in the second and third embodiments, the second processing unit 823 generates a boundary based on the region detected by the first processing unit 822. Is detected.
　また、この他に、第一の処理部８２２が未学習の画像が入力された場合にも第二の処理部８２３による処理を実行する。この場合には、ステップＳ１６４２において、第一の処理部８２２による処理をスキップすることが選択されるため、第二の処理部８２３によって、学習済モデルを使用せずにルールベースの画像セグメンテーション処理によって境界検出を行う。 {Circle around (4)} In addition, the first processing unit 822 also executes the processing by the second processing unit 823 when an unlearned image is input. In this case, in step S1642, it is selected to skip the processing by the first processing unit 822. Therefore, the second processing unit 823 performs the rule-based image segmentation processing without using the learned model. Performs boundary detection.
　一方で、第二の処理部８２３による処理を実行しない場合とは、例えば、第一の処理部８２２が学習済モデルを用いて、対象となる境界を全て検出できる場合である。この場合は、第一の処理部８２２のみで処理が完結するため、第二の処理部８２３の処理をスキップすることが可能である。 On the other hand, the case where the processing by the second processing unit 823 is not performed is, for example, the case where the first processing unit 822 can detect all target boundaries using the learned model. In this case, since the processing is completed only by the first processing unit 822, the processing of the second processing unit 823 can be skipped.
　ただし、第一の処理部８２２が学習済モデルを用いて、対象となる境界をすべて検出できる場合であっても、第二の処理部８２３によって、検出された境界に基づいて層厚の計測等を行う場合には、第二の処理部８２３による処理を実行することが選択されてもよい。一方で、検出された境界に基づく層厚の計測等は第二の処理部８２３によって行われる構成に限られず、画像処理部１５２０において行われればよい。そのため、層厚の計測等が行われる場合であっても、第二の処理部８２３による処理の実行が選択されなくてもよい。 However, even if the first processing unit 822 can detect all target boundaries using the learned model, the second processing unit 823 measures the layer thickness based on the detected boundaries. In the case of performing, the execution of the processing by the second processing unit 823 may be selected. On the other hand, the measurement of the layer thickness based on the detected boundary is not limited to the configuration performed by the second processing unit 823, but may be performed by the image processing unit 1520. Therefore, even when the measurement of the layer thickness or the like is performed, the execution of the process by the second processing unit 823 may not be selected.
　ステップＳ１６４６では、選択部１５２４は、ステップＳ１６４１で取得された撮影条件に基づいて、第二の処理部８２３による処理を実行するために必要な画像処理と画像処理を適用する際のルールの選択を行う。例えば、入力画像が、図１４Ａ、図１４Ｂ及び図１４Ｄに示すような断層画像である場合においては、選択部１５２４は、第一の処理部８２２が検出した領域に基づいて残りの境界を検出するための処理とルールを選択する。 In step S1646, the selection unit 1524 performs image processing necessary for performing the processing by the second processing unit 823 and selection of a rule when applying the image processing based on the imaging conditions acquired in step S1641. Do. For example, when the input image is a tomographic image as shown in FIGS. 14A, 14B, and 14D, the selection unit 1524 detects the remaining boundary based on the region detected by the first processing unit 822. Process and rules to be performed.
　具体的には、例えば、入力画像が図１４Ａに示す黄斑部を撮影した断層画像１４０１の場合には、選択部１５２４は、黄斑部が正しく層認識できる画像処理とルールを選択する。また、例えば、図１４Ｂに示す視神経乳頭部を撮影した断層画像１４１１の場合には、選択部１５２４は、ブルッフ膜開口端（ＢＭＯ）、篩状板、Ｃｕｐ、及びＤｉｓｃ等の影響を考慮した視神経乳頭部の例外処理を考慮した画像処理とルールを選択する。さらに、例えば、図１４Ｄに示す前眼部を撮影した場合の断層画像１４３１の場合には、選択部１５２４は、角膜部の更なる層認識を行いうる画像処理とルールを選択する。 Specifically, for example, when the input image is the tomographic image 1401 obtained by photographing the macula shown in FIG. 14A, the selection unit 1524 selects an image process and a rule that enable the macula to be correctly recognized as a layer. Further, for example, in the case of the tomographic image 1411 obtained by photographing the optic papilla illustrated in FIG. Select image processing and rules that take into account exceptional processing of the nipple. Further, for example, in the case of the tomographic image 1431 obtained by photographing the anterior segment shown in FIG. 14D, the selection unit 1524 selects image processing and a rule that can perform further layer recognition of the cornea.
　さらに、ステップＳ１６４７において層境界の検出に加え又は単独で、検出された境界や境界に挟まれた層領域の厚さの計測を実施する場合には、選択部１５２４は、そのような画像計測機能に必要な画像処理を選択することもできる。 Further, in step S1647, in addition to the detection of the layer boundary or the measurement of the thickness of the detected boundary or the layer region sandwiched by the boundary, the selection unit 1524 performs such an image measurement function. Image processing required for the image processing can be selected.
　ステップＳ１６４７では、第二の処理部８２３が境界の検出及び／又は検出された境界や領域に対する計測を行う。なお、実施例２及び３で説明したものと同様に、第一の処理部８２２が検出した領域に基づいて領域内の境界を検出する処理や境界等に対する計測する処理に関しては、説明を省略する。 In step S1647, the second processing unit 823 detects a boundary and / or measures the detected boundary or region. Note that, similarly to the description of the second and third embodiments, the description of the process of detecting the boundary in the region based on the region detected by the first processing unit 822 and the process of measuring the boundary or the like will be omitted. .
　ここでは、第一の処理部８２２の学習済モデルが未学習である画像が入力された場合に、第二の処理部８２３が行う処理の例について説明する。この場合、入力画像から検出されている網膜領域の候補がないため、第二の処理部８２３は、実施例２の変形例で説明したように、ＩＬＭとＮＦＬとの境界と、ＲＰＥ又はＩＳＯＳとを最初に検出する。 Here, an example of processing performed by the second processing unit 823 when an image for which the learned model of the first processing unit 822 has not been learned is input will be described. In this case, since there is no candidate for the retinal region detected from the input image, the second processing unit 823 determines the boundary between the ILM and the NFL and the RPE or ISSO as described in the modification of the second embodiment. Is detected first.
　第二の処理部８２３は、ＩＬＭとＮＦＬとの境界と、ＲＰＥ又はＩＳＯＳとを検出した後は、それら境界の間の領域に基づいて残りの境界を検出する。当該検出処理は、実施例２及び３における検出処理と同様であるため、説明を省略する。第二の処理部８２３がこれらの処理を実行したら、処理はステップＳ３０５に移行する。なお、実施例２の変形例と同様に、第二の処理部８２３は、ＩＬＭとＮＦＬとの境界と、ＢＭとを最初に検出してもよい。また、以降の処理については実施例１乃至３における処理と同様であるため説明を省略する。 {Circle around (2)} After detecting the boundary between the ILM and the NFL and the RPE or ISSO, the second processing unit 823 detects the remaining boundary based on the region between the boundaries. Since the detection processing is the same as the detection processing in the second and third embodiments, the description is omitted. When the second processing unit 823 executes these processes, the process proceeds to step S305. Note that, similarly to the modification of the second embodiment, the second processing unit 823 may first detect the boundary between the ILM and the NFL and the BM. Further, the subsequent processing is the same as the processing in the first to third embodiments, and thus the description is omitted.
　上記のように、本実施例に係る画像処理装置１５２では、取得部２１は、被検眼の断層画像に関する撮影条件を取得する。画像処理装置１５２は、撮影条件に基づいて、処理の選択を行う選択部１５２４を更に備える。選択部１５２４は、撮影条件に基づいて、第一の検出処理と第二の検出処理のうち少なくとも一つを選択する。 As described above, in the image processing device 152 according to the present embodiment, the acquisition unit 21 acquires the imaging conditions for the tomographic image of the subject's eye. The image processing device 152 further includes a selection unit 1524 for selecting a process based on a shooting condition. The selection unit 1524 selects at least one of the first detection processing and the second detection processing based on the imaging conditions.
　このため、本実施例に係る画像処理装置１５２では、撮影条件に基づいて、第一の処理部８２２が実行する学習済モデルによる境界検出の可否、及び第二の処理部８２３が実行する画像特徴による境界検出の処理実行の要否が判断される。これにより、学習済モデルによる境界検出が特定の画像にだけ対応している場合においても入力画像に応じて処理を適切に行うことができる。そのため、学習モデルが様々な画像のパターンに対応していない場合でも、境界検出処理を確実に実行することができる。従って、機械学習で作成されるさまざまな成熟段階にある機械学習モデルの少なくとも一つと、画像特徴抽出の結果をルールベースで判断して網膜層の境界検出を行う画像処理方法を併用することにより、境界検出の精度を向上させることができる。 For this reason, in the image processing device 152 according to the present embodiment, based on the imaging conditions, whether or not the boundary detection can be performed by the learned model executed by the first processing unit 822 and the image feature executed by the second processing unit 823 Is required to execute the boundary detection process. Thus, even when the boundary detection by the learned model corresponds to only a specific image, the processing can be appropriately performed according to the input image. Therefore, even when the learning model does not correspond to various image patterns, the boundary detection processing can be executed reliably. Therefore, by using at least one of the machine learning models at various stages of maturity created by machine learning and the image processing method of judging the result of the image feature extraction based on a rule base and detecting the boundary of the retinal layer, The accuracy of boundary detection can be improved.
　また、第一の処理部８２２は、異なる教師データを用いて機械学習が行われた複数の学習済モデルを含む。さらに、第一の処理部８２２は、複数の学習済モデルのうち、撮影条件に対応する教師データを用いて機械学習が行われた学習済モデルを用いて、第一の検出処理を実行する。 {Circle around (1)} The first processing unit 822 includes a plurality of learned models on which machine learning has been performed using different teacher data. Further, the first processing unit 822 executes the first detection process using a learned model that has been machine-learned using teacher data corresponding to the imaging condition, among the plurality of learned models.
　本実施例によれば、撮影条件に基づいて、適切な学習モデルを用いて網膜層の検出を行うことができ、入力画像に応じてより適切な処理を行うことができる。また、追加で学習モデルを増やしていくことも可能であるため、性能が徐々に向上するようなバージョンアップを行うことも期待できる。さらに、選択部１５２４によれば、撮影条件に基づいて、第二の処理部８２３において用いる画像処理とルールの選択を行うことができ、入力画像に応じてより適切な処理を行うことができる。 According to the present embodiment, the retinal layer can be detected using an appropriate learning model based on the imaging conditions, and more appropriate processing can be performed according to the input image. Further, since it is possible to increase the number of learning models additionally, it is expected that a version upgrade in which performance is gradually improved can be performed. Further, according to the selection unit 1524, the image processing and the rule used in the second processing unit 823 can be selected based on the imaging conditions, and more appropriate processing can be performed according to the input image.
　なお、本実施例では、ステップＳ１６４２及びステップＳ１６４５において、第一の処理部８２２及び第二の処理部８２３による処理の選択を別々に行ったが、処理の選択の手順はこれに限られない。例えば、選択部１５２４が、一ステップで、第一の処理部８２２だけによる処理、第二の処理部８２３だけによる処理、又は第一の処理部８２２及び第二の処理部８２３による処理を選択するように構成されてもよい。 In the present embodiment, in steps S1642 and S1645, the first processing unit 822 and the second processing unit 823 select the processing separately, but the procedure for selecting the processing is not limited to this. For example, the selection unit 1524 selects, in one step, processing by only the first processing unit 822, processing by only the second processing unit 823, or processing by the first processing unit 822 and the second processing unit 823. It may be configured as follows.
（実施例４の変形例）
　実施例４では、第一の処理部８２２と第二の処理部８２３が境界検出等を双方で分担する場合、又はどちらか一方のみで完結できる場合等、さまざまな場合において適切な処理が可能であることを示した。これに対し、第一の処理部８２２と第二の処理部８２３が同一の処理、例えば同一の境界を検出する処理を並行して実行してもよい。 (Modification of Embodiment 4)
In the fourth embodiment, appropriate processing can be performed in various cases, such as when the first processing unit 822 and the second processing unit 823 share the boundary detection and the like, or when only one of them can complete the detection. It was shown. On the other hand, the first processing unit 822 and the second processing unit 823 may execute the same processing, for example, the processing of detecting the same boundary in parallel.
　このような例として、例えば、第一の処理部８２２が学習済モデルを用いて対象となる境界を全て検出でき、且つ、第二の処理部８２３が第一の検出対象である網膜領域を含む対象となる境界を全て検出できる場合を考える。この場合、第一の処理部８２２と第二の処理部８２３はそれぞれの境界を別々に検出した結果を出力する。 As such an example, for example, the first processing unit 822 can detect all target boundaries using the learned model, and the second processing unit 823 includes a retinal region that is the first detection target. Consider a case where all target boundaries can be detected. In this case, the first processing unit 822 and the second processing unit 823 output the results of detecting the respective boundaries separately.
　これらの検出結果は、それぞれ学習済モデルによる処理とルールベースによる画像処理の結果であるため、両結果には差異が生じる場合がある。そのため、本変形例では、表示制御部２５が、この両結果を表示部５０に並べて表示させたり、切り替えて表示させたり、重ねて表示させたりすることができる。また、画像処理部１５２０において、両結果の一致不一致を判定し、表示制御部２５が当該不一致部分を強調して表示部５０に表示させることもできる。この場合には、操作者に層検出の信頼度を示すことができる。さらに、表示制御部２５は、不一致部分を表示部５０に表示させ、操作者の指示に応じて、より納得度の高い結果を選択できるようにしてもよい。 検 出 These detection results are the result of the process by the learned model and the result of the image processing by the rule base. For this reason, in the present modification, the display control unit 25 can display the two results side by side on the display unit 50, display the results by switching, or display them in a superimposed manner. In addition, the image processing unit 1520 can determine whether the two results match or not, and the display control unit 25 can display the mismatched portion on the display unit 50 in an emphasized manner. In this case, the reliability of the layer detection can be shown to the operator. Further, the display control unit 25 may display the mismatched portion on the display unit 50 so that a more satisfactory result can be selected according to the instruction of the operator.
（実施例５）
　実施例２乃至４においては、学習済モデルを用いて網膜領域を検出し、検出された網膜領域に対してルールベースで網膜内層の境界を検出する例について示した。これに対し、実施例５では、学習済モデルを用いて検出した領域に対して、医学的特徴に基づいて補正を行う。 (Example 5)
In the second to fourth embodiments, an example has been described in which a retinal region is detected using a learned model and a boundary of an inner retinal layer is detected based on a rule based on the detected retinal region. On the other hand, in the fifth embodiment, a region detected using the trained model is corrected based on medical characteristics.
　以下、図１７乃至図１９Ｄを参照して、本実施例に係る画像処理システム１７０による画像処理について、実施例２に係る画像処理との違いを中心として説明する。なお、実施例２に係る画像処理システム８の構成及び処理と同様である本実施例による画像処理システムの構成及び処理については、同一の参照符号を用いて示し、説明を省略する。 Hereinafter, image processing performed by the image processing system 170 according to the present embodiment will be described with reference to FIGS. 17 to 19D, focusing on differences from the image processing according to the second embodiment. Note that the configuration and processing of the image processing system according to the present embodiment that are the same as the configuration and processing of the image processing system 8 according to the second embodiment are denoted by the same reference numerals, and description thereof is omitted.
　図１７は、本実施例に係る画像処理システム１７０の概略的な構成の一例を示す。本実施例に係る画像処理システム１７０における画像処理装置１７２の画像処理部１７２０には、断層画像生成部２２１、第一の処理部８２２、及び第二の処理部８２３に加えて、補正部１７２４が設けられている。 FIG. 17 shows an example of a schematic configuration of an image processing system 170 according to the present embodiment. The image processing unit 1720 of the image processing device 172 in the image processing system 170 according to the present embodiment includes a correction unit 1724 in addition to the tomographic image generation unit 221, the first processing unit 822, and the second processing unit 823. Is provided.
　補正部１７２４は、第一の処理部８２２が学習済モデルを用いて得たラベル画像について、眼の医学的特徴に基づいて、ラベル付けされた領域を補正する。これにより、画像処理部１７２０はより適切に網膜領域や特徴領域を検出することができる。 The correction unit 1724 corrects the labeled area of the label image obtained by the first processing unit 822 using the learned model based on the medical characteristics of the eye. Accordingly, the image processing unit 1720 can more appropriately detect the retinal region and the characteristic region.
　次に、図１８Ａ及び図１８Ｂを参照して、本実施例に係る一連の処理について説明する。図１８Ａは、本実施例に係る一連の処理のフローチャートであり、図１８Ｂは、本実施例に係る境界検出のフローチャートである。なお、境界検出処理以外の処理については実施例２の処理と同様であるため、説明を省略する。ステップＳ３０３において断層画像が生成されると、処理はステップＳ１８０４に移行する。処理がステップＳ１８０４に移行し、ステップＳ９４１において第一の処理部８２２による処理が実行されると、処理はステップＳ１８４１に移行する。 Next, a series of processes according to the present embodiment will be described with reference to FIGS. 18A and 18B. FIG. 18A is a flowchart of a series of processing according to the present embodiment, and FIG. 18B is a flowchart of boundary detection according to the present embodiment. Note that the processing other than the boundary detection processing is the same as the processing of the second embodiment, and a description thereof will not be repeated. When a tomographic image is generated in step S303, the process proceeds to step S1804. The process shifts to step S1804, and when the process by the first processing unit 822 is executed in step S941, the process shifts to step S1841.
　ステップＳ１８４１では、補正部１７２４が、ステップＳ９４１において第一の処理部８２２によって検出された網膜領域を補正する。より具体的には、第一の処理部８２２が学習済モデルを用いて得たラベル画像について、眼の医学的特徴に基づいて、ラベル付けされた領域を補正する。 In step S1841, the correction unit 1724 corrects the retinal region detected in step S941 by the first processing unit 822. More specifically, the first processing unit 822 corrects the labeled area of the label image obtained using the learned model based on the medical characteristics of the eye.
　ここで、図１９Ａ乃至図１９Ｄを参照して、本実施例に係る補正部１７２４による補正処理について説明する。図１９Ａは、第一の処理部８２２への入力となる断層画像１９０１の一例を示す。図１９Ｂは、断層画像１９０１を入力として第一の処理部８２２が学習済モデルを用いて得たラベル画像１９０２の一例を示す。ラベル画像１９０２には、網膜内層のラベル１９０４、網膜よりも浅層側（硝子体側）のラベル１９０３、及び網膜よりも深層側（脈絡膜側）のラベル１９０５が示されている。 Here, the correction processing by the correction unit 1724 according to the present embodiment will be described with reference to FIGS. 19A to 19D. FIG. 19A shows an example of a tomographic image 1901 to be input to the first processing unit 822. FIG. 19B illustrates an example of a label image 1902 obtained by the first processing unit 822 using the tomographic image 1901 as an input and using the learned model. The label image 1902 shows a label 1904 on the inner layer of the retina, a label 1903 on the shallower side (vitreous side) than the retina, and a label 1905 on the deeper side (choroid side) than the retina.
　なお、当該ラベル付けについては、学習済モデルの学習時のラベルの設定に基づいている。そのため、ラベルの種類はこれに限られるものではなく、実施例３で示したように網膜層内に複数のラベルを設定してもよい。その場合においても、本実施例に係る補正処理を適用することが可能である。 Note that the labeling is based on the label setting at the time of learning of the trained model. Therefore, the type of label is not limited to this, and a plurality of labels may be set in the retinal layer as shown in the third embodiment. Even in that case, the correction processing according to the present embodiment can be applied.
　本実施例において、第一の処理部８２２は学習済モデルを用いて画素単位で画像セグメンテーション処理を行う。そのため、図１９Ｂのラベル１９０３’，１９０４’に示すように部分的に誤検出してしまうことがある。補正部１７２４は、眼の医学的特徴に基づいてこれらの誤検出を補正する。 In the present embodiment, the first processing unit 822 performs image segmentation processing in pixel units using the learned model. Therefore, as shown by the labels 1903 'and 1904' in FIG. 19B, erroneous detection may occur partially. The correction unit 1724 corrects these erroneous detections based on the medical characteristics of the eye.
　第一の処理部８２２では、検出したラベル毎にラベリング処理を行い、隣接する画素において同じラベルをもつ画素は一つの領域として統合される。本実施例において付与されるラベルの種類は、網膜内層のラベル１９０４、網膜よりも浅層側（硝子体側）のラベル１９０３、及び網膜よりも深層側（脈絡膜側）のラベル１９０５の３種類である。また、網膜の断層画像を撮影した画像を対象としているため、これらラベルが現れる順番は、画像に対して、上からラベル１９０３、ラベル１９０４、ラベル１９０５の順となる。なお、脈絡膜側を強調して撮影するＥＤＩ（Ｅｎｈａｎｃｅｄ　Ｄｅｐｔｈ　Ｉｍａｇｉｎｇ）モードの場合には、網膜が反転して撮影されるため、ラベルが現れる順番は画像の上からラベル１９０５、ラベル１９０４、ラベル１９０３の順となる。 The first processing unit 822 performs a labeling process for each detected label, and pixels having the same label in adjacent pixels are integrated as one region. In this embodiment, three types of labels are provided: a label 1904 on the inner layer of the retina, a label 1903 on the shallower side (vitreous side) than the retina, and a label 1905 on the deeper side (choroid side) than the retina. . Also, since the target is an image obtained by capturing a retinal tomographic image, the order in which these labels appear is, in order from the top, a label 1903, a label 1904, and a label 1905. In the EDI (Enhanced \ Depth \ Imaging) mode in which the image is emphasized on the choroid side, the retina is inverted and the image is taken. Therefore, the order in which the labels appear is from the top of the image to the labels 1905, 1904, and 1903 In order.
　上述したように、第一の処理部８２２に入力する画像は網膜の断層画像であるため、撮影時の条件や撮影部位から医学的特徴に基づいてラベルの位置関係を推定することができる。そこで、補正部１７２４は、ラベリングされた領域毎に検出結果を特定し、誤検出とみなされる領域を、医学的特徴に基づいて推定される領域に補正する。 As described above, since the image input to the first processing unit 822 is a tomographic image of the retina, the positional relationship between labels can be estimated based on medical conditions and imaging conditions. Therefore, the correction unit 1724 specifies the detection result for each of the labeled regions, and corrects the region regarded as erroneous detection to a region estimated based on medical characteristics.
　具体的には、補正部１７２４は、例えば、ラベリングされた領域の面積が大きい方から領域を特定していき、ラベリングされた領域の面積が閾値以下のものや、既に特定された領域から空間的な距離が離れているものは誤検出であると判断する。その後、補正部１７２４は、誤検出であると判断したラベル情報をリセットする。この場合の例を図１９Ｃに示す。図１９Ｃに示される領域１９１０は、補正部１７２４によって、誤検出とみなされた領域である、ラベル１９０３’及びラベル１９０４’で示された領域についてラベル情報がリセットされた領域を示す。 Specifically, for example, the correction unit 1724 specifies the area from the larger area of the labeled area, and determines the area where the area of the labeled area is equal to or smaller than the threshold, or the spatial area from the already specified area. It is determined that an object that is far away is an erroneous detection. After that, the correction unit 1724 resets the label information determined to be an erroneous detection. FIG. 19C shows an example of this case. An area 1910 illustrated in FIG. 19C indicates an area where label information has been reset for the areas indicated by the labels 1903 'and 1904', which are areas that have been determined to be erroneous by the correction unit 1724.
　補正部１７２４は、ラベル情報をリセットした領域１９１０に対して、周辺のラベル情報から推測されるラベル情報を割り当てる。図１９Ｃに示す例では、ラベル１９０３で囲まれた領域１９１０についてラベル１９０３を割り当て、ラベル１９０５で囲まれた領域１９１０についてラベル１９０５を割り当てる。 The correction unit 1724 assigns label information estimated from surrounding label information to the area 1910 in which the label information has been reset. In the example shown in FIG. 19C, a label 1903 is assigned to an area 1910 surrounded by a label 1903, and a label 1905 is assigned to an area 1910 surrounded by a label 1905.
　補正部１７２４によるこれらの処理により、図１９Ｄに示すように、最終的なラベル画像１９２０が出力される。これにより、画像処理部１７２０は、より適切に網膜領域を検出することができる。 By these processes by the に よ る correction unit 1724, a final label image 1920 is output as shown in FIG. 19D. Accordingly, the image processing unit 1720 can more appropriately detect the retinal region.
　補正部１７２４による補正処理が行われると、処理はステップＳ９４２に移行する。ステップＳ９４２では、第二の処理部８２３が補正された網膜領域に基づいて、実施例２と同様に、第二の境界検出処理を行う。以降の処理は、実施例２の処理と同様であるため説明を省略する。 When the correction process is performed by the correction unit 1724, the process proceeds to step S942. In step S942, the second processing unit 823 performs a second boundary detection process based on the corrected retinal region, as in the second embodiment. Subsequent processing is the same as the processing of the second embodiment, and a description thereof will be omitted.
　上記のように、本実施例に係る画像処理装置１７２は、網膜層における医学的特徴に基づいて、第一の処理部８２２が検出した網膜層の構造を補正する補正部１７２４を更に備える。 As described above, the image processing device 172 according to the present embodiment further includes the correction unit 1724 that corrects the structure of the retinal layer detected by the first processing unit 822 based on the medical characteristics of the retinal layer.
　このため、本実施例に係る画像処理装置１７２では、学習済モデルを用いて検出した領域に対して医学的特徴を用いて領域の補正を行うことができる。そのため、画素単位で画像を検出する場合においても、誤検出を低減することができる。 Therefore, in the image processing device 172 according to the present embodiment, the region detected using the learned model can be corrected using the medical features. Therefore, erroneous detection can be reduced even when an image is detected in pixel units.
　なお、本実施例では、実施例２に係る処理に、補正部１７２４による補正処理を加えたが、実施例３や実施例４に係る処理に当該補正処理を加えてもよい。 In the present embodiment, the correction processing by the correction unit 1724 is added to the processing according to the second embodiment. However, the correction processing may be added to the processing according to the third or fourth embodiment.
（実施例６）
　実施例１乃至５においては、撮影した断層画像について、学習済モデルを用いて網膜内層の境界や網膜領域を検出した。これに対し、本実施例では、別の学習済モデルを用いて断層画像の画質を改善した高画質画像を生成し、高画質画像に対して、実施例１や２等に係る学習済モデルを用いた境界検出や領域検出を行う。なお、本実施例における画質の改善とは、ノイズの低減や、撮影対象を観察しやすい色や階調への変換、解像度や空間分解能の向上、及び解像度の低下を抑えた画像サイズの拡大等を含む。 (Example 6)
In the first to fifth embodiments, the boundary of the inner layer of the retina and the retinal region are detected using the learned model in the captured tomographic image. In contrast, in the present embodiment, a high-quality image in which the image quality of a tomographic image is improved is generated using another learned model, and the learned model according to the first or second embodiment is used for the high-quality image. The used boundary detection and area detection are performed. Note that the image quality improvement in the present embodiment includes noise reduction, conversion of a photographing target into colors and gradations that are easy to observe, improvement in resolution and spatial resolution, and enlargement of the image size while suppressing a decrease in resolution including.
　以下、本実施例に係る画像処理システムによる画像処理について、実施例２による画像処理との違いを中心として説明する。なお、本実施例に係る画像処理システムの構成及び処理手順は、実施例２に係る画像処理システム８の構成及び処理手順と同様であるため、同一の参照符号を用いて示し、説明を省略する。 Hereinafter, the image processing by the image processing system according to the present embodiment will be described focusing on the difference from the image processing according to the second embodiment. Note that the configuration and the processing procedure of the image processing system according to the present embodiment are the same as the configuration and the processing procedure of the image processing system 8 according to the second embodiment. .
　本実施例において、第一の処理部８２２は、網膜領域を検出するための機械学習モデルとは別の機械学習モデルである高画質化モデルに関する学習済モデルを用いて入力画像の画質を改善する処理を行う。高画質化モデルは、任意の機械学習アルゴリズムによる機械学習モデルに対して、事前に適切な教師データを用いてトレーニングし、入力画像の画質を改善した画像を出力する学習済モデルである。 In the present embodiment, the first processing unit 822 improves the image quality of the input image using a learned model related to a high image quality model, which is a machine learning model different from a machine learning model for detecting a retinal region. Perform processing. The image quality improvement model is a trained model that trains a machine learning model using an arbitrary machine learning algorithm using appropriate teacher data in advance, and outputs an image with an improved image quality of an input image.
　ここで、本実施例に係る高画質化モデルの教師データの一例を図２０に示す。図２０において、断層画像２００１はＯＣＴの撮影によって取得された断層画像の一例を示し、断層画像２００２は断層画像２００１を高画質化処理した断層画像を示す。断層画像２００１は入力データの一例を示し、断層画像２００２を出力データの一例を示し、これらの画像によって構成されるペア群により教師データが構成されている。 Here, FIG. 20 shows an example of the teacher data of the image quality improvement model according to the present embodiment. In FIG. 20, a tomographic image 2001 shows an example of a tomographic image acquired by OCT imaging, and a tomographic image 2002 shows a tomographic image obtained by performing high quality processing on the tomographic image 2001. A tomographic image 2001 shows an example of input data, a tomographic image 2002 shows an example of output data, and teacher data is composed of a group of pairs composed of these images.
　なお、高画質化処理としては、空間的に同じ位置を複数回撮影した断層画像について位置合わせを行い、それら位置合わせ済みの断層画像を加算平均処理することが挙げられる。なお、高画質化処理は加算平均処理に限られず、例えば、平滑化フィルタを用いた処理や最大事後確率推定処理（ＭＡＰ推定処理）、階調変換処理等であってもよい。また、高画質化処理された画像としては、例えば、ノイズ除去とエッジ強調などのフィルタ処理を行った画像でもよいし、低輝度な画像から高輝度な画像とするようなコントラストが調整された画像を用いてもよい。さらに、高画質化モデルに係る教師データの出力データは、高画質な画像であればよいため、入力データである断層画像を撮影した際のＯＣＴ装置よりも高性能なＯＣＴ装置を用いて撮影された断層画像や、高負荷な設定により撮影された断層画像であってもよい。 高 As the image quality improvement processing, there is a method of performing position alignment on tomographic images obtained by photographing the same position spatially a plurality of times, and performing an averaging process on the aligned tomographic images. Note that the image quality improvement processing is not limited to the averaging processing, and may be, for example, processing using a smoothing filter, maximum a posteriori probability estimation processing (MAP estimation processing), gradation conversion processing, or the like. Further, as the image subjected to the high image quality processing, for example, an image which has been subjected to filter processing such as noise removal and edge emphasis may be used, or an image in which the contrast is adjusted such that a low luminance image is converted to a high luminance image. May be used. Furthermore, since the output data of the teacher data related to the high image quality model only needs to be a high image quality image, the image data is captured using an OCT device having higher performance than the OCT device used when capturing the tomographic image as input data. Tomographic images, or tomographic images taken with a high load setting.
　第一の処理部８２２は、このような教師データを用いてトレーニングされた高画質化モデルに対して、ＯＣＴの撮影によって取得された断層画像を入力し、高画質化された断層画像を取得する。なお、第一の処理部８２２は、ラスタスキャンにより網膜を三次元的にスキャンしたボリュームの断層画像を高画質化モデルに入力することで、高画質化されたボリューム断層画像を取得することができる。 The first processing unit 822 inputs a tomographic image obtained by OCT imaging to the high-quality model trained using such teacher data, and obtains a high-quality tomographic image. . Note that the first processing unit 822 can acquire a high-quality volume tomographic image by inputting a tomographic image of a volume obtained by three-dimensionally scanning the retina by raster scanning into a high-quality model. .
　第一の処理部８２２は、高画質化モデルを用いて取得した高画質画像を入力として、実施例２乃至５と同様に、学習済モデルを用いて網膜領域又は特徴領域を検出する。 The first processing unit 822 receives the high-quality image acquired using the high-quality image model, and detects the retinal region or the characteristic region using the learned model as in the second to fifth embodiments.
　また、第二の処理部８２３は、第一の処理部８２２で取得した高画質画像及び検出した網膜領域や特徴領域に基づいて、網膜層を検出することができる。 {Circle around (2)} The second processing unit 823 can detect the retinal layer based on the high-quality image acquired by the first processing unit 822 and the detected retinal region or characteristic region.
　上記のように、本実施例に係る画像処理装置８０では、第一の処理部８２２は学習済モデルを用いて高画質化された断層画像について、第一の検出処理を行う。 As described above, in the image processing device 80 according to the present embodiment, the first processing unit 822 performs the first detection process on the tomographic image of which the image quality has been improved using the learned model.
　これにより、本実施例に係る画像処理装置８０は、機械学習モデルの学習済モデルを用いて入力画像の画質を改善し、画質を改善した画像に対して網膜層の検出を行うことができる。そのため、ノイズ低減等の画質改善がなされた画像を用いて網膜層の検出を行うことができ、誤検出を低減することができる。 Accordingly, the image processing device 80 according to the present embodiment can improve the image quality of the input image using the learned model of the machine learning model, and can detect the retinal layer in the image with the improved image quality. Therefore, it is possible to detect the retinal layer using an image with improved image quality such as noise reduction, and to reduce erroneous detection.
　なお、本実施例では、実施例２に係る処理に、入力画像である断層画像を高画質化する処理を加えたが、実施例１や実施例３乃至５に係る処理に、当該高画質化の処理を加えてもよい。 Note that, in the present embodiment, a process of increasing the image quality of the tomographic image, which is an input image, is added to the process of the second embodiment. However, the process of the first embodiment and the third to fifth embodiments includes the process of improving the image quality. May be added.
　また、本実施例では、高画質化を行う高画質化モデルを、検出処理を行う機械学習モデルとは別の機械学習モデルとした。しかしながら、検出処理を行う機械学習モデルに当該高画質化処理を学習させ、高画質化と検出処理の両方を行うように機械学習モデルを構成してもよい。 Further, in the present embodiment, the image quality improvement model for improving the image quality is a machine learning model different from the machine learning model for performing the detection processing. However, the machine learning model that performs the detection process may learn the high image quality process, and the machine learning model may be configured to perform both the high image quality process and the detection process.
　なお、本実施例では、第一の処理部８２２が高画質化処理に関する学習済モデル（高画質化モデル）を用いて断層画像の画質を改善した高画質画像を生成した。しかしながら、高画質化モデルを用いて高画質画像を生成する構成要素は第一の処理部８２２に限られない。例えば、第一の処理部８２２とは別の第三の処理部（高画質化部）を設け、第三の処理部が高画質化モデルを用いて高画質画像を生成してもよい。このため、第一の処理部８２２又は当該第三の処理部は、高画質化用の学習済モデルを用いて、断層画像から、当該断層画像と比べて高画質化された断層画像を生成する生成部の一例として機能することができる。なお、第三の処理部や高画質化モデルは、ＣＰＵやＭＰＵ、ＧＰＵ、ＦＰＧＡ等のプロセッサーによって実行されるソフトウェアモジュール等で構成されてもよいし、ＡＳＩＣ等の特定の機能を果たす回路等によって構成されてもよい。 In the present embodiment, the first processing unit 822 generates a high-quality image in which the image quality of the tomographic image is improved using a learned model (high-quality model) related to the high-quality process. However, a component that generates a high-quality image using the high-quality model is not limited to the first processing unit 822. For example, a third processing unit (image quality improving unit) different from the first processing unit 822 may be provided, and the third processing unit may generate a high quality image using the image quality improving model. For this reason, the first processing unit 822 or the third processing unit generates a tomographic image having a higher image quality as compared with the tomographic image from the tomographic image using the learned model for improving the image quality. It can function as an example of a generation unit. The third processing unit and the high image quality model may be configured by a software module or the like executed by a processor such as a CPU, an MPU, a GPU, or an FPGA, or by a circuit or the like that performs a specific function such as an ASIC. It may be configured.
（実施例７）
　次に、図２１Ａ乃至図２３を参照して、実施例７に係る画像処理装置８０について説明する。実施例６では、第一の処理部８２２は、高画質化モデルを用いて高画質化された断層画像について第一の検出処理を行い、網膜領域又は特徴領域を検出した。これに関し、第一の処理部８２２は、他の画像について高画質化モデルを用いて高画質化処理を行ってもよく、表示制御部２５は、高画質化された各種画像を表示部５０に表示させてもよい。例えば、第一の処理部８２２は、第一の検出処理及び第二の検出処理により検出された網膜層の情報（例えば境界画像）に基づいて生成された輝度のＥｎ－Ｆａｃｅ画像やＯＣＴＡ正面画像等を高画質化処理してもよい。また、表示制御部２５は、第一の処理部８２２によって高画質化処理された断層画像、輝度のＥｎ－Ｆａｃｅ画像、及びＯＣＴＡ正面画像の少なくとも１つを表示部５０に表示させることができる。なお、高画質化し表示する画像は、ＳＬＯ画像や、眼底カメラ等で取得された眼底画像、蛍光眼底画像等であってもよい。ここでＳＬＯ画像とは、不図示のＳＬＯ（Ｓｃａｎｎｉｎｇ　Ｌａｓｅｒ　Ｏｐｈｔｈａｌｍｏｓｃｏｐｅ：走査型検眼鏡）光学系によって取得した眼底の正面画像である。 (Example 7)
Next, an image processing apparatus 80 according to a seventh embodiment will be described with reference to FIGS. 21A to 23. In the sixth embodiment, the first processing unit 822 performs the first detection process on the tomographic image of which the image quality has been improved using the image quality improvement model, and has detected the retinal region or the characteristic region. In this regard, the first processing unit 822 may perform an image quality improvement process on another image using an image quality enhancement model, and the display control unit 25 may display the various image quality enhanced images on the display unit 50. It may be displayed. For example, the first processing unit 822 may generate a luminance En-Face image or an OCTA front image generated based on retinal layer information (eg, a boundary image) detected by the first detection processing and the second detection processing. And the like may be subjected to high image quality processing. Further, the display control unit 25 can cause the display unit 50 to display at least one of the tomographic image, the luminance En-Face image, and the OCTA front image that have been subjected to the high image quality processing by the first processing unit 822. The image to be displayed with high image quality may be an SLO image, a fundus image acquired by a fundus camera or the like, a fluorescent fundus image, or the like. Here, the SLO image is a front image of the fundus oculi acquired by an SLO (Scanning Laser Ophthalmoscope) optical system (not shown).
　ここで、各種画像を高画質化処理するための高画質化モデルの学習データは、各種画像について、実施例６に係る高画質化モデルの学習データと同様に、高画質化処理前の画像を入力データとし、高画質化処理後の画像を出力データとする。なお、学習データに関する高画質化処理については、実施例６と同様に、例えば、加算平均処理や、平滑化フィルタを用いた処理、最大事後確率推定処理（ＭＡＰ推定処理）、階調変換処理等であってよい。また、高画質化処理後の画像としては、例えば、ノイズ除去とエッジ強調などのフィルタ処理を行った画像でもよいし、低輝度な画像から高輝度な画像とするようなコントラストが調整された画像を用いてもよい。さらに、高画質化モデルに係る教師データの出力データは、高画質な画像であればよいため、入力データである画像を撮影した際のＯＣＴ装置よりも高性能なＯＣＴ装置を用いて撮影された画像や、高負荷な設定により撮影された画像であってもよい。 Here, the learning data of the image quality improvement model for performing the image quality improvement processing on the various images is the same as the learning data of the image quality improvement model according to the sixth embodiment. The image after the image quality improvement processing is used as input data and output data. Note that, as in the sixth embodiment, the image quality improvement processing on the learning data includes, for example, an averaging processing, a processing using a smoothing filter, a maximum posterior probability estimation processing (MAP estimation processing), a gradation conversion processing, and the like. It may be. Further, as the image after the image quality improvement processing, for example, an image on which filter processing such as noise removal and edge enhancement has been performed may be used, or an image in which contrast is adjusted such that a low-luminance image is changed to a high-luminance image. May be used. Further, since the output data of the teacher data relating to the high image quality model only needs to be a high quality image, the image was captured using an OCT device having higher performance than the OCT device used when capturing the image as the input data. The image may be an image or an image captured with a high load setting.
　また、高画質化モデルは、高画質化処理を行う画像の種類毎に用意されてもよい。例えば、断層画像用の高画質化モデルや輝度のＥｎ－Ｆａｃｅ画像用の高画質化モデル、ＯＣＴＡ正面画像用の高画質化モデルが用意されてよい。さらに、輝度のＥｎ－Ｆａｃｅ画像用の高画質化モデルやＯＣＴＡ正面画像用の高画質化モデルは、画像の生成に係る深度範囲（生成範囲）について異なる深度範囲の画像を網羅的に学習した学習済モデルであってよい。異なる深度範囲の画像としては、例えば、図２１Ａに示すように、表層（Ｉｍ２１１０）、深層（Ｉｍ２１２０）、外層（Ｉｍ２１３０）、及び脈絡膜血管網（Ｉｍ１９４０）などの画像が含まれてよい。また、輝度のＥｎ－Ｆａｃｅ画像用の高画質化モデルやＯＣＴＡ正面画像用の高画質化モデルは、異なる深度範囲毎の画像を学習した複数の高画質化モデルが用意されてもよい。 The image quality improvement model may be prepared for each type of image to be subjected to image quality improvement processing. For example, a high-quality model for a tomographic image, a high-quality model for a luminance En-Face image, and a high-quality model for an OCTA front image may be prepared. Further, the image quality enhancement model for the luminance En-Face image and the image quality enhancement model for the OCTA front image include a learning method in which images of different depth ranges are comprehensively learned for a depth range (generation range) related to image generation. Model may be used. The images in different depth ranges may include, for example, images of the surface layer (Im2110), the deep layer (Im2120), the outer layer (Im2130), and the choroidal vascular network (Im1940), as shown in FIG. 21A. Further, as the image quality improvement model for the luminance En-Face image and the image quality improvement model for the OCTA front image, a plurality of image quality improvement models that have learned images for different depth ranges may be prepared.
　また、断層画像用の高画質化モデルを用意する場合には、異なる副走査（Ｙ）方向の位置で得られた断層画像を網羅的に学習した学習済モデルであってよい。図２１Ｂに示す断層画像Ｉｍ２１５１～Ｉｍ２１５３は、異なる副走査方向の位置で得られた断層画像の例である。ただし、撮影部位（例えば、黄斑部中心、視神経乳頭部中心）が異なる場所を撮影した画像の場合には、部位ごとに別々に学習をするようにしてもよいし、撮影部位を気にせずに一緒に学習をするようにしてもよい。なお、高画質化する断層画像としては、輝度の断層画像と、モーションコントラストデータの断層画像とが含まれてよい。ただし、輝度の断層画像とモーションコントラストデータの断層画像においては画像特徴量が大きく異なるため、それぞれの高画質化モデルとして別々に学習を行ってもよい。 In the case where a high quality image model for a tomographic image is prepared, a learned model that comprehensively learns tomographic images obtained at different positions in the sub-scanning (Y) direction may be used. The tomographic images Im2151 to Im2153 shown in FIG. 21B are examples of tomographic images obtained at different positions in the sub-scanning direction. However, in the case of an image obtained by capturing an image of a place where the imaging part (for example, the center of the macula, the center of the optic nerve head) is different, learning may be separately performed for each part, or the imaging part may be taken care of. You may learn together. It should be noted that the tomographic image to be improved in image quality may include a tomographic image of luminance and a tomographic image of motion contrast data. However, since the image feature amount greatly differs between the tomographic image of the luminance and the tomographic image of the motion contrast data, learning may be separately performed as the respective image quality improvement models.
　以下、本実施例に係る画像処理システムによる画像処理について、実施例６による画像処理との違いを中心として説明する。なお、本実施例に係る画像処理システムの構成及び処理手順は、実施例６に係る画像処理システム８の構成及び処理手順と同様であるため、同一の参照符号を用いて示し、説明を省略する。 Hereinafter, image processing by the image processing system according to the present embodiment will be described focusing on differences from the image processing according to the sixth embodiment. The configuration and the processing procedure of the image processing system according to the present embodiment are the same as the configuration and the processing procedure of the image processing system 8 according to the sixth embodiment. .
　本実施例では、第一の処理部８２２が高画質化処理を行った画像を表示制御部２５が表示部５０に表示を行う例について説明を行う。なお、本実施例では、図２２Ａ及び図２２Ｂを用いて説明を行うが表示画面はこれに限らない。経過観察のように、異なる日時で得た複数の画像を並べて表示する表示画面においても同様に高画質化処理（画質向上処理）は適用可能である。また、撮影確認画面のように、検者が撮影直後に撮影成否を確認する表示画面においても同様に高画質化処理は適用可能である。表示制御部２５は、第一の処理部８２２が生成した複数の高画質画像や高画質化を行っていない低画質画像を表示部５０に表示させることができる。また、表示制御部２５は、表示部５０に表示された複数の高画質画像や高画質化を行っていない低画質画像について、検者の指示に応じて選択された低画質画像及び高画質画像をそれぞれ表示部５０に表示させることができる。また、画像処理装置８０は、当該検者の指示に応じて選択された低画質画像及び高画質画像を外部に出力することもできる。 In the present embodiment, an example will be described in which the display control unit 25 displays an image on which the first processing unit 822 has performed the image quality improvement processing on the display unit 50. In this embodiment, description will be given with reference to FIGS. 22A and 22B, but the display screen is not limited to this. Like a follow-up observation, the image quality improvement processing (image quality improvement processing) can be similarly applied to a display screen in which a plurality of images obtained at different dates and times are displayed side by side. The image quality improvement processing can be similarly applied to a display screen in which the examiner confirms the success or failure of imaging immediately after imaging, such as an imaging confirmation screen. The display control unit 25 can cause the display unit 50 to display the plurality of high-quality images generated by the first processing unit 822 and the low-quality images that have not been subjected to high-quality image processing. In addition, the display control unit 25 may select a low-quality image and a high-quality image selected according to an instruction from the examiner, for a plurality of high-quality images and low-quality images that have not been subjected to the high-quality image displayed on the display unit 50. Can be displayed on the display unit 50 respectively. Further, the image processing device 80 can also output the low-quality image and the high-quality image selected according to the instruction of the examiner to the outside.
　以下、図２２Ａ及び図２２Ｂを参照して、本実施例に係るインターフェースの表示画面２２００の一例を示す。表示画面２２００は画面全体を示し、表示画面２２００には、患者タブ２２０１、撮影タブ２２０２、レポートタブ２２０３、設定タブ２２０４が示されている。また、レポートタブ２２０３における斜線は、レポート画面のアクティブ状態を表している。本実施例においては、レポート画面を表示する例について説明する。 Hereinafter, an example of the display screen 2200 of the interface according to the present embodiment will be described with reference to FIGS. 22A and 22B. The display screen 2200 shows the entire screen. The display screen 2200 shows a patient tab 2201, an imaging tab 2202, a report tab 2203, and a setting tab 2204. A hatched line in the report tab 2203 indicates an active state of the report screen. In the present embodiment, an example in which a report screen is displayed will be described.
　図２２Ａに示されるレポート画面には、ＳＬＯ画像Ｉｍ２２０５、ＯＣＴＡ正面画像Ｉｍ２２０７，２２０８、輝度のＥｎ－Ｆａｃｅ画像Ｉｍ２２０９、断層画像Ｉｍ２２１１，２２１２、及びボタン２２２０が示されている。また、ＳＬＯ画像Ｉｍ２２０５には、ＯＣＴＡ正面画像Ｉｍ２２０７に対応するＯＣＴＡ正面画像Ｉｍ２２０６が重畳表示されている。さらに、断層画像Ｉｍ２２１１，２２１２には、それぞれＯＣＴＡ正面画像Ｉｍ２２０７，Ｉｍ２２０８の深度範囲の境界線２２１３，２２１４が重畳表示されている。ボタン２２２０は、高画質化処理の実行を指定するためのボタンである。ボタン２２２０は、後述するように、高画質画像の表示を指示するためのボタンであってもよい。 レ ポ ー ト The report screen shown in FIG. 22A shows an SLO image Im2205, OCTA front images Im2207 and 2208, a luminance En-Face image Im2209, tomographic images Im2211 and 2122, and a button 2220. Further, an OCTA front image Im2206 corresponding to the OCTA front image Im2207 is superimposed on the SLO image Im2205. Further, boundary lines 2213 and 2214 of the depth range of the OCTA front images Im2207 and Im2208 are superimposed on the tomographic images Im2211 and Im2212, respectively. The button 2220 is a button for designating execution of the image quality improvement processing. The button 2220 may be a button for instructing display of a high-quality image, as described later.
　本実施例において、高画質化処理の実行はボタン２２２０を指定して行うか、データベースに保存（記憶）されている情報に基づいて実行の有無を判断する。初めに、検者からの指示に応じてボタン２２２０を指定することで高画質画像の表示と低画質画像の表示を切り替える例について説明する。なお、以下、高画質化処理の対象画像はＯＣＴＡ正面画像として説明する。 In the present embodiment, the execution of the image quality improvement processing is performed by designating the button 2220, or the presence or absence of the execution is determined based on information stored (stored) in the database. First, an example will be described in which the button 2220 is designated according to an instruction from the examiner to switch between the display of a high-quality image and the display of a low-quality image. Hereinafter, the target image of the image quality improvement processing will be described as an OCTA front image.
　なお、ＯＣＴＡ正面画像Ｉｍ２２０７，Ｉｍ２２０８の深度範囲は、第一の検出処理及び第二の検出処理により検出された網膜層の情報を用いて定められてよい。深度範囲は、例えば、検出された網膜層に関する２つの層境界の間の範囲としてもよいし、検出された網膜層に関する２つの層境界の一方を基準として、より深い方向又はより浅い方向に所定の画素数分だけ含んだ範囲であってもよい。また、深度範囲は、例えば、検出された網膜層に関する２つの層境界の間の範囲から、操作者の指示に応じて変更された（オフセットされた）範囲であってもよい。 Note that the depth range of the OCTA front images Im2207 and Im2208 may be determined using the information of the retinal layers detected by the first detection processing and the second detection processing. The depth range may be, for example, a range between two layer boundaries relating to the detected retinal layer, or may be predetermined in a deeper or shallower direction based on one of the two layer boundaries relating to the detected retinal layer. May be included in the range of the number of pixels. Further, the depth range may be, for example, a range that is changed (offset) according to an instruction of the operator from a range between two detected layer boundaries of the retinal layer.
　検者がレポートタブ２２０３を指定してレポート画面に遷移した際には、表示制御部２５は、低画質なＯＣＴＡ正面画像Ｉｍ２２０７，Ｉｍ２２０８を表示する。その後、検者がボタン２２２０を指定することで、第一の処理部８２２は画面に表示されているＯＣＴＡ正面画像Ｉｍ２２０７，Ｉｍ２２０８に対して高画質化処理を実行する。高画質化処理が完了後、表示制御部２５は、第一の処理部８２２が生成した高画質画像をレポート画面に表示する。なお、ＯＣＴＡ正面画像Ｉｍ２２０６は、ＯＣＴＡ正面画像Ｉｍ２２０７をＳＬＯ画像Ｉｍ２２０５に重畳表示しているものであるため、表示制御部２５は、ＯＣＴＡ正面Ｉｍ２２０６についても高画質化処理した画像を表示させることができる。また、表示制御部２５は、ボタン２２２０の表示をアクティブ状態に変更し、高画質化処理を実行したことが分かるような表示とすることができる。 When the examiner designates the report tab 2203 and transits to the report screen, the display control unit 25 displays the low-quality OCTA front images Im2207 and Im2208. Thereafter, when the examiner specifies the button 2220, the first processing unit 822 executes the image quality improvement processing on the OCTA front images Im2207 and Im2208 displayed on the screen. After the completion of the high image quality processing, the display control unit 25 displays the high quality image generated by the first processing unit 822 on the report screen. Since the OCTA front image Im2206 is obtained by superimposing and displaying the OCTA front image Im2207 on the SLO image Im2205, the display control unit 25 can also display the OCTA front image Im2206 with a high-quality image. . In addition, the display control unit 25 can change the display of the button 2220 to the active state, and display the button 2220 so that the user can know that the image quality improvement processing has been performed.
　ここで、第一の処理部８２２における処理の実行は、検者がボタン２２２０を指定したタイミングに限る必要はない。レポート画面を開く際に表示するＯＣＴＡ正面画像Ｉｍ２２０７，Ｉｍ２２０８の種類は事前に分かっているため、第一の処理部８２２は、表示される画面がレポート画面に遷移する際に高画質化処理の実行をしてもよい。そして、ボタン２２２０が押下されたタイミングで、表示制御部２５が高画質画像をレポート画面に表示するようにしてもよい。さらに、検者からの指示に応じて、又はレポート画面に遷移する際に高画質化処理を行う画像の種類は２種類である必要はない。表示する可能性の高い画像、例えば、図２１Ａで示すような表層（Ｉｍ２１１０）、深層（Ｉｍ２１２０）、外層（Ｉｍ２１３０）、及び脈絡膜血管網（Ｉｍ２１４０）などの複数のＯＣＴＡ正面画像に対して処理を行うようにしてもよい。この場合、高画質化処理を行った画像を一時的にメモリに記憶、あるいはデータベースに記憶しておくようにしてもよい。 Here, the execution of the process in the first processing unit 822 does not need to be limited to the timing at which the examiner specifies the button 2220. Since the types of the OCTA front images Im2207 and Im2208 to be displayed when the report screen is opened are known in advance, the first processing unit 822 executes the high image quality processing when the displayed screen transitions to the report screen. You may do. Then, at the timing when the button 2220 is pressed, the display control unit 25 may display a high-quality image on the report screen. Furthermore, there is no need for two types of images to be subjected to the image quality improvement processing in response to an instruction from the examiner or when transitioning to the report screen. Processing is performed on a plurality of OCTA front images such as the surface layer (Im2110), the deep layer (Im2120), the outer layer (Im2130), and the choroidal vascular network (Im2140) as shown in FIG. 21A, which are likely to be displayed. It may be performed. In this case, the image on which the image quality improvement processing has been performed may be temporarily stored in a memory or may be stored in a database.
　次に、データベースに保存（記録）されている情報に基づいて高画質化処理を実行する場合について説明する。データベースに高画質化処理の実行を行う状態が保存されている場合、レポート画面に遷移した際に、第一の処理部８２２が高画質化処理を実行して得た高画質画像を表示制御部２５がデフォルトで表示部５０に表示させる。そして、表示制御部２５が、ボタン２２２０をアクティブ状態としてデフォルトで表示させることで、検者に対しては高画質化処理を実行して得た高画質画像が表示されていることが分かるように構成することができる。検者は、高画質化処理前の低画質画像を表示したい場合には、ボタン２２２０を指定してアクティブ状態を解除することで、表示制御部２５が低画質画像を表示部５０に表示させることができる。この際、検者は、表示される画像を高画質画像に戻したい場合には、ボタン２２２０を指定してアクティブ状態とすることで、表示制御部２５が高画質画像を表示部５０に再び表示させる。 Next, a description will be given of a case where the image quality improvement processing is executed based on the information stored (recorded) in the database. In the case where a state in which the image quality improvement process is performed is stored in the database, the first processing unit 822 displays the high image quality image obtained by executing the image quality improvement process when the display screen transitions to the report screen. 25 is displayed on the display unit 50 by default. Then, the display control unit 25 displays the button 2220 as an active state by default so that the examiner can see that the high-quality image obtained by executing the high-quality processing is displayed. Can be configured. When the examiner wants to display the low-quality image before the high-quality processing, the display control unit 25 causes the display unit 50 to display the low-quality image by specifying the button 2220 and releasing the active state. Can be. At this time, when the examiner wants to return the displayed image to the high-quality image, the display control unit 25 displays the high-quality image on the display unit 50 again by designating the button 2220 to be in the active state. Let it.
　データベースへの高画質化処理の実行有無は、データベースに保存されているデータ全体に対して共通、及び撮影データ毎（検査毎）など、階層別に指定するものとする。例えば、データベース全体に対して高画質化処理を実行する状態を保存してある場合において、個別の撮影データ（個別の検査）に対して、検者が高画質化処理を実行しない状態を保存することができる。この場合、高画質化処理を実行しないとした状態が保存された個別の撮影データについては次回表示する際に高画質化処理を実行しない状態で表示を行うことができる。このような構成によれば、撮影データ単位（検査単位）で高画質化処理の実行の有無が指定されていない場合、データベース全体に対して指定されている情報に基づいて処理を実行することができる。また、撮影データ単位（検査単位）で指定されている場合には、その情報に基づいて個別に処理を実行することができる。 (4) Whether or not the image quality improvement processing is performed on the database is specified for each layer, such as common to all the data stored in the database and for each photographing data (for each inspection). For example, when the state in which the image quality improvement processing is executed is stored for the entire database, the state in which the examiner does not execute the image quality improvement processing is stored for individual imaging data (individual examination). be able to. In this case, the individual imaging data in which the state in which the image quality improvement processing is not performed can be displayed without performing the image quality improvement processing at the next display. According to such a configuration, when it is not specified whether or not the image quality improvement processing is to be performed in units of imaging data (inspection units), it is possible to execute the processing based on the information specified for the entire database. it can. In the case where the image data is specified in units of imaging data (inspection units), it is possible to individually execute processing based on the information.
　なお、撮影データ毎（検査毎）に高画質化処理の実行状態を保存するために、不図示のユーザーインターフェース（例えば、保存ボタン）を用いてもよい。また、他の撮影データ（他の検査）や他の患者データに遷移（例えば、検者からの指示に応じてレポート画面以外の表示画面に変更）する際に、表示状態（例えば、ボタン２２２０の状態）に基づいて、高画質化処理の実行を行う状態が保存されるようにしてもよい。 Note that a user interface (for example, a save button) (not shown) may be used to save the execution state of the image quality improvement processing for each piece of imaging data (for each inspection). In addition, when transitioning to another imaging data (other examination) or another patient data (for example, changing to a display screen other than the report screen in response to an instruction from the examiner), the display state (for example, the button 2220 Based on the (state), the state in which the image quality improving process is performed may be stored.
　本実施例では、ＯＣＴＡ正面画像として、ＯＣＴＡ正面画像Ｉｍ２２０７，Ｉｍ２２０８を表示する例を示しているが、表示するＯＣＴＡ正面画像は検者の指定により変更することが可能である。そのため、高画質化処理の実行が指定されている場合（ボタン２２２０がアクティブ状態）における、表示する画像の変更について説明する。 In the present embodiment, an example is shown in which the OCTA front images Im2207 and Im2208 are displayed as the OCTA front images, but the displayed OCTA front images can be changed by the examiner's designation. Therefore, a description will be given of the change of the image to be displayed when the execution of the image quality improvement processing is specified (the button 2220 is in the active state).
　表示する画像の変更は、不図示のユーザーインターフェース（例えば、コンボボックス）を用いて行うことができる。例えば、検者が画像の種類を表層から脈絡膜血管網に変更した場合には、第一の処理部８２２は脈絡膜血管網画像に対して高画質化処理を実行し、表示制御部２５は第一の処理部８２２が生成した高画質な画像をレポート画面に表示する。すなわち、表示制御部２５は、検者からの指示に応じて、第一の深度範囲の高画質画像の表示を、第一の深度範囲とは少なくとも一部が異なる第二の深度範囲の高画質画像の表示に変更してもよい。このとき、表示制御部２５は、検者からの指示に応じて第一の深度範囲が第二の深度範囲に変更されることにより、第一の深度範囲の高画質画像の表示を、第二の深度範囲の高画質画像の表示に変更してもよい。なお、上述したようにレポート画面遷移時に表示する可能性の高い画像に対しては、既に高画質画像が生成済みである場合、表示制御部２５は生成済みの高画質な画像を表示すればよい。 画像 The displayed image can be changed using a user interface (for example, a combo box) not shown. For example, when the examiner changes the type of the image from the surface layer to the choroidal vascular network, the first processing unit 822 performs the image quality improvement processing on the choroidal vascular network image, and the display control unit 25 performs the first image processing. The high-quality image generated by the processing unit 822 is displayed on the report screen. That is, the display control unit 25 changes the display of the high-quality image in the first depth range to the high-quality image in the second depth range that is at least partially different from the first depth range in response to an instruction from the examiner. The display may be changed to an image display. At this time, the display control unit 25 changes the first depth range to the second depth range in response to an instruction from the examiner, thereby displaying the high-quality image in the first depth range. May be changed to display a high-quality image in a depth range of. As described above, if a high-quality image has already been generated for an image that is likely to be displayed when the report screen transitions, the display control unit 25 may display the generated high-quality image. .
　また、画像の種類の変更方法は上記したものに限らず、基準となる層やオフセットの値を変えて異なる深度範囲を設定したＯＣＴＡ正面画像を生成し、生成したＯＣＴＡ正面画像に高画質化処理を実行した高画質画像を表示させることも可能である。その場合、基準となる層、又はオフセット値が変更された時に、第一の処理部８２２は任意のＯＣＴＡ正面画像に対して高画質化処理を実行し、表示制御部２５は高画質画像をレポート画面に表示する。なお、基準となる層やオフセット値の変更は、不図示のユーザーインターフェース（例えば、コンボボックスやテキストボックス）を用いて行われることができる。また、断層画像Ｉｍ２２１１，Ｉｍ２２１２にそれぞれ重畳表示している境界線２２１３，２２１４のいずれかをドラッグ（層境界を移動）することで、ＯＣＴＡ正面画像の深度範囲（生成範囲）を変更することもできる。 Further, the method of changing the type of image is not limited to the above-described method. An OCTA front image in which a different depth range is set by changing a reference layer or an offset value is generated, and the generated OCTA front image is subjected to a high quality processing. Can be displayed. In this case, when the reference layer or the offset value is changed, the first processing unit 822 executes the high image quality processing on an arbitrary OCTA front image, and the display control unit 25 reports the high image image. Display on the screen. The reference layer and the offset value can be changed using a user interface (not shown) (for example, a combo box or a text box). Also, the depth range (generation range) of the OCTA front image can be changed by dragging (moving the layer boundary) any of the boundaries 2213 and 2214 superimposed on the tomographic images Im2211 and Im2212. .
　境界線をドラッグによって変更する場合、高画質化処理の実行命令が連続的に実施される。そのため、第一の処理部８２２は実行命令に対して常に処理を行ってもよいし、ドラッグによる層境界の変更後に実行するようにしてもよい。又は、高画質化処理の実行は連続的に命令されるが、次の命令が来た時点で前回の命令をキャンセルし、最新の命令を実行するようにしてもよい。 す る When the boundary line is changed by dragging, the execution instruction of the image quality improvement processing is continuously executed. Therefore, the first processing unit 822 may always process the execution instruction, or may execute the processing after the layer boundary is changed by dragging. Alternatively, the execution of the image quality improvement processing is continuously instructed, but when the next instruction comes, the previous instruction may be canceled and the latest instruction may be executed.
　なお、高画質化処理には比較的時間がかかる場合がある。このため、上述したどのようなタイミングで命令が実行されたとしても、高画質画像が表示されるまでに比較的時間がかかる場合がある。そこで、検者からの指示に応じてＯＣＴＡ正面画像を生成するための深度範囲が設定されてから、高画質画像が表示されるまでの間、該設定された深度範囲に対応する低画質なＯＣＴＡ正面画像（低画質画像）が表示されてもよい。すなわち、上記深度範囲が設定されると、該設定された深度範囲に対応する低画質なＯＣＴＡ正面画像（低画質画像）が表示され、高画質化処理が終了すると、該低画質なＯＣＴＡ正面画像の表示が高画質画像の表示に変更されるように構成されてもよい。また、上記深度範囲が設定されてから、高画質画像が表示されるまでの間、高画質化処理が実行されていることを示す情報が表示されてもよい。なお、これらの処理は、高画質化処理の実行が既に指定されている状態（ボタン２２２０がアクティブ状態）を前提とする場合に適用される構成に限られない。例えば、検者からの指示に応じて高画質化処理の実行が指示された際に、高画質画像が表示されるまでの間においても、これらの処理を適用することが可能である。 The high-quality processing may take a relatively long time. Therefore, even if the command is executed at any timing described above, it may take a relatively long time before a high-quality image is displayed. Therefore, after a depth range for generating an OCTA front image according to an instruction from the examiner is set and before a high-quality image is displayed, a low-quality OCTA corresponding to the set depth range is displayed. A front image (low-quality image) may be displayed. That is, when the above-described depth range is set, a low-quality OCTA front image (low-quality image) corresponding to the set depth range is displayed. May be changed to display a high-quality image. In addition, information indicating that the high-quality image processing is being performed may be displayed from when the depth range is set until the high-quality image is displayed. Note that these processes are not limited to the configuration applied when it is assumed that the execution of the image quality improvement process has already been specified (the button 2220 is in the active state). For example, when the execution of the image quality improvement processing is instructed in response to the instruction from the examiner, these processings can be applied until the high quality image is displayed.
　本実施例では、ＯＣＴＡ正面画像として、異なる層に関するＯＣＴＡ正面画像Ｉｍ２２０７，２１０８を表示し、低画質と高画質な画像は切り替えて表示する例を示したが、表示される画像はこれに限らない。例えば、ＯＣＴＡ正面画像Ｉｍ２２０７として低画質なＯＣＴＡ正面画像、ＯＣＴＡ正面画像Ｉｍ２２０８として高画質なＯＣＴＡ正面画像を並べて表示するようにしてもよい。画像を切り替えて表示する場合には、同じ場所で画像を切り替えるので変化がある部分の比較を行いやすく、並べて表示する場合には、同時に画像を表示することができるので画像全体を比較しやすい。 In this embodiment, the OCTA front images Im2207 and 2108 relating to different layers are displayed as the OCTA front images, and the low-quality and high-quality images are switched and displayed. However, the displayed images are not limited thereto. . For example, a low-quality OCTA front image may be displayed side by side as the OCTA front image Im2207, and a high-quality OCTA front image may be displayed as the OCTA front image Im2208. When the images are switched and displayed, the images are switched at the same place, so that it is easy to compare the changed portions. When the images are displayed side by side, the images can be displayed at the same time, so that the entire image can be easily compared.
　次に、図２２Ａ及び図２２Ｂを用いて、画面遷移における高画質化処理の実行について説明を行う。図２２Ｂは、図２２ＡにおけるＯＣＴＡ正面画像Ｉｍ２２０７を拡大表示した画面例である。図２２Ｂにおいても、図２２Ａと同様にボタン２２２０を表示する。図２２Ａから図２２Ｂへの画面遷移は、例えば、ＯＣＴＡ正面画像Ｉｍ２２０７をダブルクリックすることで遷移し、図２２Ｂから図２２Ａへは閉じるボタン２２３０で遷移する。なお、画面遷移に関しては、ここで示した方法に限らず、不図示のユーザーインターフェースを用いてもよい。 Next, with reference to FIGS. 22A and 22B, a description will be given of the execution of the image quality improvement processing at the screen transition. FIG. 22B is a screen example in which the OCTA front image Im2207 in FIG. 22A is enlarged and displayed. Also in FIG. 22B, a button 2220 is displayed as in FIG. 22A. The screen transition from FIG. 22A to FIG. 22B is made, for example, by double-clicking on the OCTA front image Im2207, and is made with the close button 2230 from FIG. 22B to FIG. 22A. Note that the screen transition is not limited to the method shown here, and a user interface (not shown) may be used.
　画面遷移の際に高画質化処理の実行が指定されている場合（ボタン２２２０がアクティブ）、画面遷移時においてもその状態を保つ。すなわち、図２２Ａの画面で高画質画像を表示している状態で図２２Ｂの画面に遷移する場合、図２２Ｂの画面においても高画質画像を表示する。そして、ボタン２２２０はアクティブ状態にする。図２２Ｂから図２２Ｂへ遷移する場合にも同様である。図２２Ｂにおいて、ボタン２２２０を指定して低画質画像に表示を切り替えることもできる。 (4) When the execution of the image quality improvement processing is designated at the time of the screen transition (the button 2220 is active), the state is maintained even at the time of the screen transition. That is, in a case where a transition is made to the screen of FIG. 22B while the high-quality image is being displayed on the screen of FIG. 22A, the high-quality image is also displayed on the screen of FIG. 22B. Then, the button 2220 is activated. The same applies to the transition from FIG. 22B to FIG. 22B. In FIG. 22B, the display can be switched to a low-quality image by designating a button 2220.
　画面遷移に関して、ここで示した画面に限らず、経過観察用の表示画面、又はパノラマ用の表示画面など同じ撮影データを表示する画面への遷移であれば、高画質画像の表示状態を保ったまま遷移を行うことができる。すなわち、遷移後の表示画面において、遷移前の表示画面におけるボタン２２２０の状態に対応する画像が表示されることができる。例えば、遷移前の表示画面におけるボタン２２２０がアクティブ状態であれば、遷移後の表示画面において高画質画像が表示される。また、例えば、遷移前の表示画面におけるボタン２２２０のアクティブ状態が解除されていれば、遷移後の表示画面において低画質画像が表示される。なお、経過観察用の表示画面におけるボタン２２２０がアクティブ状態になると、経過観察用の表示画面に並べて表示される異なる日時（異なる検査日）で得た複数の画像が高画質画像に切り換わるようにしてもよい。すなわち、経過観察用の表示画面におけるボタン２２２０がアクティブ状態になると、異なる日時で得た複数の画像に対して一括で反映されるように構成してもよい。 Regarding the screen transition, not only the screen shown here, but also a transition to a screen that displays the same shooting data such as a display screen for follow-up observation or a display screen for panorama, the display state of the high-quality image is maintained. The transition can be performed as it is. That is, an image corresponding to the state of the button 2220 on the display screen before the transition can be displayed on the display screen after the transition. For example, if the button 2220 on the display screen before the transition is in the active state, a high-quality image is displayed on the display screen after the transition. Further, for example, if the active state of the button 2220 on the display screen before the transition is released, a low image quality image is displayed on the display screen after the transition. When the button 2220 on the display screen for follow-up observation is activated, a plurality of images obtained at different dates and times (different inspection dates) displayed side by side on the display screen for follow-up observation are switched to high-quality images. You may. That is, when the button 2220 on the display screen for follow-up observation becomes active, it may be configured to be reflected collectively on a plurality of images obtained at different dates and times.
　なお、経過観察用の表示画面の例を、図２３に示す。検者からの指示に応じてタブ２３０１が選択されると、図２３のように、経過観察用の表示画面が表示される。このとき、ＯＣＴＡ正面画像の深度範囲を、リストボックス２３０２，２３０３に表示された既定の深度範囲セットから検者が所望するセットを選択することで変更できる。例えば、リストボックス２３０２では網膜表層が選択され、また、リストボックス２３０３では網膜深層が選択されている。上側の表示領域には網膜表層のＯＣＴＡ正面画像の解析結果が表示され、また、下側の表示領域には網膜深層のＯＣＴＡ正面画像の解析結果が表示されている。深度範囲が選択されると、異なる日時の複数の画像について、選択された深度範囲の複数のＯＣＴＡ正面画像の解析結果の並列表示に一括して変更される。 FIG. 23 shows an example of a display screen for follow-up observation. When the tab 2301 is selected according to an instruction from the examiner, a display screen for follow-up observation is displayed as shown in FIG. At this time, the depth range of the OCTA front image can be changed by selecting a desired set from the predetermined depth range sets displayed in the list boxes 2302 and 2303. For example, the retina surface layer is selected in the list box 2302, and the retina deep layer is selected in the list box 2303. The analysis result of the OCTA front image of the retinal surface is displayed in the upper display area, and the analysis result of the OCTA front image of the deep retina is displayed in the lower display area. When the depth range is selected, a plurality of images at different dates and times are collectively changed to a parallel display of analysis results of a plurality of OCTA front images in the selected depth range.
　このとき、解析結果の表示を非選択状態にすると、異なる日時の複数のＯＣＴＡ正面画像の並列表示に一括して変更されてもよい。そして、検者からの指示に応じてボタン２２２０が指定されると、複数のＯＣＴＡ正面画像の表示が複数の高画質画像の表示に一括して変更される。 At this time, if the display of the analysis result is set to the non-selection state, the display may be changed to a parallel display of a plurality of OCTA front images at different dates and times. When the button 2220 is designated in response to an instruction from the examiner, the display of a plurality of OCTA front images is changed to the display of a plurality of high-quality images at once.
　また、解析結果の表示が選択状態である場合には、検者からの指示に応じてボタン２２２０が指定されると、複数のＯＣＴＡ正面画像の解析結果の表示が複数の高画質画像の解析結果の表示に一括して変更される。ここで、解析結果の表示は、解析結果を任意の透明度により画像に重畳表示させたものであってもよい。このとき、画像の表示から解析結果の表示への変更は、例えば、表示されている画像に対して任意の透明度により解析結果を重畳させた状態に変更したものであってもよい。また、画像の表示から解析結果の表示への変更は、例えば、解析結果と画像とを任意の透明度によりブレンド処理して得た画像（例えば、二次元マップ）の表示への変更であってもよい。 When the display of the analysis result is in the selected state, when the button 2220 is designated in response to the instruction from the examiner, the display of the analysis result of the plurality of OCTA front images is changed to the analysis result of the plurality of high-quality images. Will be changed to the display. Here, the display of the analysis result may be a display in which the analysis result is superimposed on the image with an arbitrary transparency. At this time, the change from the display of the image to the display of the analysis result may be, for example, a change in a state in which the analysis result is superimposed on the displayed image with arbitrary transparency. Further, the change from the display of the image to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency. Good.
　また、深度範囲の指定に用いる層境界の種類とオフセット位置をそれぞれ、ユーザーインターフェース２３０５，２３０６から一括して変更することができる。なお、層境界の種類やオフセット位置を変更するためのユーザーインターフェース２３０５，２３０６は一例であり、他の任意の態様のインターフェースを用いてよい。なお、断層画像も一緒に表示させ、断層画像上に重畳された層境界データを検者からの指示に応じて移動させることにより、異なる日時の複数のＯＣＴＡ正面画像の深度範囲を一括して変更してもよい。このとき、異なる日時の複数の断層画像を並べて表示し、１つの断層画像上で上記移動が行われると、他の断層画像上でも同様に層境界データが移動されてもよい。 種類 In addition, the type and offset position of the layer boundary used to specify the depth range can be changed collectively from the user interfaces 2305 and 2306. Note that the user interfaces 2305 and 2306 for changing the type of layer boundary and the offset position are merely examples, and an interface of any other mode may be used. The depth range of a plurality of OCTA front images at different dates and times is collectively changed by displaying the tomographic image together and moving the layer boundary data superimposed on the tomographic image in accordance with an instruction from the examiner. May be. At this time, when a plurality of tomographic images at different dates and times are displayed side by side and the above-described movement is performed on one tomographic image, the layer boundary data may be similarly moved on other tomographic images.
　また、画像投影法やプロジェクションアーチファクト抑制処理の有無を、例えば、コンテキストメニューのようなユーザーインターフェースから選択することにより変更してもよい。 The presence or absence of the image projection method or the projection artifact suppression process may be changed by selecting the user interface such as a context menu, for example.
　また、選択ボタン２３０７を選択して不図示の選択画面を表示させ、該選択画面上に表示された画像リストから選択された画像が表示されてもよい。なお、図２３の上部に表示されている矢印２３０４は現在選択されている検査であることを示す印であり、基準検査（Ｂａｓｅｌｉｎｅ）はＦｏｌｌｏｗ－ｕｐ撮影の際に選択した検査（図２３の一番左側の画像）である。もちろん、基準検査を示すマークを表示部に表示させてもよい。 Alternatively, the selection button 2307 may be selected to display a selection screen (not shown), and the image selected from the image list displayed on the selection screen may be displayed. Note that an arrow 2304 displayed at the top of FIG. 23 is a mark indicating that the test is currently selected, and the reference test (Baseline) is a test selected at the time of Follow-up imaging (one of FIG. 23). Left image). Of course, a mark indicating the reference inspection may be displayed on the display unit.
　また、「Ｓｈｏｗ　Ｄｉｆｆｅｒｅｎｃｅ」チェックボックス２３０８が指定された場合には、基準画像上に基準画像に対する計測値分布（マップもしくはセクタマップ）を表示する。さらに、この場合には、それ以外の検査日に対応する領域に、基準画像に対して算出した計測値分布と当該領域に表示される画像に対して算出した計測値分布との差分計測値マップを表示する。計測結果としては、レポート画面上にトレンドグラフ（経時変化計測によって得られた各検査日の画像に対する計測値のグラフ）を表示させてもよい。すなわち、異なる日時の複数の画像に対応する複数の解析結果の時系列データ（例えば、時系列グラフ）が表示されてもよい。このとき、表示されている複数の画像に対応する複数の日時以外の日時に関する解析結果についても、表示されている複数の画像に対応する複数の解析結果と判別可能な状態で（例えば、時系列グラフ上の各点の色が画像の表示の有無で異なる）時系列データとして表示させてもよい。また、該トレンドグラフの回帰直線（曲線）や対応する数式をレポート画面に表示させてもよい。 If the “Show @ Difference” check box 2308 is specified, the measured value distribution (map or sector map) for the reference image is displayed on the reference image. Further, in this case, a difference measurement value map between the measurement value distribution calculated for the reference image and the measurement value distribution calculated for the image displayed in the region is provided in an area corresponding to the other inspection date. Is displayed. As the measurement result, a trend graph (a graph of the measurement values for the images on each inspection day obtained by the measurement over time) may be displayed on the report screen. That is, time-series data (for example, a time-series graph) of a plurality of analysis results corresponding to a plurality of images at different dates and times may be displayed. At this time, the analysis results regarding the dates and times other than the multiple dates and times corresponding to the multiple displayed images are also distinguished from the multiple analysis results corresponding to the multiple displayed images (for example, time-series (The color of each point on the graph differs depending on whether an image is displayed or not). Further, a regression line (curve) of the trend graph and a corresponding mathematical expression may be displayed on a report screen.
　本実施例においては、ＯＣＴＡ正面画像に関して説明を行ったが、本実施例に係る処理が適用される画像はこれに限らない。本実施例に係る表示、高画質化、及び画像解析等の処理に関する画像は、輝度のＥｎ－Ｆａｃｅ画像でもよい。さらには、Ｅｎ－Ｆａｃｅ画像だけではなく、Ｂ－スキャンによる断層画像、ＳＬＯ画像、眼底画像、又は蛍光眼底画像など、異なる画像であってもよい。その場合、高画質化処理を実行するためのユーザーインターフェースは、種類の異なる複数の画像に対して高画質化処理の実行を指示するもの、種類の異なる複数の画像から任意の画像を選択して高画質化処理の実行を指示するものがあってもよい。 In the present embodiment, the OCTA front image has been described, but the image to which the processing according to the present embodiment is applied is not limited to this. An image related to processing such as display, image quality improvement, and image analysis according to the present embodiment may be an En-Face image of luminance. Furthermore, not only the En-Face image but also a different image such as a B-scan tomographic image, SLO image, fundus image, or fluorescent fundus image may be used. In that case, the user interface for executing the image quality improvement processing is to instruct execution of the image quality improvement processing for a plurality of different types of images, or to select an arbitrary image from the plurality of types of the different images. There may be one that instructs execution of the high image quality processing.
　例えば、Ｂ－スキャンによる断層画像を高画質化して表示する場合には、図２２Ａに示す断層画像Ｉｍ２２１１，Ｉｍ２２１２を高画質化して表示してもよい。また、ＯＣＴＡ正面画像Ｉｍ２２０７，Ｉｍ２２０８が表示されている領域に高画質化された断層画像が表示されてもよい。なお、高画質化され、表示される断層画像は、１つだけ表示されてもよいし、複数表示されてもよい。複数の断層画像が表示される場合には、それぞれ異なる副走査方向の位置で取得された断層画像が表示されてもよいし、例えばクロススキャン等により得られた複数の断層画像を高画質化して表示する場合には、異なる走査方向の画像がそれぞれ表示されてもよい。また、例えばラジアルスキャン等により得られた複数の断層画像を高画質化して表示する場合には、一部選択された複数の断層画像（例えば基準ラインに対して互いに対称な位置の２つの断層画像）がそれぞれ表示されてもよい。さらに、図２３に示されるような経過観察用の表示画面に複数の断層画像を表示し、上述の方法と同様の手法により高画質化の指示や解析結果（例えば、特定の層の厚さ等）の表示が行われてもよい。また、上述の方法と同様の手法によりデータベースに保存されている情報に基づいて断層画像に高画質化処理を実行してもよい。 For example, when displaying a tomographic image by B-scan with high image quality, the tomographic images Im2211 and Im2212 shown in FIG. 22A may be displayed with high image quality. Further, a high-quality tomographic image may be displayed in an area where the OCTA front images Im2207 and Im2208 are displayed. In addition, only one tomographic image to be displayed with high image quality may be displayed, or a plurality of tomographic images may be displayed. When a plurality of tomographic images are displayed, tomographic images acquired at different positions in the sub-scanning direction may be displayed, or a plurality of tomographic images obtained by, for example, a cross scan may be displayed with high image quality. When displaying, images in different scanning directions may be displayed. For example, when displaying a plurality of tomographic images obtained by a radial scan or the like with high image quality, a plurality of tomographic images partially selected (for example, two tomographic images at positions symmetrical to each other with respect to a reference line). ) May be displayed. Further, a plurality of tomographic images are displayed on a display screen for follow-up observation as shown in FIG. 23, and an instruction for higher image quality or an analysis result (for example, the thickness of a specific layer, etc. ) May be displayed. Further, the image quality improvement processing may be performed on the tomographic image based on the information stored in the database by a method similar to the above-described method.
　同様に、ＳＬＯ画像を高画質化して表示する場合には、例えば、ＳＬＯ画像Ｉｍ２２０５を高画質化して表示してよい。さらに、輝度のＥｎ－Ｆａｃｅ画像を高画質化して表示する場合には、例えば輝度のＥｎ－Ｆａｃｅ画像２２０９を高画質化して表示してよい。さらに、図２３に示されるような経過観察用の表示画面に複数のＳＬＯ画像や輝度のＥｎ－Ｆａｃｅ画像を表示し、上述の方法と同様の手法により高画質化の指示や解析結果（例えば、特定の層の厚さ等）の表示が行われてもよい。また、上述の方法と同様の手法によりデータベースに保存されている情報に基づいてＳＬＯ画像や輝度のＥｎ－Ｆａｃｅ画像に高画質化処理を実行してもよい。なお、断層画像、ＳＬＯ画像、及び輝度のＥｎ－Ｆａｃｅ画像の表示は例示であり、これらの画像は所望の構成に応じて任意の態様で表示されてよい。また、ＯＣＴＡ正面画像、断層画像、ＳＬＯ画像、及び輝度のＥｎ－Ｆａｃｅ画像の少なくとも２つ以上が、一度の指示で高画質化され表示されてもよい。 Similarly, when displaying an SLO image with high image quality, for example, the SLO image Im2205 may be displayed with high image quality. Further, in the case where the luminance En-Face image is displayed with high image quality, for example, the luminance En-Face image 2209 may be displayed with high image quality. Further, a plurality of SLO images and En-Face images of luminance are displayed on the display screen for follow-up observation as shown in FIG. 23, and an instruction for higher image quality or an analysis result (for example, The display of the thickness of a specific layer, etc.) may be performed. Further, the image quality improvement processing may be performed on the SLO image or the luminance En-Face image based on the information stored in the database by the same method as the method described above. The display of the tomographic image, the SLO image, and the luminance En-Face image is merely an example, and these images may be displayed in any manner according to a desired configuration. Further, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the luminance En-Face image may be displayed with high image quality by a single instruction.
　このような構成により、本実施例に係る第一の処理部８２２が高画質化処理した画像を表示制御部２５が表示部５０に表示することができる。このとき、上述したように、高画質画像の表示、解析結果の表示、及び表示される正面画像の深度範囲等に関する複数の条件のうち少なくとも１つが選択された状態である場合には、表示画面が遷移されても、選択された状態が維持されてもよい。 With this configuration, the display control unit 25 can display the image on which the first processing unit 822 according to the present embodiment has performed the image quality improvement processing on the display unit 50. At this time, as described above, when at least one of the plurality of conditions regarding the display of the high-quality image, the display of the analysis result, and the depth range of the front image to be displayed is selected, the display screen is displayed. May be transited, or the selected state may be maintained.
　また、上述したように、複数の条件のうち少なくとも１つが選択された状態である場合には、他の条件が選択された状態に変更されても、該少なくとも１つが選択された状態が維持されてもよい。例えば、表示制御部２５は、解析結果の表示が選択状態である場合に、検者からの指示に応じて（例えば、ボタン２２２０が指定されると）、低画質画像の解析結果の表示を高画質画像の解析結果の表示に変更してもよい。また、表示制御部２５は、解析結果の表示が選択状態である場合に、検者からの指示に応じて（例えば、ボタン２２２０の指定が解除されると）、高画質画像の解析結果の表示を低画質画像の解析結果の表示に変更してもよい。 Further, as described above, when at least one of the plurality of conditions is in a selected state, the state in which at least one is selected is maintained even if another condition is changed to a selected state. You may. For example, when the display of the analysis result is in the selected state, the display control unit 25 changes the display of the analysis result of the low image quality image to high according to an instruction from the examiner (for example, when the button 2220 is designated). The display may be changed to a display of the analysis result of the image quality image. When the display of the analysis result is in the selected state, the display control unit 25 displays the analysis result of the high-quality image in response to an instruction from the examiner (for example, when the designation of the button 2220 is released). May be changed to the display of the analysis result of the low-quality image.
　また、表示制御部２５は、高画質画像の表示が非選択状態である場合に、検者からの指示に応じて（例えば、解析結果の表示の指定が解除されると）、低画質画像の解析結果の表示を低画質画像の表示に変更してもよい。また、表示制御部２５は、高画質画像の表示が非選択状態である場合に、検者からの指示に応じて（例えば、解析結果の表示が指定されると）、低画質画像の表示を低画質画像の解析結果の表示に変更してもよい。また、表示制御部２５は、高画質画像の表示が選択状態である場合に、検者からの指示に応じて（例えば、解析結果の表示の指定が解除されると）、高画質画像の解析結果の表示を高画質画像の表示に変更してもよい。また、表示制御部２５は、高画質画像の表示が選択状態である場合に、検者からの指示に応じて（例えば、解析結果の表示が指定されると）、高画質画像の表示を高画質画像の解析結果の表示に変更してもよい。 Further, when the display of the high-quality image is in the non-selection state, the display control unit 25 responds to an instruction from the examiner (for example, when the display of the analysis result is released), and The display of the analysis result may be changed to the display of a low-quality image. In addition, when the display of the high-quality image is in the non-selection state, the display control unit 25 displays the low-quality image in response to an instruction from the examiner (for example, when the display of the analysis result is specified). The display of the analysis result of the low-quality image may be changed. Further, when the display of the high-quality image is in the selected state, the display control unit 25 analyzes the high-quality image in response to an instruction from the examiner (for example, when the display of the analysis result is released). The display of the result may be changed to the display of a high-quality image. In addition, when the display of the high-quality image is in the selected state, the display control unit 25 increases the display of the high-quality image according to the instruction from the examiner (for example, when the display of the analysis result is specified). The display may be changed to a display of the analysis result of the image quality image.
　また、高画質画像の表示が非選択状態で且つ第一の種類の解析結果の表示が選択状態である場合を考える。この場合には、表示制御部２５は、検者からの指示に応じて（例えば、第二の種類の解析結果の表示が指定されると）、低画質画像の第一の種類の解析結果の表示を低画質画像の第二の種類の解析結果の表示に変更してもよい。また、高画質画像の表示が選択状態で且つ第一の種類の解析結果の表示が選択状態である場合を考える。この場合には、表示制御部２５は、検者からの指示に応じて（例えば、第二の種類の解析結果の表示が指定されると）、高画質画像の第一の種類の解析結果の表示を高画質画像の第二の種類の解析結果の表示に変更してもよい。 {Suppose that the display of the high-quality image is in the non-selected state and the display of the first type of analysis result is in the selected state. In this case, in response to an instruction from the examiner (for example, when the display of the second type of analysis result is designated), the display control unit 25 converts the first type of analysis result of the low-quality image into an image. The display may be changed to the display of the second type of analysis result of the low image quality image. It is also assumed that the display of the high-quality image is in the selected state and the display of the first type of analysis result is in the selected state. In this case, the display control unit 25 responds to an instruction from the examiner (for example, when the display of the second type of analysis result is specified), the first type of analysis result of the high-quality image is displayed. The display may be changed to the display of the second type of analysis result of the high quality image.
　なお、経過観察用の表示画面においては、上述したように、これらの表示の変更が、異なる日時で得た複数の画像に対して一括で反映されるように構成してもよい。ここで、解析結果の表示は、解析結果を任意の透明度により画像に重畳表示させたものであってもよい。このとき、解析結果の表示への変更は、例えば、表示されている画像に対して任意の透明度により解析結果を重畳させた状態に変更したものであってもよい。また、解析結果の表示への変更は、例えば、解析結果と画像とを任意の透明度によりブレンド処理して得た画像（例えば、二次元マップ）の表示への変更であってもよい。 Note that, as described above, the display screen for follow-up observation may be configured so that these display changes are collectively reflected on a plurality of images obtained at different dates and times. Here, the display of the analysis result may be a display in which the analysis result is superimposed on the image with an arbitrary transparency. At this time, the display of the analysis result may be changed, for example, to a state where the analysis result is superimposed on the displayed image with arbitrary transparency. The change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.
　なお、本実施例では、第一の処理部８２２が高画質化処理に関する学習済モデル（高画質化モデル）を用いて断層画像の画質を改善した高画質画像を生成した。しかしながら、高画質化モデルを用いて高画質画像を生成する構成要素は第一の処理部８２２に限られない。例えば、第一の処理部８２２とは別の第三の処理部（高画質化部）を設け、第三の処理部が高画質化モデルを用いて高画質画像を生成してもよい。この場合、第三の処理部や高画質化モデルは、ＣＰＵやＭＰＵ、ＧＰＵ、ＦＰＧＡ等のプロセッサーによって実行されるソフトウェアモジュール等で構成されてもよいし、ＡＳＩＣ等の特定の機能を果たす回路等によって構成されてもよい。 In the present embodiment, the first processing unit 822 generates a high-quality image in which the image quality of the tomographic image is improved using a learned model (high-quality model) related to the high-quality process. However, a component that generates a high-quality image using the high-quality model is not limited to the first processing unit 822. For example, a third processing unit (image quality improving unit) different from the first processing unit 822 may be provided, and the third processing unit may generate a high quality image using the image quality improving model. In this case, the third processing unit and the high image quality model may be configured by a software module or the like executed by a processor such as a CPU, an MPU, a GPU, or an FPGA, or a circuit that performs a specific function such as an ASIC. May be configured.
（実施例６及び７の変形例）
　実施例６及び７において、表示制御部２５は、第一の処理部８２２によって生成された高画質画像と入力画像のうち、検者からの指示に応じて選択された画像を表示部５０に表示させることができる。また、表示制御部２５は、検者からの指示に応じて、表示部５０上の表示を撮影画像（入力画像）から高画質画像に切り替えてもよい。すなわち、表示制御部２５は、検者からの指示に応じて、低画質画像の表示を高画質画像の表示に変更してもよい。また、表示制御部２５は、検者からの指示に応じて、高画質画像の表示を低画質画像の表示に変更してもよい。 (Modifications of Embodiments 6 and 7)
In the sixth and seventh embodiments, the display control unit 25 displays, on the display unit 50, an image selected from the high-quality image and the input image generated by the first processing unit 822 in accordance with an instruction from the examiner. Can be done. In addition, the display control unit 25 may switch the display on the display unit 50 from a captured image (input image) to a high-quality image according to an instruction from the examiner. That is, the display control unit 25 may change the display of the low-quality image to the display of the high-quality image according to an instruction from the examiner. In addition, the display control unit 25 may change the display of the high-quality image to the display of the low-quality image according to an instruction from the examiner.
　さらに、第一の処理部８２２が、高画質化エンジン（高画質化モデル）による高画質化処理の開始（高画質化エンジンへの画像の入力）を検者からの指示に応じて実行し、表示制御部２５が、生成された高画質画像を表示部５０に表示させてもよい。これに対し、撮影装置（ＯＣＴ装置１０）によって入力画像が撮影されると、第一の処理部８２２が自動的に高画質化エンジンを用いて入力画像に基づいて高画質画像を生成し、表示制御部２５が、検者からの指示に応じて高画質画像を表示部５０に表示させてもよい。ここで、高画質化エンジンとは、上述した画質向上処理（高画質化処理）を行う学習済モデルを含む。 Further, the first processing unit 822 executes the start of image quality improvement processing (input of an image to the image quality improvement engine) by the image quality improvement engine (image quality improvement model) in accordance with an instruction from the examiner, The display control unit 25 may cause the display unit 50 to display the generated high-quality image. On the other hand, when the input image is captured by the imaging device (OCT device 10), the first processing unit 822 automatically generates a high-quality image based on the input image using the high-quality image engine and displays the image. The control unit 25 may cause the display unit 50 to display a high-quality image according to an instruction from the examiner. Here, the image quality improvement engine includes a learned model that performs the above-described image quality improvement processing (image quality improvement processing).
　なお、これらの処理は解析結果の出力についても同様に行うことができる。すなわち、表示制御部２５は、検者からの指示に応じて、低画質画像の解析結果の表示を高画質画像の解析結果の表示に変更してもよい。また、表示制御部２５は、検者からの指示に応じて、高画質画像の解析結果の表示を低画質画像の解析結果の表示に変更してもよい。さらに、表示制御部２５は、検者からの指示に応じて、低画質画像の解析結果の表示を低画質画像の表示に変更してもよい。また、表示制御部２５は、検者からの指示に応じて、低画質画像の表示を低画質画像の解析結果の表示に変更してもよい。さらに、表示制御部２５は、検者からの指示に応じて、高画質画像の解析結果の表示を高画質画像の表示に変更してもよい。また、表示制御部２５は、検者からの指示に応じて、高画質画像の表示を高画質画像の解析結果の表示に変更してもよい。 処理 Note that these processes can be similarly performed on the output of the analysis result. That is, the display control unit 25 may change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image according to an instruction from the examiner. In addition, the display control unit 25 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image according to an instruction from the examiner. Further, the display control unit 25 may change the display of the analysis result of the low-quality image to the display of the low-quality image according to an instruction from the examiner. In addition, the display control unit 25 may change the display of the low-quality image to the display of the analysis result of the low-quality image according to an instruction from the examiner. Further, the display control unit 25 may change the display of the analysis result of the high-quality image to the display of the high-quality image according to an instruction from the examiner. In addition, the display control unit 25 may change the display of the high-quality image to the display of the analysis result of the high-quality image according to an instruction from the examiner.
　さらに、表示制御部２５は、検者からの指示に応じて、低画質画像の解析結果の表示を低画質画像の他の種類の解析結果の表示に変更してもよい。また、表示制御部２５は、検者からの指示に応じて、高画質画像の解析結果の表示を高画質画像の他の種類の解析結果の表示に変更してもよい。 (4) Further, the display control unit 25 may change the display of the analysis result of the low-quality image to the display of another type of analysis result of the low-quality image according to an instruction from the examiner. In addition, the display control unit 25 may change the display of the analysis result of the high-quality image to the display of another type of analysis result of the high-quality image according to an instruction from the examiner.
　ここで、高画質画像の解析結果の表示は、高画質画像の解析結果を任意の透明度により高画質画像に重畳表示させたものであってもよい。また、低画質画像の解析結果の表示は、低画質画像の解析結果を任意の透明度により低画質画像に重畳表示させたものであってもよい。このとき、解析結果の表示への変更は、例えば、表示されている画像に対して任意の透明度により解析結果を重畳させた状態に変更したものであってもよい。また、解析結果の表示への変更は、例えば、解析結果と画像とを任意の透明度によりブレンド処理して得た画像（例えば、二次元マップ）の表示への変更であってもよい。 Here, the analysis result of the high-quality image may be displayed by superimposing and displaying the analysis result of the high-quality image on the high-quality image with any transparency. The display of the analysis result of the low-quality image may be a display in which the analysis result of the low-quality image is superimposed and displayed on the low-quality image with arbitrary transparency. At this time, the display of the analysis result may be changed, for example, to a state where the analysis result is superimposed on the displayed image with arbitrary transparency. The change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.
　なお、本変形例では、第一の処理部８２２が高画質化処理に関する学習済モデル（高画質化モデル）を用いて断層画像の画質を改善した高画質画像を生成した。しかしながら、高画質化モデルを用いて高画質画像を生成する構成要素は第一の処理部８２２に限られない。例えば、第一の処理部８２２とは別の第三の処理部を設け、第三の処理部が高画質化モデルを用いて高画質画像を生成してもよい。この場合、第三の処理部や高画質化モデルは、ＣＰＵやＭＰＵ、ＧＰＵ、ＦＰＧＡ等のプロセッサーによって実行されるソフトウェアモジュール等で構成されてもよいし、ＡＳＩＣ等の特定の機能を果たす回路等によって構成されてもよい。 In this modification, the first processing unit 822 generates a high-quality image in which the image quality of the tomographic image is improved by using a learned model (high-quality model) related to the high-quality processing. However, a component that generates a high-quality image using the high-quality model is not limited to the first processing unit 822. For example, a third processing unit different from the first processing unit 822 may be provided, and the third processing unit may generate a high quality image using the high quality image model. In this case, the third processing unit and the high image quality model may be configured by a software module or the like executed by a processor such as a CPU, an MPU, a GPU, or an FPGA, or a circuit that performs a specific function such as an ASIC. May be configured.
　また、実施例７では、表示画面のボタン２２２０のアクティブ状態に応じて、高画質化モデルを用いた高画質化処理が行われた画像が表示された。これに対し、ボタン２２２０のアクティブ状態に応じて、学習済モデルを用いた画像セグメンテーション処理の結果を用いた解析値が表示されるように構成してもよい。この場合、例えば、ボタン２２２０が非アクティブ状態（学習済モデルを用いた画像セグメンテーション処理が非選択状態）の場合には、表示制御部２５は、第二の処理部８２３が行った画像セグメンテーション処理の結果を用いた解析結果を表示部５０に表示させる。これに対し、ボタン２２２０がアクティブ状態にされると、表示制御部２５は、第一の処理部８２２が単独で又は第一の処理部８２２及び第二の処理部８２３が行った画像セグメンテーション処理の結果を用いた解析結果を表示部５０に表示させる。 {Circle around (7)} In the seventh embodiment, the image on which the image quality improvement processing using the image quality enhancement model has been performed is displayed according to the active state of the button 2220 on the display screen. On the other hand, according to the active state of the button 2220, an analysis value using the result of the image segmentation processing using the learned model may be displayed. In this case, for example, when the button 2220 is in an inactive state (the image segmentation process using the learned model is in a non-selected state), the display control unit 25 executes the image segmentation process performed by the second processing unit 823. An analysis result using the result is displayed on the display unit 50. On the other hand, when the button 2220 is activated, the display control unit 25 performs the image segmentation processing performed by the first processing unit 822 alone or by the first processing unit 822 and the second processing unit 823. An analysis result using the result is displayed on the display unit 50.
　このような構成では、学習済モデルを用いない画像セグメンテーション処理の結果を用いた解析結果と、学習済モデルを用いた画像セグメンテーション処理の結果を用いた解析結果が、ボタンのアクティブ状態に応じて切り替えて表示される。これらの解析結果は、それぞれ学習済モデルによる処理とルールベースによる画像処理の結果に基づくため、両結果には差異が生じる場合がある。そのため、これらの解析結果を切り替えて表示させることで、検者は両者を対比し、より納得できる解析結果を診断に用いることができる。 In such a configuration, the analysis result using the result of the image segmentation processing using the learned model and the analysis result using the result of the image segmentation processing using the learned model are switched according to the active state of the button. Is displayed. Since these analysis results are based on the results of the processing by the learned model and the image processing by the rule base, there may be a difference between the two results. Therefore, by switching and displaying these analysis results, the examiner can compare the two and use the analysis results that are more satisfactory for the diagnosis.
　なお、画像セグメンテーション処理が切り替えられた際には、例えば、表示される画像が断層画像である場合には、層毎に解析された層厚の数値が切り替えられて表示されてよい。また、例えば、層毎に色やハッチングパターン等で分けられた断層画像が表示される場合には、画像セグメンテーション処理の結果に応じて層の形状が変化した断層画像が切り替えられて表示されてよい。さらに、解析結果として厚みマップが表示される場合には、厚みを示す色が画像セグメンテーション処理の結果に応じて変化した厚みマップが表示されてよい。また、高画質化処理を指定するボタンと学習済モデルを用いた画像セグメンテーション処理を指定するボタンは別々に設けられてもよいし、いずれか一方のも設けられてもよいし、両方のボタンを一つのボタンとして設けてもよい。 When the image segmentation process is switched, for example, when the displayed image is a tomographic image, the numerical value of the layer thickness analyzed for each layer may be switched and displayed. Further, for example, when a tomographic image divided by a color, a hatching pattern, or the like is displayed for each layer, a tomographic image in which the shape of the layer is changed according to the result of the image segmentation processing may be switched and displayed. . Furthermore, when a thickness map is displayed as an analysis result, a thickness map in which the color indicating the thickness has changed according to the result of the image segmentation process may be displayed. Further, a button for specifying the image quality improvement processing and a button for specifying the image segmentation processing using the learned model may be provided separately, one of them may be provided, and both buttons may be provided. It may be provided as one button.
　また、画像セグメンテーション処理の切り替えは、上述の高画質化処理の切り替えと同様に、データベースに保存（記録）されている情報に基づいて行われてもよい。なお、画面遷移時に処理についても、画像セグメンテーション処理の切り替えは、上述の高画質化処理の切り替えと同様に行われてよい。 The switching of the image segmentation process may be performed based on information stored (recorded) in the database, similarly to the switching of the image quality improvement process described above. It should be noted that the switching of the image segmentation process may be performed in the same manner as the above-described switching of the image quality improvement process also at the time of the screen transition.
　実施例１乃至７では、取得部２１は、ＯＣＴ装置１０で取得された干渉信号や断層画像生成部２２１で生成された三次元断層データを取得した。しかしながら、取得部２１がこれらの信号やデータを取得する構成はこれに限られない。例えば、取得部２１は、画像処理装置２０とＬＡＮ、ＷＡＮ、又はインターネット等を介して接続されるサーバや撮影装置からこれらの信号を取得してもよい。この場合、撮影に関する処理を省略し、撮影済みの三次元の断層データを取得することができる。そして、ステップＳ３０４やステップＳ９０４等で境界検出処理を行うことができる。そのため、断層情報の取得から正面画像や厚みマップ等の表示までの一連の処理時間を短くすることができる。 In the first to seventh embodiments, the obtaining unit 21 obtains the interference signal obtained by the OCT apparatus 10 and the three-dimensional tomographic data generated by the tomographic image generating unit 221. However, the configuration in which the acquisition unit 21 acquires these signals and data is not limited to this. For example, the acquisition unit 21 may acquire these signals from a server or an imaging device connected to the image processing apparatus 20 via a LAN, a WAN, the Internet, or the like. In this case, it is possible to omit the processing related to imaging, and acquire three-dimensional tomographic data that has been captured. Then, the boundary detection processing can be performed in step S304, step S904, or the like. Therefore, a series of processing times from acquisition of tomographic information to display of a front image, a thickness map, and the like can be shortened.
　なお、処理部２２２及び第一の処理部８２２が用いる画像セグメンテーション用の学習済モデルや高画質化用の学習済モデルは、画像処理装置２０，８０，１５２に設けられることができる。学習済モデルは、例えば、ＣＰＵや、ＭＰＵ、ＧＰＵ、ＦＰＧＡ等のプロセッサーによって実行されるソフトウェアモジュール等で構成されてもよいし、ＡＳＩＣ等の特定の機能を果たす回路等によって構成されてもよい。また、これら学習済モデルは、画像処理装置２０，８０，１５２と接続される別のサーバの装置等に設けられてもよい。この場合には、画像処理装置２０，８０，１５２は、インターネット等の任意のネットワークを介して学習済モデルを備えるサーバ等に接続することで、学習済モデルを用いることができる。ここで、学習済モデルを備えるサーバは、例えば、クラウドサーバや、フォグサーバ、エッジサーバ等であってよい。 Note that the learned model for image segmentation and the learned model for improving image quality used by the processing unit 222 and the first processing unit 822 can be provided in the image processing devices 20, 80, and 152. The learned model may be configured by, for example, a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may be configured by a circuit that performs a specific function such as an ASIC. Further, these learned models may be provided in a device of another server or the like connected to the image processing devices 20, 80, 152. In this case, the image processing apparatuses 20, 80, and 152 can use the learned model by connecting to a server or the like having the learned model via an arbitrary network such as the Internet. Here, the server including the learned model may be, for example, a cloud server, a fog server, an edge server, or the like.
　また、実施例２乃至７では、ラベル画像として画素毎にラベル付けされたラベル画像について説明したが、ラベル画像として領域毎にラベル付けされたラベル画像を用いてもよい。 In the second to seventh embodiments, a label image labeled for each pixel has been described as a label image, but a label image labeled for each region may be used as a label image.
　なお、ＯＣＴ装置１０の構成は、上記の構成に限られず、ＯＣＴ装置１０に含まれる構成の一部をＯＣＴ装置１０と別体の構成としてもよい。また、ボタンなどのユーザーインターフェースや表示のレイアウトは上記で示したものに限定されるものではない。 The configuration of the OCT apparatus 10 is not limited to the above configuration, and a part of the configuration included in the OCT apparatus 10 may be configured separately from the OCT apparatus 10. The layout of the user interface such as buttons and the display are not limited to those described above.
　上述の実施例１乃至７及びその変形例によれば、疾患や部位等によらず網膜層の境界検出を行うことができる。 According to the first to seventh embodiments and the modifications thereof, it is possible to detect the boundary of the retinal layer irrespective of a disease or a part.
（実施例８）
　医療分野においては、被検者の疾患を特定したり、疾患の程度を観察したりするために、様々な撮影装置によって取得された画像を利用した画像診断が実施されている。撮影装置の種類には、例えば放射線科分野では、Ｘ線撮影装置、Ｘ線コンピュータ断層撮影装置（ＣＴ）、磁気共鳴イメージング装置（ＭＲＩ）、及び陽電子放出断層撮影装置（ＰＥＴ）などがある。また、例えば眼科分野では、眼底カメラ、走査型レーザ検眼鏡（ＳＬＯ）、光コヒーレンストモグラフィ（ＯＣＴ）装置、及びＯＣＴアンギオグラフィ（ＯＣＴＡ）装置などがある。 (Example 8)
2. Description of the Related Art In the medical field, image diagnosis using images acquired by various imaging devices is performed to identify a disease of a subject or to observe the degree of the disease. Examples of the type of imaging apparatus include, for example, an X-ray imaging apparatus, an X-ray computed tomography apparatus (CT), a magnetic resonance imaging apparatus (MRI), and a positron emission tomography apparatus (PET) in the field of radiology. In the field of ophthalmology, for example, there are a fundus camera, a scanning laser ophthalmoscope (SLO), an optical coherence tomography (OCT) device, an OCT angiography (OCTA) device, and the like.
　画像診断は、基本的には医療従事者が画像に描出された病変等を観察することによって実施されるが、近年では画像解析技術の向上によって診断に役立つ様々な情報が得られるようになった。例えば、画像解析をすることによって、見落とす可能性のある小さな病変を検出して医療従事者を支援したり、病変の形状や体積について定量的な計測を行ったり、さらには医療従事者の観察なしに疾患を特定したりすることができるようになった。 Image diagnosis is basically performed by medical staff observing lesions and the like drawn in the image, but in recent years, various information useful for diagnosis has been obtained by improving image analysis technology. . For example, image analysis helps detect small lesions that may be overlooked, assists healthcare professionals, performs quantitative measurements on the shape and volume of lesions, and does not require observation by healthcare professionals And to identify the disease.
　なお、画像解析の手法には様々なものがあるが、画像セグメンテーション処理と呼ばれる、画像に描出された臓器や病変といった領域を特定するための処理は、多くの画像解析の手法を実施するにあたって、必要となる処理である。以下において、簡略化のため画像セグメンテーション処理を単にセグメンテーション処理ともいう。 Although there are various image analysis methods, a process called image segmentation processing for identifying a region such as an organ or a lesion depicted in an image is performed when performing many image analysis methods. This is a necessary process. Hereinafter, the image segmentation process is also simply referred to as a segmentation process for simplification.
　従来の画像セグメンテーション処理は、特許文献１のように、対象の臓器や病変に関する医学的知識や画像特性に基づいた画像処理アルゴリズムによって実施される。しかしながら、実際の医療現場において、撮影装置から取得される画像は、被検者の病態や撮影装置の撮影環境、撮影者の技術不足等の、様々な要因により綺麗に撮影できていないことがある。そのため、従来の画像セグメンテーション処理にとって、対象の臓器や病変が想定通りに描出されておらず、精度よく特定の領域を抽出できないことがあった。 The conventional image segmentation process is performed by an image processing algorithm based on medical knowledge and image characteristics regarding a target organ or lesion, as disclosed in Patent Document 1. However, in an actual medical practice, an image acquired from an imaging device may not be able to be captured neatly due to various factors such as the condition of the subject, the imaging environment of the imaging device, lack of skill of the photographer, and the like. . Therefore, in the conventional image segmentation processing, a target organ or lesion is not depicted as expected, and a specific region may not be accurately extracted.
　具体的には、例えば、網膜層の消失、出血、白斑、及び新生血管の発生等のある疾患眼をＯＣＴ装置で撮影して取得された画像では、網膜層の形状の描出が不規則となることがあった。このような場合には、画像セグメンテーション処理の一種である網膜層の領域検出処理において、誤検出が発生することがあった。 Specifically, for example, in an image obtained by photographing a diseased eye such as loss of the retinal layer, hemorrhage, vitiligo, and occurrence of new blood vessels with an OCT apparatus, the shape of the retinal layer is irregularly drawn. There was something. In such a case, erroneous detection may occur in the retinal layer area detection processing, which is a type of image segmentation processing.
　下記実施例８乃至１９の目的の一つは、従来の画像セグメンテーション処理よりも精度の高い画像セグメンテーション処理を実施できる医用画像処理装置、医用画像処理方法及びプログラムを提供することである。 One of the objects of the following eighth to nineteenth embodiments is to provide a medical image processing apparatus, a medical image processing method, and a program capable of performing image segmentation processing with higher accuracy than conventional image segmentation processing.
＜用語の説明＞
　ここで、本開示において用いられる用語について説明する。 <Explanation of terms>
Here, terms used in the present disclosure will be described.
　本明細書におけるネットワークでは、各装置は有線又は無線の回線で接続されてよい。ここで、ネットワークにおける各装置を接続する回線は、例えば、専用回線、ローカルエリアネットワーク（以下、ＬＡＮと表記）回線、無線ＬＡＮ回線、インターネット回線、Ｗｉ－Ｆｉ（登録商標）、及びＢｌｕｅｔｏｏｔｈ（登録商標）等を含む。 In the network in this specification, each device may be connected by a wired or wireless line. Here, a line connecting each device in the network includes, for example, a dedicated line, a local area network (hereinafter referred to as LAN) line, a wireless LAN line, an Internet line, Wi-Fi (registered trademark), and Bluetooth (registered trademark). ).
　医用画像処理装置は、相互に通信が可能な２以上の装置によって構成されてもよいし、単一の装置によって構成されてもよい。また、医用画像処理装置の各構成要素は、ＣＰＵやＭＰＵ、ＧＰＵ、ＦＰＧＡ等のプロセッサーによって実行されるソフトウェアモジュールにより構成されてよい。また、当該各構成要素は、ＡＳＩＣ等の特定の機能を果たす回路等によって構成されてもよい。また、他の任意のハードウェアと任意のソフトウェアとの組み合わせにより構成されてもよい。 The medical image processing device may be configured by two or more devices that can communicate with each other, or may be configured by a single device. Each component of the medical image processing apparatus may be configured by a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA. In addition, each of the components may be configured by a circuit or the like that performs a specific function such as an ASIC. Also, it may be configured by a combination of any other hardware and any software.
　また、医用画像処理装置又は医用画像処理方法によって処理される医用画像は、ＯＣＴ装置を用いて取得された被検者の断層画像である。ここで、ＯＣＴ装置としては、タイムドメインＯＣＴ（ＴＤ－ＯＣＴ）装置やフーリエドメインＯＣＴ（ＦＤ－ＯＣＴ）装置を含んでよい。また、フーリエドメインＯＣＴ装置はスペクトラルドメインＯＣＴ（ＳＤ－ＯＣＴ）装置や波長掃引型ＯＣＴ（ＳＳ－ＯＣＴ）装置を含んでよい。また、ＯＣＴ装置として、波面補償光学系を用いた波面補償ＯＣＴ（ＡＯ－ＯＣＴ）装置や、被検者に照射される測定光をライン状に形成したラインＯＣＴ装置、当該測定光を面状に形成したフルフィールドＯＣＴ装置等の任意のＯＣＴ装置を含んでよい。 The medical image processed by the medical image processing apparatus or the medical image processing method is a tomographic image of the subject acquired using the OCT apparatus. Here, the OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. Further, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a wavelength sweep type OCT (SS-OCT) device. As an OCT apparatus, a wavefront compensation OCT (AO-OCT) apparatus using a wavefront compensation optical system, a line OCT apparatus in which measurement light irradiated to a subject is formed in a line, and the measurement light is formed in a plane Any OCT device, such as a formed full-field OCT device, may be included.
　医用画像には、被検者の眼（被検眼）の断層画像が含まれる。被検眼の断層画像としては、被検眼の後眼部における網膜等の断層画像に限られず、被検眼の前眼部や眼房等の断層画像が含まれる。また、ＯＣＴ装置を内視鏡等に用いる場合には、被検者の皮膚や臓器の断層画像を以下の実施例による医用画像処理装置又は医用画像処理方法の処理対象となる医用画像としてもよい。 The medical image includes a tomographic image of the subject's eye (the subject's eye). The tomographic image of the subject's eye is not limited to a tomographic image of the retina and the like in the posterior segment of the subject's eye, and includes a tomographic image of the anterior eye of the subject's eye and the eye chamber. When the OCT apparatus is used for an endoscope or the like, a tomographic image of the skin or the organ of the subject may be used as a medical image to be processed by the medical image processing apparatus or the medical image processing method according to the following embodiments. .
　画像管理システムは、ＯＣＴ装置等の撮影装置によって撮影された画像や画像処理された画像を受信して保存する装置及びシステムである。また、画像管理システムは、接続された装置の要求に応じて画像を送信したり、保存された画像に対して画像処理を行ったり、画像処理の要求を他の装置に要求したりすることができる。画像管理システムとしては、例えば、画像保存通信システム（ＰＡＣＳ）を含むことができる。特に、下記実施例に係る画像管理システムは、受信した画像とともに関連付けられた被検者の情報や撮影時間などの各種情報も保存可能なデータベースを備える。また、画像管理システムはネットワークに接続され、他の装置からの要求に応じて、画像を送受信したり、画像を変換したり、保存した画像に関連付けられた各種情報を送受信したりすることができる。 The image management system is a device and a system that receives and stores an image captured by an imaging device such as an OCT device or an image processed image. In addition, the image management system can transmit an image in response to a request from a connected device, perform image processing on a stored image, and request an image processing request to another device. it can. The image management system can include, for example, an image storage and communication system (PACS). In particular, the image management system according to the embodiment described below includes a database that can store various information such as information on a subject and an imaging time associated with the received image. In addition, the image management system is connected to a network, and can transmit and receive images, convert images, and transmit and receive various types of information associated with stored images in response to requests from other devices. .
　撮影条件とは、撮影装置によって取得された画像の撮影時の様々な情報である。撮影条件は、例えば、撮影装置に関する情報、撮影が実施された施設に関する情報、撮影に係る検査の情報、撮影者に関する情報、及び被検者に関する情報等を含む。また、撮影条件は、例えば、撮影日時、撮影部位名、撮影領域、撮影画角、撮影方式、画像の解像度や階調、画像サイズ、適用された画像フィルタ、及び画像のデータ形式に関する情報等を含む。なお、撮影領域には、特定の撮影部位からずれた周辺の領域や複数の撮影部位を含んだ領域等が含まれることができる。また、撮影方式は、スペクトラルドメイン方式や波長掃引方式等のＯＣＴの任意の撮影方式を含んでよい。 The photographing conditions are various information at the time of photographing the image acquired by the photographing device. The imaging conditions include, for example, information on the imaging device, information on the facility where the imaging was performed, information on the inspection related to the imaging, information on the photographer, information on the subject, and the like. The imaging conditions include, for example, information on the date and time of imaging, the name of the imaging region, the imaging region, the imaging angle of view, the imaging method, the resolution and gradation of the image, the image size, the applied image filter, and the data format of the image. Including. Note that the imaging region may include a peripheral region shifted from a specific imaging region, an region including a plurality of imaging regions, and the like. In addition, the imaging method may include any OCT imaging method such as a spectral domain method or a wavelength sweep method.
　撮影条件は、画像を構成するデータ構造中に保存されていたり、画像とは別の撮影条件データとして保存されていたり、撮影装置に関連するデータベースや画像管理システムに保存されたりすることができる。そのため、撮影条件は、撮影装置の撮影条件の保存手段に対応した手順により取得することができる。具体的には、撮影条件は、例えば、撮影装置が出力した画像のデータ構造を解析したり、画像に対応する撮影条件データを取得したり、撮影装置に関連するデータベースから撮影条件を取得するためのインターフェースにアクセスする等により取得される。 (4) The shooting conditions can be stored in a data structure constituting the image, stored as shooting condition data different from the image, or stored in a database or an image management system related to the shooting device. Therefore, the photographing condition can be acquired by a procedure corresponding to the photographing condition storage unit of the photographing device. Specifically, the shooting conditions are, for example, to analyze the data structure of an image output by the shooting device, to obtain shooting condition data corresponding to the image, and to obtain shooting conditions from a database related to the shooting device. It is obtained by accessing the interface of.
　なお、撮影装置によっては、保存されていない等の理由で取得できない撮影条件も存在する。例えば、撮影装置に特定の撮影条件を取得したり保存したりする機能が無い、又はそのような機能が無効にされている場合である。また、例えば、撮影装置や撮影に関係の無い撮影条件であるとして保存しないようになっている場合もある。さらに、例えば、撮影条件が隠蔽されていたり、暗号化されていたり、権利が無いと取得できないようになっていたりする場合等もある。ただし、保存されていない撮影条件であっても取得できる場合がある。例えば、画像解析を実施することによって、撮影部位名や撮影領域を特定することができる。 Some shooting conditions may not be available for some shooting devices because they are not stored. For example, there is a case where the photographing apparatus does not have a function of acquiring or storing a specific photographing condition, or such a function is invalidated. Further, for example, there is a case where the photographing condition is not stored because the photographing condition is not related to the photographing device or photographing. Further, for example, there are cases where the shooting conditions are hidden, encrypted, or cannot be obtained without the right. However, it may be possible to acquire even shooting conditions that are not stored. For example, by performing image analysis, it is possible to specify an imaging part name and an imaging region.
　領域ラベル画像とは、画素毎に領域のラベルが付されたラベル画像をいう。具体的には、図２４に示すように、撮影装置によって取得された画像Ｉｍ２４１０に描出されている領域群のうち、任意の領域を特定可能な画素値（以下、領域ラベル値）群によって分けている画像Ｉｍ２４２０のことである。ここで、特定される任意の領域には関心領域（ＲＯＩ：Ｒｅｇｉｏｎ　Ｏｆ　Ｉｎｔｅｒｅｓｔ）や関心体積（ＶＯＩ：Ｖｏｌｕｍｅ　Ｏｆ　Ｉｎｔｅｒｅｓｔ）等が含まれる。 A region label image refers to a label image in which a region is labeled for each pixel. Specifically, as shown in FIG. 24, of the group of regions depicted in the image Im2410 obtained by the imaging device, an arbitrary region is divided by a group of pixel values (hereinafter, region label values) that can be specified. Image Im2420. Here, the specified arbitrary region includes a region of interest (ROI: Region \ Interest), a volume of interest (VOI: Volume \ Of \ Interest), and the like.
　画像Ｉｍ２４２０から任意の領域ラベル値を持つ画素の座標群を特定すると、画像Ｉｍ２４１０中において対応する網膜層等の領域を描出している画素の座標群を特定できる。具体的には、例えば、網膜を構成する神経節細胞層を示す領域ラベル値が１である場合、画像Ｉｍ２４２０の画素群のうち画素値が１である座標群を特定し、画像Ｉｍ２４１０から該座標群に対応する画素群を抽出する。これにより、画像Ｉｍ２４１０における神経節細胞層の領域を特定できる。 特定 If the coordinate group of the pixel having an arbitrary region label value is specified from the image Im2420, the coordinate group of the pixel depicting the corresponding region such as the retina layer in the image Im2410 can be specified. Specifically, for example, when the region label value indicating the ganglion cell layer constituting the retina is 1, a coordinate group whose pixel value is 1 is specified from the pixel group of the image Im2420, and the coordinate group is determined from the image Im2410. Extract a pixel group corresponding to the group. Thereby, the area of the ganglion cell layer in the image Im2410 can be specified.
　なお、一部の実施例において、領域ラベル画像に対する縮小又は拡大処理を実施する処理が含まれる。このとき、領域ラベル画像の縮小又は拡大に用いる画像補完処理手法は、未定義の領域ラベル値や対応する座標に存在しないはずの領域ラベル値を誤って生成しないような、最近傍法等を使うものとする。 Note that some embodiments include a process of performing a reduction or enlargement process on an area label image. At this time, the image complement processing method used for reducing or enlarging the region label image uses a nearest neighbor method or the like that does not erroneously generate an undefined region label value or a region label value that should not exist at the corresponding coordinates. Shall be.
　画像セグメンテーション処理とは、画像に描出された臓器や病変といった、ＲＯＩやＶＯＩと呼ばれる領域を、画像診断や画像解析に利用するために特定する処理のことである。例えば、画像セグメンテーション処理によれば、後眼部を撮影対象としたＯＣＴの撮影によって取得された画像から、網膜を構成する層群の領域群を特定することができる。なお、画像に特定すべき領域が描出されていなければ特定される領域の数は０である。また、画像に特定すべき複数の領域群が描出されていれば、特定される領域の数は複数であってもよいし、或いは、該領域群を含むように囲む領域１つであってもよい。 The image segmentation process is a process of specifying an area called an ROI or a VOI, such as an organ or a lesion depicted in an image, for use in image diagnosis or image analysis. For example, according to the image segmentation process, it is possible to specify a region group of a layer group constituting the retina from an image acquired by OCT imaging of the posterior segment of the eye. It should be noted that the number of specified regions is 0 if no region to be specified is depicted in the image. Further, if a plurality of region groups to be specified are depicted in the image, the number of specified regions may be plural or one region surrounding the region group may be included. Good.
　特定された領域群は、その他の処理において利用可能な情報として出力される。具体的には、例えば、特定された領域群のそれぞれを構成する画素群の座標群を数値データ群として出力することができる。また、例えば、特定された領域群のそれぞれを含む矩形領域や楕円領域、長方体領域、楕円体領域等を示す座標群を数値データ群として出力することもできる。さらに、例えば、特定された領域群の境界にあたる直線や曲線、平面、又は曲面等を示す座標群を数値データ群として出力することもできる。また、例えば、特定された領域群を示す領域ラベル画像を出力することもできる。 領域 The specified area group is output as information that can be used in other processes. Specifically, for example, a coordinate group of a pixel group constituting each of the specified region groups can be output as a numerical data group. Further, for example, a coordinate group indicating a rectangular area, an elliptical area, a rectangular parallelepiped area, an elliptical area, and the like including each of the specified area groups can be output as a numerical data group. Further, for example, a coordinate group indicating a straight line, a curve, a plane, a curved surface, or the like, which is a boundary of the specified region group, can be output as a numerical data group. Also, for example, an area label image indicating the specified area group can be output.
　なお、以下において、画像セグメンテーション処理の精度が高いと表現したり、精度の高い領域ラベル画像と表現したりする場合は、領域を正しく特定できている割合が高いことを指す。また、逆に、画像セグメンテーション処理の精度が低いと表現する場合は、領域を誤って特定している割合が高いことを指す。 In the following description, when expressing the image segmentation processing with high accuracy or expressing it as a highly accurate region label image, it means that the ratio of correctly specifying the region is high. Conversely, when expressing that the accuracy of the image segmentation process is low, it indicates that the ratio of erroneously specifying the region is high.
　領域ラベル無し画像とは、領域ラベル画像の一種であり、画像診断や画像解析に利用するためのＲＯＩやＶＯＩに対応する情報が含まれていない領域ラベル画像である。具体的に、一例として、画像解析に利用するために医用画像に描出された網膜を構成する神経節細胞層の領域を知りたい場合を説明する。 A region label-less image is a type of region label image, and is a region label image that does not include information corresponding to ROIs and VOIs used for image diagnosis and image analysis. Specifically, as an example, a case will be described in which it is desired to know a region of a ganglion cell layer constituting a retina depicted in a medical image for use in image analysis.
　ここで、神経節細胞層の領域を示す領域ラベル値は１であり、それ以外の部分の領域を示す領域ラベル値は０であるとする。ある医用画像に対応する領域ラベル画像を、画像セグメンテーション処理等によって生成した際に、医用画像に神経節細胞層が描出されていない場合、領域ラベル画像のすべての画素値は０となる。この場合、当該領域ラベル画像は、画像解析に利用するための神経節細胞層のＲＯＩに対応する領域ラベル値１である領域が領域ラベル画像に無いため、領域ラベル無し画像である。なお、設定や実装形態によっては、領域ラベル無し画像は、画像ではなく、画像と同様の情報を持つ座標群を示す数値データ群等であってもよい。 Here, it is assumed that the region label value indicating the region of the ganglion cell layer is 1, and the region label value indicating the other region is 0. When a region label image corresponding to a certain medical image is generated by an image segmentation process or the like and no ganglion cell layer is drawn in the medical image, all the pixel values of the region label image become 0. In this case, the region label image is an image without a region label because there is no region having a region label value 1 corresponding to the ROI of the ganglion cell layer to be used for image analysis in the region label image. It should be noted that, depending on the setting and the implementation form, the image without the region label may be a numerical data group or the like indicating a coordinate group having the same information as the image instead of the image.
　ここで、機械学習モデルとは、機械学習アルゴリズムによる学習モデルをいう。機械学習の具体的なアルゴリズムとしては、最近傍法、ナイーブベイズ法、決定木、サポートベクターマシンなどが挙げられる。また、ニューラルネットワークを利用して、学習するための特徴量、結合重み付け係数を自ら生成する深層学習（ディープラーニング）も挙げられる。適宜、上記アルゴリズムのうち利用できるものを用いて実施例に係る学習モデルに適用することができる。学習済モデルとは、任意の機械学習アルゴリズムに従った機械学習モデルに対して、事前に適切な教師データ（学習データ）を用いてトレーニング（学習）を行ったモデルである。ただし、学習済モデルは、それ以上の学習を行わないものではなく、追加の学習を行うこともできるものとする。教師データは、一つ以上の、入力データと出力データとのペア群で構成される。なお、教師データを構成するペア群の入力データと出力データの形式や組み合わせは、一方が画像で他方が数値であったり、一方が複数の画像群で構成され他方が文字列であったり、双方が画像であったりする等、所望の構成に適したものであってよい。 Here, the machine learning model refers to a learning model based on a machine learning algorithm. Specific algorithms for machine learning include a nearest neighbor method, a naive Bayes method, a decision tree, a support vector machine, and the like. In addition, deep learning (deep learning) in which a feature amount for learning and a connection weighting coefficient are generated by themselves using a neural network is also included. As appropriate, any of the above algorithms that can be used can be applied to the learning model according to the embodiment. The learned model is a model in which training (learning) has been performed in advance on a machine learning model according to an arbitrary machine learning algorithm using appropriate teacher data (learning data). However, it is assumed that the learned model does not perform any further learning and can perform additional learning. The teacher data is composed of one or more pairs of input data and output data. Note that the format and combination of input data and output data of a pair group that constitutes teacher data may be such that one is an image and the other is a numerical value, one is a plurality of image groups and the other is a character string, May be an image suitable for a desired configuration.
　具体的には、例えば、ＯＣＴによって取得された画像と、該画像に対応する撮影部位ラベルとのペア群によって構成された教師データ（以下、第一の教師データ）が挙げられる。なお、撮影部位ラベルは部位を表すユニークな数値や文字列である。また、その他の教師データの例として、後眼部を撮影対象としたＯＣＴの撮影によって取得された画像と、該画像に対応する網膜層の領域ラベル画像とのペア群によって構成されている教師データ（以下、第二の教師データ）が挙げられる。さらに、その他の教師データの例として、ＯＣＴの通常撮影によって取得されたノイズの多い低画質画像と、ＯＣＴにより複数回撮影して高画質化処理した高画質画像とのペア群によって構成されている教師データ（以下、第三の教師データ）が挙げられる。 Specifically, for example, there is teacher data (hereinafter, first teacher data) configured by a pair group of an image acquired by OCT and an imaging part label corresponding to the image. Note that the imaging region label is a unique numerical value or character string representing the region. Further, as another example of teacher data, teacher data constituted by a pair group of an image acquired by OCT imaging of the posterior segment of the eye and an area label image of a retinal layer corresponding to the image. (Hereinafter, second teacher data). Further, as another example of the teacher data, a pair group of a low-quality image with a lot of noise obtained by normal imaging of OCT and a high-quality image obtained by performing multiple image capturing and high-quality processing by OCT is configured. Teacher data (hereinafter, third teacher data).
　学習済モデルに入力データを入力すると、該学習済モデルの設計に従った出力データが出力される。学習済モデルは、例えば、教師データを用いてトレーニングされた傾向に従って、入力データに対応する可能性の高い出力データを出力する。また、学習済モデルは、例えば、教師データを用いてトレーニングされた傾向に従って、出力データの種類のそれぞれについて、入力データに対応する可能性を数値として出力する等を行うことができる。 (4) When input data is input to the learned model, output data according to the design of the learned model is output. The trained model outputs output data having a high possibility of corresponding to the input data, for example, according to the tendency trained using the teacher data. Further, the trained model can output, for example, as a numerical value the possibility corresponding to the input data for each type of output data according to the tendency trained using the teacher data.
　具体的には、例えば、第一の教師データでトレーニングされた学習済モデルにＯＣＴによって取得された画像を入力すると、学習済モデルは、該画像に撮影されている撮影部位の撮影部位ラベルを出力したり、撮影部位ラベル毎の確率を出力したりする。また、例えば第二の教師データによってトレーニングされた学習済モデルに後眼部を撮影対象としたＯＣＴの撮影によって取得された網膜層を描出する画像を入力すると、学習済モデルは、該画像に描出された網膜層に対する領域ラベル画像を出力する。さらに、例えば、第三の教師データでトレーニングされた学習済モデルにＯＣＴの通常撮影によって取得されたノイズの多い低画質画像を入力すると、学習済モデルは、ＯＣＴにより複数回撮影して高画質化処理された画像相当の高画質画像を出力する。 Specifically, for example, when an image acquired by OCT is input to a trained model trained with the first teacher data, the trained model outputs an imaging region label of the imaging region photographed in the image. Or output the probability for each radiographic part label. Further, for example, when an image depicting a retinal layer obtained by OCT imaging of the posterior segment is input to a learned model trained by the second teacher data, the learned model is rendered in the image. An area label image for the retinal layer is output. Further, for example, when a low-quality image with much noise acquired by normal OCT imaging is input to the learned model trained with the third teacher data, the learned model is photographed a plurality of times by OCT to improve the image quality. A high quality image equivalent to the processed image is output.
　機械学習アルゴリズムは、畳み込みニューラルネットワーク（ＣＮＮ）等のディープラーニングに関する手法を含む。ディープラーニングに関する手法においては、ニューラルネットワークを構成する層群やノード群に対するパラメータの設定が異なると、教師データを用いてトレーニングされた傾向を出力データに再現可能な程度が異なる場合がある。 The machine learning algorithm includes a technique related to deep learning such as a convolutional neural network (CNN). In the method related to deep learning, if the parameter settings for the layers and nodes forming the neural network are different, the degree to which the tendency trained using the teacher data can be reproduced in the output data may be different.
　例えば、第一の教師データを用いたディープラーニングの学習済モデルにおいては、より適切なパラメータが設定されていると、正しい撮影部位ラベルを出力する確率がより高くなる場合がある。また、例えば、第二の教師データを用いる学習済モデルにおいては、より適切なパラメータが設定されていると、より精度の高い領域ラベル画像を出力できる場合がある。さらに、例えば、第三の教師データを用いたディープラーニングの学習済モデルにおいては、より適切なパラメータが設定されていると、より高画質な画像を出力できる場合がある。 For example, in a learned model of deep learning using the first teacher data, if more appropriate parameters are set, the probability of outputting a correct radiographed part label may be higher. Further, for example, in a trained model using the second teacher data, if a more appropriate parameter is set, a more accurate region label image may be output in some cases. Further, for example, in a learned model of deep learning using the third teacher data, if a more appropriate parameter is set, a higher quality image may be output in some cases.
　具体的には、ＣＮＮにおけるパラメータは、例えば、畳み込み層に対して設定される、フィルタのカーネルサイズ、フィルタの数、ストライドの値、及びダイレーションの値、並びに全結合層の出力するノードの数等を含むことができる。なお、パラメータ群やトレーニングのエポック数は、教師データに基づいて、学習済モデルの利用形態に好ましい値に設定することができる。例えば、教師データに基づいて、正しい撮影部位ラベルをより高い確率で出力したり、より精度の高い領域ラベル画像を出力したり、より高画質な画像を出力したりできるパラメータ群やエポック数を設定することができる。 Specifically, the parameters in the CNN are, for example, the filter kernel size, the number of filters, the stride value, and the dilation value set for the convolutional layer, and the number of nodes output from the fully connected layer. Etc. can be included. It should be noted that the parameter group and the number of training epochs can be set to values suitable for the use form of the learned model based on the teacher data. For example, based on teacher data, set a parameter group and epoch number that can output a correct radiographed part label with a higher probability, output a more accurate area label image, and output a higher quality image can do.
　このようなパラメータ群やエポック数の決定方法の一つを例示する。まず、教師データを構成するペア群の７割をトレーニング用とし、残りの３割を評価用としてランダムに設定する。次に、トレーニング用のペア群を用いて学習済モデルのトレーニングを行い、トレーニングの各エポックの終了時に、評価用のペア群を用いてトレーニング評価値を算出する。トレーニング評価値とは、例えば、各ペアを構成する入力データをトレーニング中の学習済モデルに入力したときの出力と、入力データに対応する出力データとを損失関数によって評価した値群の平均値等である。最後に、最もトレーニング評価値が小さくなったときのパラメータ群及びエポック数を、当該学習済モデルのパラメータ群やエポック数として決定する。なお、このように、教師データを構成するペア群をトレーニング用と評価用とに分けてエポック数の決定を行うことによって、学習済モデルがトレーニング用のペア群に対して過学習してしまうことを防ぐことができる。 一 つ One example of such a parameter group or epoch number determination method will be described. First, 70% of the pair group constituting the teacher data is used for training, and the remaining 30% is randomly set for evaluation. Next, the trained model is trained using the training pair group, and at the end of each training epoch, a training evaluation value is calculated using the evaluation pair group. The training evaluation value is, for example, an average value of a group of values obtained by evaluating an output when input data constituting each pair is input to a trained model being trained and output data corresponding to the input data using a loss function. It is. Finally, the parameter group and the number of epochs when the training evaluation value becomes the smallest are determined as the parameter group and the number of epochs of the learned model. In this way, by deciding the number of epochs by dividing the pair group constituting the teacher data into the one for training and the other for evaluation, the trained model may overlearn the pair group for training. Can be prevented.
　画像セグメンテーションエンジンとは、画像セグメンテーション処理を実施し、入力された入力画像に対応する領域ラベル画像を出力するモジュールのことである。入力画像の例としては、ＯＣＴのＢスキャン画像や三次元断層画像（三次元ＯＣＴボリューム画像）等がある。また、領域ラベル画像の例としては、入力画像がＯＣＴのＢスキャン画像である場合における網膜層の各層を示す領域ラベル画像や、入力画像がＯＣＴの三次元断層画像である場合における網膜層の各層を示す三次元領域を示す領域ラベル画像がある。 The image segmentation engine is a module that performs image segmentation processing and outputs an area label image corresponding to the input image that has been input. Examples of the input image include an OCT B-scan image and a three-dimensional tomographic image (three-dimensional OCT volume image). Examples of the region label image include a region label image indicating each layer of the retinal layer when the input image is an OCT B-scan image and a layer label image indicating each layer of the retinal layer when the input image is a three-dimensional tomographic image of OCT. There is an area label image indicating a three-dimensional area indicating.
　下記の実施例における画像セグメンテーション処理手法を構成する画像処理手法では、ディープラーニング等の各種機械学習アルゴリズムに従った学習済モデルを用いた処理を行う。なお、当該画像処理手法は、機械学習アルゴリズムだけでなく、他の既存の任意の処理を併せて行ってもよい。当該画像処理には、例えば、各種画像フィルタ処理、類似画像に対応する領域ラベル画像のデータベースを用いたマッチング処理、基準領域ラベル画像の画像レジストレーション処理、及び知識ベース画像処理等の処理が含まれる。 画像 In the image processing method constituting the image segmentation processing method in the following embodiments, processing using a learned model according to various machine learning algorithms such as deep learning is performed. Note that the image processing method may be performed not only with a machine learning algorithm but also with other existing arbitrary processing. The image processing includes, for example, various image filtering processes, a matching process using a database of region label images corresponding to similar images, an image registration process of a reference region label image, and a knowledge base image process. .
　特に、入力画像として入力された二次元の画像Ｉｍ２５１０を画像セグメンテーション処理して領域ラベル画像Ｉｍ２５２０を生成する畳み込みニューラルネットワーク（ＣＮＮ）の例として、図２５に示す構成２５００がある。当該ＣＮＮの構成２５００は、入力値群を加工して出力する処理を担う、複数の層群が含まれる。なお、当該構成２５００に含まれる層の種類としては、図２５に示すように、畳み込み（Ｃｏｎｖｏｌｕｔｉｏｎ）層、ダウンサンプリング（Ｄｏｗｎｓａｍｐｌｉｎｇ）層、アップサンプリング（Ｕｐｓａｍｐｌｉｎｇ）層、及び合成（Ｍｅｒｇｅｒ）層がある。なお、本実施例で用いるＣＮＮの構成２５００は、実施例１で述べたＣＮＮの構成６０１と同様に、Ｕ－ｎｅｔ型の機械学習モデルである。 In particular, a configuration 2500 shown in FIG. 25 is an example of a convolutional neural network (CNN) that generates a region label image Im2520 by performing image segmentation processing on a two-dimensional image Im2510 input as an input image. The configuration 2500 of the CNN includes a plurality of layer groups that are responsible for processing the input value group and outputting the processed value group. Note that, as shown in FIG. 25, the types of layers included in the configuration 2500 include a convolution layer, a downsampling (Downsampling) layer, an upsampling (Upsampling) layer, and a synthesis (Merger) layer. The CNN configuration 2500 used in the present embodiment is a U-net type machine learning model, like the CNN configuration 601 described in the first embodiment.
　畳み込み層は、設定されたフィルタのカーネルサイズ、フィルタの数、ストライドの値、及びダイレーションの値等のパラメータに従い、入力値群に対して畳み込み処理を行う層である。ダウンサンプリング層は、入力値群を間引いたり、合成したりすることによって、出力値群の数を入力値群の数よりも少なくする処理を行う層である。ダウンサンプリング層で行われる処理として、具体的には、例えば、Ｍａｘ　Ｐｏｏｌｉｎｇ処理がある。 The convolution layer is a layer that performs a convolution process on an input value group according to parameters such as the set filter kernel size, number of filters, stride value, and dilation value. The downsampling layer is a layer that performs processing to reduce the number of output value groups from the number of input value groups by thinning out or combining input value groups. As a process performed in the downsampling layer, specifically, for example, there is a Max @ Pooling process.
　アップサンプリング層は、入力値群を複製したり、入力値群から補間した値を追加したりすることによって、出力値群の数を入力値群の数よりも多くする処理を行う層である。アップサンプリング層で行われる処理として、具体的には、例えば、線形補間処理がある。合成層は、ある層の出力値群や画像を構成する画素値群といった値群を、複数のソースから入力し、それらを連結したり、加算したりして合成する処理を行う層である。 (4) The upsampling layer is a layer that performs processing for increasing the number of output value groups beyond the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. As the processing performed in the upsampling layer, specifically, for example, there is a linear interpolation processing. The synthesis layer is a layer that performs processing of inputting a value group such as an output value group of a certain layer or a pixel value group forming an image from a plurality of sources, and connecting or adding them to synthesize.
　なお、当該ＣＮＮの構成に含まれる畳み込み層群に設定されるパラメータとして、例えば、フィルタのカーネルサイズを幅３画素、高さ３画素、フィルタの数を６４とすることで、一定の精度の画像セグメンテーション処理が可能である。ただし、ニューラルネットワークを構成する層群やノード群に対するパラメータの設定が異なると、教師データからトレーニングされた傾向を出力データに再現可能な程度が異なる場合があるので注意が必要である。つまり、多くの場合、実施例に応じて各層群や各ノード群に対する適切なパラメータは異なるので、必要に応じて変更してもよい。 As parameters set in the convolutional layer group included in the configuration of the CNN, for example, by setting the kernel size of the filter to 3 pixels in width, 3 pixels in height, and 64 to the number of filters, an image with constant accuracy can be obtained. Segmentation processing is possible. However, it should be noted that if the parameter setting for the layer group or the node group forming the neural network is different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may be different. That is, in many cases, appropriate parameters for each layer group and each node group differ depending on the embodiment, and may be changed as necessary.
　また、実施例によっては、上述したようなパラメータを変更するという方法だけでなく、ＣＮＮの構成を変更することによって、ＣＮＮがより良い特性を得られる場合がある。より良い特性とは、例えば、画像セグメンテーション処理の精度が高かったり、画像セグメンテーション処理の時間が短かったり、学習済モデルのトレーニングにかかる時間が短かったりする等である。ＣＮＮの構成の変更例として、例えば、畳み込み層の後にバッチ正規化（Ｂａｔｃｈ　Ｎｏｒｍａｌｉｚａｔｉｏｎ）層や、正規化線形関数（Ｒｅｃｔｉｆｉｅｒ　Ｌｉｎｅａｒ　Ｕｎｉｔ）を用いた活性化層を組み込む等がある。 In some embodiments, not only the method of changing the parameters as described above but also a change in the configuration of the CNN may provide the CNN with better characteristics. The better characteristics include, for example, a higher accuracy of the image segmentation process, a shorter time for the image segmentation process, a shorter time for training the learned model, and the like. As a modification example of the configuration of the CNN, for example, a batch normalization (Batch @ Normalization) layer or an activation layer using a normalized linear function (Rectifier @ Linear @ Unit) is incorporated after the convolutional layer.
　なお、画像セグメンテーションエンジンが用いる機械学習モデルとしては、実施例１と同様に、例えば、ＦＣＮ、又はＳｅｇＮｅｔ等を用いることもできる。また、所望の構成に応じて、実施例１で述べたような領域単位で物体認識を行う機械学習モデルを用いてもよい。 As the machine learning model used by the image segmentation engine, for example, FCN or SegNet can be used as in the first embodiment. Further, according to a desired configuration, a machine learning model that performs object recognition in units of regions as described in the first embodiment may be used.
　なお、一次元画像や三次元画像、四次元画像を処理する必要がある場合には、フィルタのカーネルサイズが一次元や三次元、四次元に対応していてもよい。ここで、四次元画像とは、例えば、三次元の動画像や三次元画像の各画素位置におけるパラメータを異なる色相で示したような画像を含む。 When it is necessary to process a one-dimensional image, a three-dimensional image, or a four-dimensional image, the filter kernel size may correspond to one-dimensional, three-dimensional, or four-dimensional. Here, the four-dimensional image includes, for example, a three-dimensional moving image or an image in which parameters at each pixel position of the three-dimensional image are indicated by different hues.
　また、画像セグメンテーション処理は、１つの画像処理手法だけで実施されることもあるし、２つ以上の画像処理手法を組み合わせて実施されることもある。さらに、複数の画像セグメンテーション処理手法を実施し、複数の領域ラベル画像を生成することもできる。 {Circle around (1)} The image segmentation processing may be performed by only one image processing method, or may be performed by combining two or more image processing methods. Furthermore, a plurality of image segmentation processing methods can be performed to generate a plurality of region label images.
　また、実施例によっては、入力画像を小領域群に分割し、それぞれに対して画像セグメンテーション処理を実施して小領域の領域ラベル画像群を得て、該小領域の領域ラベル画像群を結合することで、領域ラベル画像を生成する方法がある。なお、当該小領域は、入力画像が三次元画像であれば、入力画像よりも小さな三次元画像であったり、二次元画像であったり、一次元画像であったりしてもよい。また、当該小領域は、入力画像が二次元画像であれば、入力画像よりも小さな二次元画像であったり、一次元画像であったりしてもよい。また、実施例によっては複数の領域ラベル画像群を出力してもよい。 Further, in some embodiments, the input image is divided into small area groups, image segmentation processing is performed on each of them to obtain small area area label images, and the small area area label images are combined. Thus, there is a method of generating an area label image. If the input image is a three-dimensional image, the small region may be a three-dimensional image smaller than the input image, a two-dimensional image, or a one-dimensional image. In addition, if the input image is a two-dimensional image, the small region may be a two-dimensional image smaller than the input image or a one-dimensional image. In some embodiments, a plurality of area label image groups may be output.
　また、画像セグメンテーションエンジンに対して、入力画像とともにパラメータを入力してもよい。この場合の入力されるパラメータは、例えば、病変の大きさの上限など、画像セグメンテーション処理を行う範囲の程度を指定するパラメータや、画像処理手法に用いられる画像フィルタサイズを指定するパラメータを含むことができる。なお、画像セグメンテーションエンジンは、実施例によっては領域ラベル画像の代わりに、領域を特定可能なその他の画像や座標データ群を出力してもよい。 Also, parameters may be input to the image segmentation engine together with the input image. In this case, the input parameters may include, for example, a parameter that specifies a degree of a range in which the image segmentation process is performed, such as an upper limit of a lesion size, and a parameter that specifies an image filter size used in an image processing method. it can. Note that the image segmentation engine may output another image or a coordinate data group capable of specifying an area instead of the area label image in some embodiments.
　なお、複数の画像セグメンテーション処理手法を実施したり、複数の小領域群に対して画像セグメンテーション処理を実施したりする場合には、並列的に画像セグメンテーション処理を行うことによって、処理時間を短縮できる。 When performing a plurality of image segmentation processing methods or performing an image segmentation process on a plurality of small area groups, the processing time can be reduced by performing the image segmentation processing in parallel.
　なお、ＣＮＮを用いた画像処理等、一部の画像処理手法を利用する場合には画像サイズについて注意する必要がある。具体的には、領域ラベル画像の周辺部が十分な精度でセグメンテーション処理されない問題等の対策のため、入力する画像と出力する領域ラベル画像とで異なる画像サイズを要する場合があることに留意すべきである。 When using some image processing methods such as image processing using CNN, it is necessary to pay attention to the image size. Specifically, it should be noted that the input image and the output region label image may require different image sizes in order to prevent a problem such as a problem that the peripheral portion of the region label image is not segmented with sufficient accuracy. It is.
　明瞭な説明のため、後述の実施例において明記はしないが、画像セグメンテーションエンジンに入力される画像と出力される画像とで異なる画像サイズを要する画像セグメンテーションエンジンを採用した場合には、適宜画像サイズを調整しているものとする。具体的には、学習済モデルをトレーニングするための教師データに用いる画像や、画像セグメンテーションエンジンに入力される画像といった入力画像に対して、パディングを行ったり、該入力画像の周辺の撮影領域を結合したりして、画像サイズを調整する。なお、パディングを行う領域は、効果的に画像セグメンテーション処理できるように画像セグメンテーション処理手法の特性に合わせて、一定の画素値で埋めたり、近傍画素値で埋めたり、ミラーパディングしたりする。 For the sake of clarity, although not specified in the embodiments described below, if an image segmentation engine that requires a different image size between an image input to the image segmentation engine and an output image is used, the image size may be appropriately adjusted. It has been adjusted. More specifically, padding is performed on an input image such as an image used as teacher data for training a trained model or an image input to an image segmentation engine, or a shooting area around the input image is combined. Or adjust the image size. The area to be padded is filled with a fixed pixel value, filled with neighboring pixel values, or mirror-padded according to the characteristics of the image segmentation processing method so that the image segmentation processing can be performed effectively.
　撮影箇所推定エンジンとは、入力された画像の撮影部位や撮影領域を推定するモジュールのことである。撮影箇所推定エンジンは、入力された画像に描画されている撮影部位や撮影領域がどこであるか、又は必要な詳細レベルの撮影部位ラベルや撮影領域ラベル毎に、該撮影部位や撮影領域である確率を出力することができる。 (4) The shooting location estimation engine is a module that estimates a shooting site and a shooting area of an input image. The imaging location estimation engine determines the location of the imaging region or imaging region drawn in the input image, or the probability of being the imaging region or imaging region for each imaging region label or imaging region label of a required level of detail. Can be output.
　撮影部位や撮影領域は、撮影装置によっては撮影条件として保存していない、又は撮影装置が取得できず保存できていない場合がある。また、撮影部位や撮影領域が保存されていても、必要な詳細レベルの撮影部位や撮影領域が保存されていない場合もある。例えば、撮影部位として“後眼部”と保存されているだけで、詳細には“黄斑部”なのか、“視神経乳頭部”なのか、又は、“黄斑部及び視神経乳頭部”なのか、“その他”なのかが分からないことがある。また、別の例では、撮影部位として“乳房”と保存されているだけで、詳細には“右乳房”なのか、“左乳房”なのか、又は、“両方”なのかが分からないことがある。そのため、撮影箇所推定エンジンを用いることで、これらの場合に入力画像の撮影部位や撮影領域を推定することができる。 The imaging region and the imaging region may not be stored as the imaging conditions depending on the imaging device, or may not be able to be acquired and stored by the imaging device. Further, even when the imaging region and the imaging region are stored, the imaging region and the imaging region at the necessary detailed level may not be stored. For example, only the “posterior segment” is stored as the imaging region, and in detail, whether it is the “macular region”, the “optic nerve head”, or the “macular and optic nerve head”, Others may not know what it is. Also, in another example, it is only stored as “breast” as an imaging part, and it is not known in detail whether it is “right breast”, “left breast”, or “both”. is there. Therefore, by using the imaging location estimation engine, it is possible to estimate the imaging site and the imaging area of the input image in these cases.
　撮影箇所推定エンジンの推定手法を構成する画像及びデータ処理手法では、ディープラーニング等の各種機械学習アルゴリズムに従った学習済モデルを用いた処理を行う。なお、当該画像及びデータ処理手法では、機械学習アルゴリズムを用いた処理に加えて又は代えて、自然言語処理、類似画像及び類似データのデータベースを用いたマッチング処理、知識ベース処理等の既知の任意の推定処理を行ってもよい。なお、機械学習アルゴリズムを用いて構築した学習済モデルをトレーニングする教師データは、撮影部位や撮影領域のラベルが付けられた画像とすることができる。この場合には、教師データに関して、撮影部位や撮影領域を推定すべき画像を入力データ、撮影部位や撮影領域のラベルを出力データとする。 画像 In the image and data processing method that constitutes the estimation method of the shooting location estimation engine, processing using a learned model according to various machine learning algorithms such as deep learning is performed. In addition, in the image and data processing method, in addition to or instead of the processing using the machine learning algorithm, any known known processing such as natural language processing, matching processing using a database of similar images and similar data, knowledge base processing, and the like. An estimation process may be performed. Note that the teacher data for training the trained model constructed using the machine learning algorithm may be an image to which a label of an imaging region or an imaging region is attached. In this case, regarding the teacher data, an image for estimating an imaging region or an imaging region is set as input data, and a label of the imaging region or the shooting region is set as output data.
　特に、二次元の入力画像Ｉｍ２６１０の撮影箇所を推定するＣＮＮの構成例として、図２６に示す構成２６００がある。当該ＣＮＮの構成２６００には、畳み込み層２６２１とバッチ正規化層２６２２と正規化線形関数を用いた活性化層２６２３とで構成された複数の畳み込み処理ブロック２６２０群が含まれる。また、当該ＣＮＮの構成２６００には、最後の畳み込み層２６３０と、全結合（Ｆｕｌｌ　Ｃｏｎｎｅｃｔｉｏｎ）層２６４０と、出力層２６５０が含まれる。全結合層２６４０は畳み込み処理ブロック２６２０の出力値群を全結合する。また、出力層２６５０は、Ｓｏｆｔｍａｘ関数を利用して、入力画像Ｉｍ２６１０に対する、想定される撮影部位ラベル毎の確率を推定結果２６６０（Ｒｅｓｕｌｔ）として出力する。このような構成２６００では、例えば、入力画像Ｉｍ２６１０が“黄斑部”を撮影した画像であれば、“黄斑部”に対応する撮影部位ラベルについて最も高い確率が出力される。 Particularly, as a configuration example of the CNN for estimating the shooting location of the two-dimensional input image Im2610, there is a configuration 2600 shown in FIG. The configuration 2600 of the CNN includes a plurality of convolution processing blocks 2620 each including a convolution layer 2621, a batch normalization layer 2622, and an activation layer 2623 using a normalized linear function. In addition, the configuration 2600 of the CNN includes a last convolution layer 2630, a full connection layer 2640, and an output layer 2650. The full coupling layer 2640 fully couples the output value group of the convolution processing block 2620. Further, the output layer 2650 outputs a probability of each assumed imaging region label with respect to the input image Im2610 using the Softmax function as an estimation result 2660 (Result). In such a configuration 2600, for example, if the input image Im2610 is an image obtained by imaging the “macula”, the highest probability is output for the imaging region label corresponding to the “macula”.
　なお、例えば、畳み込み処理ブロック２６２０の数を１６、畳み込み層２６２１，２６３０群のパラメータとして、フィルタのカーネルサイズを幅３画素、高さ３画素、フィルタの数を６４とすることで、一定の精度で撮影部位を推定することができる。しかしながら、実際には上記の学習済モデルの説明において述べた通り、学習済モデルの利用形態に応じた教師データを用いて、より良いパラメータ群を設定することができる。なお、一次元画像や三次元画像、四次元画像を処理する必要がある場合には、フィルタのカーネルサイズを一次元や三次元、四次元に拡張してもよい。なお、推定手法は、一つの画像及びデータ処理手法だけで実施されることもあるし、二つ以上の画像及びデータ処理手法を組み合わせて実施されることもある。 For example, assuming that the number of convolution processing blocks 2620 is 16, the parameters of the convolution layers 2621 and 2630, the filter kernel size is 3 pixels in width, 3 pixels in height, and the number of filters is 64, the constant accuracy is maintained. Can be used to estimate the imaging region. However, in practice, as described in the description of the learned model, a better parameter group can be set using the teacher data according to the use form of the learned model. When it is necessary to process a one-dimensional image, a three-dimensional image, or a four-dimensional image, the filter kernel size may be extended to one-dimensional, three-dimensional, or four-dimensional. Note that the estimation method may be performed using only one image and data processing method, or may be performed using a combination of two or more image and data processing methods.
　領域ラベル画像評価エンジンとは、入力された領域ラベル画像が尤もらしく画像セグメンテーション処理できているか否かを評価するモジュールのことである。領域ラベル画像評価エンジンは、具体的には、画像評価指数として、入力された領域ラベル画像が尤もらしければ真値、そうでなければ偽値を出力する。当該評価を行う手法の例としては、ディープラーニング等の各種機械学習アルゴリズムに従った学習済モデルを用いた処理、又は知識ベース処理等がある。知識ベース処理の方法の１つには、例えば、解剖学的な知識を利用したものがあり、例えば網膜の形状の規則性等の既知の解剖学的な知識を利用して領域ラベル画像の評価を行う。 The area label image evaluation engine is a module that evaluates whether or not an input area label image is likely to be subjected to image segmentation processing. Specifically, the area label image evaluation engine outputs, as an image evaluation index, a true value if the input area label image is likely, and outputs a false value otherwise. Examples of the method for performing the evaluation include processing using a learned model according to various machine learning algorithms such as deep learning, or knowledge base processing. One of the methods of the knowledge base processing uses, for example, anatomical knowledge. For example, evaluation of a region label image using known anatomical knowledge such as regularity of a retinal shape is performed. I do.
　具体的に、一例として、知識ベース処理により、後眼部を撮影対象としたＯＣＴの断層画像に対応する領域ラベル画像を評価する場合について説明する。後眼部では、解剖学的に組織群には決まった位置がある。そのため、領域ラベル画像における画素値群、すなわち、領域ラベル値群の座標を確認し、位置が正しく出力されているか否かを評価する方法がある。当該評価方法では、例えば、ある範囲において前眼部に近い座標に水晶体に対応する領域ラベル値があり、遠い座標に網膜層群に対応する領域ラベル値があれば、尤もらしく画像セグメンテーション処理できていると評価する。一方、これらの領域ラベル値がそのような想定される位置にない場合には、適切に画像セグメンテーション処理できていないと評価する。 Specifically, as an example, a case will be described in which an area label image corresponding to an OCT tomographic image in which the posterior segment is imaged is evaluated by the knowledge base processing. In the posterior segment, anatomical tissue groups have fixed positions. Therefore, there is a method of checking the pixel value group in the area label image, that is, the coordinates of the area label value group, and evaluating whether or not the position is correctly output. In the evaluation method, for example, if there is a region label value corresponding to the crystalline lens at coordinates close to the anterior segment in a certain range and a region label value corresponding to the retinal layer group at distant coordinates, it is possible to perform the image segmentation process likelihood. Evaluate On the other hand, when these region label values are not at such assumed positions, it is evaluated that the image segmentation processing has not been properly performed.
　図２７に示す、後眼部を撮影対象としたＯＣＴの断層画像に対応する、網膜層を構成する層群の領域ラベル画像Ｉｍ２７１０を用いて、当該知識ベースの評価方法について、より具体的に説明する。後眼部では、解剖学的に組織群には決まった位置があるため、領域ラベル画像における画素値群、すなわち、領域ラベル値群の座標を確認することで、領域ラベル画像が尤もらしい画像か否かを判断することができる。 The evaluation method of the knowledge base will be described more specifically using an area label image Im2710 of a layer group constituting the retinal layer corresponding to an OCT tomographic image of the posterior segment shown in FIG. I do. In the posterior segment, the anatomical tissue group has a fixed position.Therefore, by checking the pixel value group in the region label image, that is, the coordinates of the region label value group, whether the region label image is likely to be an image Can be determined.
　領域ラベル画像Ｉｍ２７１０には、同じ領域ラベル値を持つ画素群が連続して構成される領域Ｓｅｇ２７１１、領域Ｓｅｇ２７１２、領域Ｓｅｇ２７１３、及び領域Ｓｅｇ２７１４が含まれている。領域Ｓｅｇ２７１１及び領域Ｓｅｇ２７１４は同じ領域ラベル値だが、解剖学的に網膜層を構成する層群は層構造を成していることから、形状や他の領域との位置関係より、領域Ｓｅｇ２７１４は誤って画像セグメンテーション処理されていると評価される。この場合、領域ラベル画像評価エンジンは画像評価指数として偽値を出力する。 The region label image Im2710 includes a region Seg 2711, a region Seg 2712, a region Seg 2713, and a region Seg 2714 in which a pixel group having the same region label value is continuously formed. Although the region Seg 2711 and the region Seg 2714 have the same region label value, the region group constituting the retinal layer anatomically has a layer structure. It is evaluated that the image segmentation processing has been performed. In this case, the area label image evaluation engine outputs a false value as the image evaluation index.
　また、知識ベースの評価処理の方法には、撮影対象に必ず存在するはずの領域に対応する領域ラベル値を持つ画素が領域ラベル画像に含まれるか否かを評価する方法もある。さらに、例えば、撮影対象に必ず存在するはずの領域に対応する領域ラベル値を持つ画素が領域ラベル画像に一定数以上含まれるか否かを評価する方法等もある。 知識 Also, as a method of the knowledge-based evaluation processing, there is a method of evaluating whether or not a pixel having an area label value corresponding to an area that is supposed to be present in the imaging target is included in the area label image. Further, for example, there is a method of evaluating whether or not a predetermined number or more of pixels having an area label value corresponding to an area that is supposed to be present in the imaging target are included in the area label image.
　また、画像セグメンテーションエンジンが複数の画像セグメンテーション処理手法を実施し、複数の領域ラベル画像を生成する場合、領域ラベル画像評価エンジンは、複数のラベル画像のうち最も尤もらしい領域ラベル画像を１つ選択して出力することもできる。 When the image segmentation engine performs a plurality of image segmentation processing methods and generates a plurality of region label images, the region label image evaluation engine selects one of the most likely region label images from the plurality of label images. Can also be output.
　なお、複数の領域ラベル画像群の各々が尤もらしい領域ラベル画像である場合、出力すべき領域ラベル画像が１つに選択しきれない場合がある。このような、領域ラベル画像が１つに選択しきれない場合には、領域ラベル評価エンジンは、例えば、予め決められた優先度で領域ラベル画像を１つに選択することができる。また、領域ラベル評価エンジンは、例えば、複数の領域ラベル画像に重みづけをして１つの領域ラベル画像にマージすることもできる。さらに、例えば、領域ラベル評価エンジンは、任意の表示部等に備えられたユーザーインターフェースに複数の領域ラベル画像群を表示して、検者（ユーザー）の指示に応じて１つに選択してもよい。なお、領域ラベル画像評価エンジンは、尤もらしい複数の領域画像の全てを出力してもよい。 When each of the plurality of region label image groups is a likely region label image, there is a case where one region label image to be output cannot be completely selected. When such an area label image cannot be completely selected, the area label evaluation engine can select one area label image with a predetermined priority, for example. In addition, the area label evaluation engine can, for example, weight a plurality of area label images and merge them into one area label image. Furthermore, for example, the area label evaluation engine displays a plurality of area label images on a user interface provided on an arbitrary display unit or the like, and selects one of them according to an instruction of an examiner (user). Good. Note that the region label image evaluation engine may output all of the plurality of likely region images.
　領域ラベル画像修正エンジンとは、入力された領域ラベル画像中の誤って画像セグメンテーション処理された領域を修正するモジュールのことである。当該修正を行う手法の例としては、知識ベース処理等がある。知識ベース処理の方法の１つには、例えば、解剖学的な知識を利用したものがある。 The region label image correction engine is a module that corrects a region in an input region label image that has been erroneously subjected to image segmentation processing. As an example of the method for performing the correction, there is a knowledge base process or the like. One of the methods of the knowledge base processing uses, for example, anatomical knowledge.
　図２７に示す、後眼部を撮影対象としたＯＣＴの断層画像に対応する、網膜層を構成する層群の領域ラベル画像Ｉｍ２７１０を再び用いて、当該領域ラベル画像の修正に係る知識ベースの修正方法について、より具体的に説明する。上述したように、領域ラベル画像Ｉｍ２７１０において、解剖学的に網膜層を構成する層群は、層構造を成していることから、形状や他の領域との位置関係より、領域Ｓｅｇ２７１４は誤って画像セグメンテーション処理された領域だということが分かる。領域ラベル画像修正エンジンは、誤ってセグメンテーション処理された領域を検出し、検出された領域を別の領域ラベル値で上書きする。例えば、図２７の場合は、領域Ｓｅｇ２７１４を、網膜層のいずれでもないことを示す領域ラベル値で上書きする。 Using the region label image Im2710 of the layer group constituting the retinal layer corresponding to the OCT tomographic image of the posterior segment shown in FIG. 27 again, the knowledge base correction for the region label image is corrected. The method will be described more specifically. As described above, in the region label image Im2710, since the layer group anatomically forming the retinal layer has a layer structure, the region Seg2714 is erroneously determined from the shape and the positional relationship with other regions. It can be seen that the area has been subjected to image segmentation processing. The region label image correction engine detects a region that has been erroneously segmented, and overwrites the detected region with another region label value. For example, in the case of FIG. 27, the area Seg 2714 is overwritten with an area label value indicating that the area is not any of the retinal layers.
　なお、領域ラベル画像修正エンジンでは、誤ってセグメンテーションされた領域について、領域ラベル評価エンジンによる評価結果を用いて検出又は特定を行ってもよい。また、領域ラベル画像修正エンジンは、検出した誤ってセグメンテーションされた領域のラベル値を、当該領域の周辺のラベル情報から推測されるラベル情報を上書きしてもよい。図２７の例では、領域Ｓｅｇ２７１４を囲う領域にラベル情報が付されている場合には、領域Ｓｅｇ２７１４のラベル情報を当該領域のラベル情報で上書きすることができる。なお、周辺のラベル情報に関しては、修正すべき領域を完全に囲む領域のラベル情報に限られず、修正すべき領域に隣接する領域のラベル情報のうち、最も数の多いラベル情報であってもよい。 Note that the region label image correction engine may detect or specify an erroneously segmented region using the evaluation result of the region label evaluation engine. Further, the region label image correction engine may overwrite the detected label value of the erroneously segmented region with the label information estimated from the label information around the region. In the example of FIG. 27, when label information is attached to an area surrounding the area Seg2714, the label information of the area Seg2714 can be overwritten with the label information of the area. In addition, the peripheral label information is not limited to the label information of the area completely surrounding the area to be corrected, and may be the label information having the largest number among the label information of the areas adjacent to the area to be corrected. .
（実施例８に係る画像処理装置の構成）
　以下、図２８乃至図３２を参照して、実施例８による医用画像処理装置について説明する。なお、以下において、簡略化のため、医用画像処理装置を単に画像処理装置という。図２８は、本実施例に係る画像処理装置の概略的な構成の一例を示す。 (Configuration of Image Processing Apparatus According to Eighth Embodiment)
The medical image processing apparatus according to the eighth embodiment will be described below with reference to FIGS. In the following, the medical image processing apparatus is simply referred to as an image processing apparatus for simplification. FIG. 28 illustrates an example of a schematic configuration of an image processing apparatus according to the present embodiment.
　画像処理装置２８００は、撮影装置２８１０及び表示部２８２０に、回路やネットワークを介して接続されている。また、撮影装置２８１０及び表示部２８２０が直接接続されていてもよい。なお、これらの装置は本実施例では別個の装置とされているが、これらの装置の一部又は全部を一体的に構成してもよい。また、これらの装置は、他の任意の装置と回路やネットワークを介して接続されてもよいし、他の任意の装置と一体的に構成されてもよい。 The image processing device 2800 is connected to the imaging device 2810 and the display unit 2820 via a circuit or a network. Further, the imaging device 2810 and the display unit 2820 may be directly connected. Although these devices are separate devices in this embodiment, a part or all of these devices may be integrally configured. These devices may be connected to any other device via a circuit or a network, or may be configured integrally with any other device.
　画像処理装置２８００には、取得部２８０１と、撮影条件取得部２８０２と、処理可否判定部２８０３と、セグメンテーション処理部２８０４と、評価部２８０５と、解析部２８０６と、出力部２８０７（表示制御部）とが設けられている。なお、画像処理装置２８００は、これら構成要素のうちの一部が設けられた複数の装置で構成されてもよい。 The image processing apparatus 2800 includes an acquisition unit 2801, a shooting condition acquisition unit 2802, a processing availability determination unit 2803, a segmentation processing unit 2804, an evaluation unit 2805, an analysis unit 2806, and an output unit 2807 (display control unit). Are provided. Note that the image processing device 2800 may be configured by a plurality of devices in which some of these components are provided.
　取得部２８０１は、撮影装置２８１０や他の装置から各種データや画像を取得したり、不図示の入力装置を介して検者からの入力を取得したりすることができる。なお、入力装置としては、マウス、キーボード、タッチパネル及びその他任意の入力装置を採用してよい。また、表示部２８２０をタッチパネルディスプレイとして構成してもよい。 The acquisition unit 2801 can acquire various data and images from the imaging device 2810 and other devices, and can acquire input from the examiner via an input device (not shown). In addition, as the input device, a mouse, a keyboard, a touch panel, and any other input device may be employed. Further, display portion 2820 may be configured as a touch panel display.
　撮影条件取得部２８０２は、取得部２８０１が取得した医用画像（入力画像）の撮影条件を取得する。具体的には、医用画像のデータ形式に応じて、医用画像を構成するデータ構造に保存された撮影条件群を取得する。なお、医用画像に撮影条件が保存されていない場合には、取得部２８０１を介して、撮影装置２８１０や画像管理システムから撮影情報群を取得することができる。 The imaging condition acquisition unit 2802 acquires the imaging conditions of the medical image (input image) acquired by the acquisition unit 2801. More specifically, a group of imaging conditions stored in a data structure forming a medical image is acquired according to the data format of the medical image. Note that when the imaging conditions are not stored in the medical image, the imaging information group can be acquired from the imaging device 2810 or the image management system via the acquisition unit 2801.
　処理可否判定部（判定部）２８０３は、撮影条件取得部２８０２によって取得された撮影条件群を用いてセグメンテーション処理部２８０４によって医用画像が対処可能であるか否かを判定する。セグメンテーション処理部２８０４は、学習済モデルを含む画像セグメンテーションエンジン（セグメンテーションエンジン）を用いて、対処可能である医用画像について画像セグメンテーション処理を行い、領域ラベル画像（領域情報）を生成する。 (4) The processing availability determination unit (determination unit) 2803 determines whether the medical image can be handled by the segmentation processing unit 2804 using the imaging condition group acquired by the imaging condition acquisition unit 2802. The segmentation processing unit 2804 performs image segmentation processing on a medical image that can be dealt with using an image segmentation engine (segmentation engine) including the learned model, and generates an area label image (area information).
　評価部２８０５は、領域ラベル画像評価エンジン（評価エンジン）を用いて、セグメンテーション処理部２８０４によって生成された領域ラベル画像を評価し、評価結果に基づいて領域ラベル画像を出力するか否かを判断する。領域ラベル画像評価エンジンは、画像評価指数として、入力された領域ラベル画像が尤もらしければ真値、そうでなければ偽値を出力する。評価部２８０５は、領域ラベル画像を評価した画像評価指数が真値である場合には、領域ラベル画像を出力すると判断する。 The evaluation unit 2805 evaluates the area label image generated by the segmentation processing unit 2804 using an area label image evaluation engine (evaluation engine), and determines whether to output the area label image based on the evaluation result. . The area label image evaluation engine outputs a true value as the image evaluation index if the input area label image is likely, and outputs a false value otherwise. The evaluation unit 2805 determines that the area label image is output when the image evaluation index obtained by evaluating the area label image is a true value.
　解析部２８０６は、評価部２８０５によって出力すべきと判断された領域ラベル画像や入力画像を用いて、入力画像の画像解析処理を行う。解析部２８０６は、画像解析処理により、例えば、網膜層に含まれる組織の形状変化や層厚等を算出することができる。なお、画像解析処理としては、既知の任意の画像解析処理を用いてよい。出力部２８０７は、領域ラベル画像や解析部２８０６による解析結果を表示部２８２０に表示させる。また、出力部２８０７は、領域ラベル画像や解析結果を画像処理装置２８００に接続される記憶装置や外部装置等に記憶させてもよい。 The analysis unit 2806 performs an image analysis process on the input image using the area label image or the input image determined to be output by the evaluation unit 2805. The analysis unit 2806 can calculate, for example, a shape change and a layer thickness of the tissue included in the retinal layer by the image analysis processing. Note that any known image analysis process may be used as the image analysis process. The output unit 2807 causes the display unit 2820 to display the region label image and the analysis result by the analysis unit 2806. The output unit 2807 may store the area label image and the analysis result in a storage device connected to the image processing device 2800, an external device, or the like.
　次に、セグメンテーション処理部２８０４について詳細に説明する。セグメンテーション処理部２８０４は、画像セグメンテーションエンジンを用いて、入力された画像（入力）に対応する領域ラベル画像を生成する。本実施例に係る画像セグメンテーションエンジンが備える画像セグメンテーション処理手法では、学習済モデルを用いた処理を行う。 Next, the segmentation processing unit 2804 will be described in detail. The segmentation processing unit 2804 generates an area label image corresponding to the input image (input) using the image segmentation engine. In the image segmentation processing method included in the image segmentation engine according to the present embodiment, a process using a learned model is performed.
　本実施例では、機械学習モデルのトレーニングに、処理対象として想定される特定の撮影条件で取得された画像である入力データと、入力データに対応する領域ラベル画像である出力データのペア群で構成された教師データを用いる。なお、特定の撮影条件には、具体的には、予め決定された撮影部位、撮影方式、撮影画角、及び画像サイズ等が含まれる。 In the present embodiment, the training of the machine learning model is configured by a pair group of input data that is an image acquired under a specific imaging condition assumed as a processing target and output data that is an area label image corresponding to the input data. Use the obtained teacher data. It should be noted that the specific imaging conditions specifically include a predetermined imaging region, imaging method, imaging angle of view, image size, and the like.
　本実施例において、教師データの入力データは、撮影装置２８１０と同じ機種、撮影装置２８１０と同じ設定により取得された画像である。なお、教師データの入力データは、撮影装置２８１０と同じ画質傾向を持つ撮影装置から取得された画像であってもよい。 In the present embodiment, the input data of the teacher data is an image acquired with the same model as the imaging device 2810 and the same settings as the imaging device 2810. Note that the input data of the teacher data may be an image acquired from a photographing device having the same image quality tendency as the photographing device 2810.
　また、教師データの出力データは、入力データに対応する領域ラベル画像である。例えば、図２４を参照して説明すると、入力データは、ＯＣＴによって撮影された網膜層の断層画像Ｉｍ２４１０である。また、出力データは、断層画像Ｉｍ２４１０に描出された網膜層の種類に合わせて、対応する座標に網膜層の種類を表す領域ラベル値を付して、各領域を分けた領域ラベル画像Ｉｍ２４２０である。領域ラベル画像は、断層画像を参照して専門医が作成したり、任意の画像セグメンテーション処理によって作成したり、該画像セグメンテーション処理によって作成された領域ラベル画像を専門医が修正して作成したりすることによって用意できる。 The output data of the teacher data is an area label image corresponding to the input data. For example, with reference to FIG. 24, the input data is a tomographic image Im2410 of a retinal layer captured by OCT. The output data is an area label image Im2420 obtained by attaching an area label value representing the type of the retinal layer to the corresponding coordinates according to the type of the retinal layer depicted in the tomographic image Im2410, and dividing each area. . The area label image is created by a specialist with reference to a tomographic image, created by an arbitrary image segmentation process, or created by a specialist correcting an area label image created by the image segmentation process. Can be prepared.
　なお、教師データのペア群の入力データ群には処理対象として想定される、様々な条件を有する入力画像が網羅的に含まれる。様々な条件とは、具体的には、被検者の病態、撮影装置の撮影環境、撮影者の技術レベル等のバリエーションの組み合わせの違いによって生じる画像の条件である。入力データ群に含まれる画像の条件が網羅的になることによって、従来の画像セグメンテーション処理では精度が低くなってしまう悪条件の画像に対しても、精度の高い画像セグメンテーション処理が実施可能なように機械学習モデルがトレーニングされる。そのため、セグメンテーション処理部２８０４は、このようなトレーニングが行われた学習済モデルを含む画像セグメンテーションエンジンを用いることで、様々な条件の画像に対して、安定して精度の高い領域ラベル画像を生成することができる。 入 力 Note that the input data group of the teacher data pair group comprehensively includes input images having various conditions that are assumed to be processed. The various conditions are, specifically, image conditions caused by a combination of variations such as a disease state of a subject, a photographing environment of a photographing device, and a technical level of a photographer. By making the conditions of the images included in the input data group exhaustive, it is possible to perform highly accurate image segmentation processing even for images with bad conditions where accuracy is low in conventional image segmentation processing. The machine learning model is trained. Therefore, by using the image segmentation engine including the trained model that has undergone such training, the segmentation processing unit 2804 generates a stable and highly accurate region label image for images under various conditions. be able to.
　なお、教師データを構成するペア群のうち、画像セグメンテーション処理に寄与しないペアは教師データから取り除くことができる。例えば、教師データのペアを構成する出力データである領域ラベル画像の領域ラベル値が誤っている場合には、当該教師データを用いて学習した学習済モデルを用いて得た領域ラベル画像の領域ラベル値も誤ってしまう可能性が高くなる。つまり、画像セグメンテーション処理の精度が低くなる。そのため、誤った領域ラベル値を持つ領域ラベル画像を出力データに持つ、ペアを教師データから取り除くことで、画像セグメンテーションエンジンに含まれる学習済モデルの精度を向上できる可能性がある。 ペ ア Note that, of the group of pairs forming the teacher data, pairs that do not contribute to the image segmentation process can be removed from the teacher data. For example, if the region label value of the region label image, which is output data forming a pair of teacher data, is incorrect, the region label of the region label image obtained using the trained model learned using the teacher data is used. It is more likely that the value will be wrong. That is, the accuracy of the image segmentation processing is reduced. Therefore, there is a possibility that the accuracy of the trained model included in the image segmentation engine can be improved by removing the pair having the area label image having the incorrect area label value as the output data from the teacher data.
　このような学習済モデルを含む画像セグメンテーションエンジンを用いることで、セグメンテーション処理部２８０４は、撮影で取得された医用画像が入力された場合に、該医用画像に描出された臓器や病変を特定可能な領域ラベル画像を出力することができる。 By using an image segmentation engine including such a trained model, the segmentation processing unit 2804 can specify an organ or a lesion depicted in the medical image when a medical image acquired by imaging is input. An area label image can be output.
　次に、図２９のフローチャートを参照して、本実施例に係る一連の画像処理について説明する。図２９は本実施例に係る一連の画像処理のフローチャートである。まず、本実施例に係る一連の画像処理が開始されると、処理はステップＳ２９１０に移行する。 Next, a series of image processing according to the present embodiment will be described with reference to the flowchart in FIG. FIG. 29 is a flowchart of a series of image processing according to the present embodiment. First, when a series of image processing according to the present embodiment is started, the process proceeds to step S2910.
　ステップＳ２９１０では、取得部２８０１が、回路やネットワークを介して接続された撮影装置２８１０から、撮影装置２８１０が撮影した画像を入力画像として取得する。なお、取得部２８０１は、撮影装置２８１０からの要求に応じて、入力画像を取得してもよい。このような要求は、例えば、撮影装置２８１０が画像を生成した時、撮影装置２８１０が生成した画像を撮影装置２８１０の記録装置に保存する前や保存した後、保存された画像を表示部２８２０に表示する時、画像解析処理に領域ラベル画像を利用する時等に発行されてよい。 In step S2910, the acquiring unit 2801 acquires, as an input image, an image photographed by the photographing device 2810 from the photographing device 2810 connected via a circuit or a network. Note that the acquisition unit 2801 may acquire an input image in response to a request from the imaging device 2810. Such a request may be made, for example, when the imaging device 2810 generates an image, before or after storing the image generated by the imaging device 2810 in the recording device of the imaging device 2810, and then displays the stored image on the display unit 2820. It may be issued at the time of display, when an area label image is used for image analysis processing, or the like.
　なお、取得部２８０１は、撮影装置２８１０から画像を生成するためのデータを取得し、画像処理装置２８００が当該データに基づいて生成した画像を入力画像として取得してもよい。この場合、画像処理装置２８００が各種画像を生成するための画像生成方法としては、既存の任意の画像生成方法を採用してよい。 Note that the obtaining unit 2801 may obtain data for generating an image from the imaging device 2810, and obtain an image generated based on the data by the image processing device 2800 as an input image. In this case, any existing image generation method may be adopted as an image generation method for the image processing apparatus 2800 to generate various images.
　ステップＳ２９２０では、撮影条件取得部２８０２が、入力画像の撮影条件群を取得する。具体的には、入力画像のデータ形式に応じて、入力画像を構成するデータ構造に保存された撮影条件群を取得する。なお、上述のように、入力画像に撮影条件が保存されていない場合には、撮影条件取得部２８０２は、撮影装置２８１０や不図示の画像管理システムから撮影情報群を取得することができる。 In step S2920, the photographing condition acquiring unit 2802 acquires a photographing condition group of the input image. More specifically, a group of photographing conditions stored in a data structure forming the input image is acquired according to the data format of the input image. Note that, as described above, when the imaging conditions are not stored in the input image, the imaging condition acquisition unit 2802 can acquire the imaging information group from the imaging device 2810 or an image management system (not illustrated).
　ステップＳ２９３０においては、処理可否判定部２８０３が、取得された撮影条件群を用いて、セグメンテーション処理部２８０４が用いる画像セグメンテーションエンジンによって入力画像を画像セグメンテーション処理可能であるか否かを判定する。具体的には、処理可否判定部２８０３は、入力画像の撮影部位、撮影方式、撮影画角、及び画像サイズが、画像セグメンテーションエンジンの学習済モデルを用いて対処可能な条件と一致するか否かを判定する。 In step S2930, the processability determination unit 2803 determines whether or not the input image can be subjected to image segmentation processing by the image segmentation engine used by the segmentation processing unit 2804, using the acquired shooting condition group. Specifically, the processing possibility determination unit 2803 determines whether the imaging region, imaging method, imaging angle of view, and image size of the input image match the conditions that can be dealt with using the learned model of the image segmentation engine. Is determined.
　処理可否判定部２８０３が、すべての撮影条件を判定し、対処可能と判定された場合には、処理はステップＳ２９４０に移行する。一方、処理可否判定部２８０３が、これら撮影条件に基づいて、画像セグメンテーションエンジンが入力画像を対処不可能であると判定した場合には、処理はステップＳ２９７０に移行する。 (4) The processing availability determination unit 2803 determines all shooting conditions, and if it is determined that it can be dealt with, the process proceeds to step S2940. On the other hand, when the processability determination unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970.
　なお、画像処理装置２８００の設定や実装形態によっては、撮影部位、撮影方式、撮影画角、及び画像サイズのうちの一部に基づいて入力画像が処理不可能であると判定されたとしても、ステップＳ２９４０が実施されてもよい。例えば、画像セグメンテーションエンジンが、被検者のいずれの撮影部位に対しても網羅的に対応可能であると想定され、入力データに未知の撮影部位が含まれていたとしても対処可能であるように実装されている場合等には、このような処理を行ってもよい。また、処理可否判定部２８０３は所望の構成に応じて、入力画像の撮影部位、撮影方式、撮影画角、及び画像サイズのうちの少なくとも一つが画像セグメンテーションエンジンによって対処可能な条件と一致するか否かを判定してもよい。 Note that, depending on the settings and the implementation form of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on a part of the imaging part, the imaging method, the imaging angle of view, and the image size, Step S2940 may be performed. For example, it is assumed that the image segmentation engine can comprehensively deal with any imaging region of the subject, and can cope with the case where the input data includes an unknown imaging region. Such a process may be performed when it is implemented. In addition, according to a desired configuration, the processing availability determination unit 2803 determines whether at least one of a shooting region, a shooting method, a shooting angle of view, and an image size of an input image matches a condition that can be handled by the image segmentation engine. May be determined.
　ステップＳ２９４０においては、セグメンテーション処理部２８０４が、画像セグメンテーションエンジンを用いて、入力画像に対して画像セグメンテーション処理を行い、入力画像から領域ラベル画像を生成する。具体的には、セグメンテーション処理部２８０４は、入力画像を画像セグメンテーションエンジンに入力する。画像セグメンテーションエンジンは、教師データを用いて機械学習を行った学習済モデルに基づいて、入力画像に描出された臓器や病変を特定可能な領域情報として領域ラベル画像を生成する。 In step S2940, the segmentation processing unit 2804 performs an image segmentation process on the input image using the image segmentation engine, and generates a region label image from the input image. Specifically, the segmentation processing unit 2804 inputs the input image to the image segmentation engine. The image segmentation engine generates an area label image as area information that can specify an organ or a lesion depicted in an input image, based on a learned model in which machine learning has been performed using teacher data.
　なお、画像処理装置２８００の設定や実装形態によっては、セグメンテーション処理部２８０４が、撮影条件群に応じて、画像セグメンテーションエンジンに入力画像とともにパラメータを入力して、画像セグメンテーション処理の範囲の程度等を調節してもよい。また、セグメンテーション処理部２８０４は、検者の入力に応じたパラメータを画像セグメンテーションエンジンに入力画像とともに入力して画像セグメンテーション処理の範囲の程度等を調整してもよい。 Note that, depending on the settings and the implementation form of the image processing apparatus 2800, the segmentation processing unit 2804 inputs parameters together with the input image to the image segmentation engine according to the shooting condition group, and adjusts the extent of the image segmentation processing and the like. May be. In addition, the segmentation processing unit 2804 may input a parameter corresponding to the input of the examiner to the image segmentation engine together with the input image to adjust the extent of the image segmentation processing.
　ステップＳ２９５０では、評価部２８０５が領域ラベル画像評価エンジンを用いて、セグメンテーション処理部２８０４によって生成された領域ラベル画像について、当該領域ラベル画像が尤もらしい画像か否かを評価する。本実施例では、評価部２８０５は、知識ベースの評価方法を用いる領域ラベル評価エンジンを用いて、領域ラベル画像が尤もらしい画像か否かを評価する。 In step S2950, the evaluation unit 2805 evaluates whether or not the region label image generated by the segmentation processing unit 2804 is a likely image using the region label image evaluation engine. In this embodiment, the evaluation unit 2805 evaluates whether or not the area label image is a likely image using an area label evaluation engine that uses a knowledge-based evaluation method.
　具体的には、領域ラベル評価エンジンは、領域ラベル画像における画素値群、すなわち、領域ラベル値群の座標を確認し、位置が解剖学的に正しい位置に出力されているか否かを評価する。この場合、例えば、ある範囲において前眼部に近い座標に水晶体に対応する領域ラベル値があり、遠い座標に網膜層群に対応する領域ラベル値があれば、尤もらしく画像セグメンテーション処理できていると評価する。一方、これらの領域ラベル値がそのような想定される位置にない場合には、適切に画像セグメンテーション処理できていないと評価する。領域ラベル評価エンジンは、領域ラベルについて、尤もらしく画像セグメンテーション処理できていると評価した場合には画像評価指数として真値を出力し、尤もらしく画像セグメンテーション処理できていないと評価した場合には偽値を出力する。 Specifically, the area label evaluation engine checks the pixel value group in the area label image, that is, the coordinates of the area label value group, and evaluates whether or not the position is output to an anatomically correct position. In this case, for example, if there is an area label value corresponding to the crystalline lens at coordinates close to the anterior segment in a certain range and an area label value corresponding to the retinal layer group at distant coordinates, it is likely that the image segmentation process has been performed. evaluate. On the other hand, when these region label values are not at such assumed positions, it is evaluated that the image segmentation processing has not been properly performed. The area label evaluation engine outputs a true value as an image evaluation index when the area label is evaluated to be likely to be able to perform the image segmentation processing, and a false value when the area label is evaluated to be unlikely to be able to perform the image segmentation processing. Is output.
　評価部２８０５は、領域ラベル評価エンジンから出力された画像評価指数に基づいて、領域ラベル画像を出力するか否かを判断する。具体的には、評価部２８０５は、画像評価指数が真値である場合には領域ラベル画像を出力すると判断する。一方、画像評価指数が偽値である場合にはセグメンテーション処理部２８０４で生成された領域ラベル画像を出力しないと判断する。なお、評価部２８０５は、セグメンテーション処理部２８０４で生成した領域ラベル画像を出力しないと判断した場合に、領域ラベル無し画像を生成することができる。 The evaluation unit 2805 determines whether to output an area label image based on the image evaluation index output from the area label evaluation engine. Specifically, if the image evaluation index is a true value, the evaluation unit 2805 determines to output the area label image. On the other hand, when the image evaluation index is a false value, it is determined that the region label image generated by the segmentation processing unit 2804 is not output. When the evaluation unit 2805 determines that the region label image generated by the segmentation processing unit 2804 is not output, the evaluation unit 2805 can generate an image without a region label.
　ステップＳ２９６０では、解析部２８０６が、評価部２８０５によって領域ラベル画像を出力すると判断されたら、領域ラベル画像及び入力画像を用いて、入力画像の画像解析処理を行う。解析部２８０６は、例えば、画像解析処理により入力画像に描出されている層厚や組織形状の変化等を算出する。なお、画像解析処理の方法は既知の任意の処理を採用してよい。また、評価部２８０５によって領域ラベル画像を出力しないと判断された場合又は領域ラベル無し画像が生成された場合には、画像解析を行わずに処理を進める。 In step S2960, when the evaluating unit 2805 determines that the area label image is to be output, the analyzing unit 2806 performs image analysis processing of the input image using the area label image and the input image. The analysis unit 2806 calculates, for example, a change in a layer thickness or a tissue shape depicted in an input image by image analysis processing. The method of the image analysis processing may employ any known processing. If the evaluation unit 2805 determines that an area label image is not to be output, or if an image without an area label is generated, the process proceeds without performing image analysis.
　ステップＳ２９７０では、出力部２８０７が、評価部２８０５によって領域ラベル画像を出力すると判断されたら、領域ラベル画像及び画像解析結果を出力して、表示部２８２０に表示させる。なお、出力部２８０７は、表示部２８２０に領域ラベル画像及び画像解析結果を表示させるのに代えて、撮影装置２８１０や他の装置にこれらを表示させたり、記憶させたりしてもよい。また、出力部２８０７は、画像処理装置２８００の設定や実装形態によっては、これらを撮影装置２８１０や他の装置が利用可能なように加工したり、画像管理システム等に送信可能なようにデータ形式を変換したりしてもよい。また、出力部２８０７は、領域ラベル画像及び画像解析結果の両方を出力する構成に限られず、これらのうちのいずれか一方のみを出力してもよい。 In step S2970, when the output unit 2807 determines that the area label image is output by the evaluation unit 2805, the output unit 2807 outputs the area label image and the image analysis result, and causes the display unit 2820 to display the image. Note that the output unit 2807 may display or store these on the imaging device 2810 or another device instead of displaying the region label image and the image analysis result on the display unit 2820. The output unit 2807 may process the image processing device 2800 so that it can be used by the image capturing device 2810 or another device, or may transmit the data to an image management system or the like, depending on the setting or implementation form of the image processing device 2800. May be converted. Also, the output unit 2807 is not limited to a configuration that outputs both the area label image and the image analysis result, and may output only one of them.
　一方、ステップＳ２９３０において画像セグメンテーション処理が不可能であるとされていた場合には、出力部２８０７は、領域ラベル画像の一種である領域ラベル無し画像を出力し、表示部２８２０に表示させる。なお、領域ラベル無し画像を出力する代わりに、撮影装置２８１０に対して、画像セグメンテーション処理が不可能であったことを示す信号を送信してもよい。 On the other hand, if it is determined in step S2930 that the image segmentation process cannot be performed, the output unit 2807 outputs an image without an area label, which is a type of area label image, and causes the display unit 2820 to display the image. Note that, instead of outputting the image without the region label, a signal indicating that the image segmentation processing has been impossible may be transmitted to the imaging device 2810.
　また、ステップＳ２９５０において画像セグメンテーション処理を適切に行えなかったと判断された場合にも、出力部２８０７は、領域ラベル無し画像を出力し、表示部２８２０に表示させる。この場合にも、出力部２８０７は、領域ラベル無し画像を出力する代わりに、撮影装置２８１０に対して、画像セグメンテーション処理を適切に行えなかったことを示す信号を送信してもよい。ステップＳ２９７０における出力処理が終了すると、一連の画像処理が終了する。 Also, when it is determined in step S2950 that the image segmentation process has not been properly performed, the output unit 2807 outputs an image without an area label and causes the display unit 2820 to display the image. Also in this case, the output unit 2807 may transmit a signal indicating that the image segmentation processing has not been properly performed to the imaging device 2810 instead of outputting an image without an area label. When the output processing in step S2970 ends, a series of image processing ends.
　上記のように、本実施例に係る画像処理装置２８００は、取得部２８０１と、セグメンテーション処理部２８０４を備える。取得部２８０１は、被検者の所定部位の断層画像である入力画像を取得する。セグメンテーション処理部２８０４は、学習済モデルを含む画像セグメンテーションエンジンを用いて、入力画像から、解剖学的な領域を識別可能な領域情報である領域ラベル画像を生成する。画像セグメンテーションエンジンは、様々な条件の断層画像と、領域ラベル画像とを学習データとした学習済モデルを含む。当該画像セグメンテーションエンジンは、断層画像を入力とし、領域ラベル画像を出力とする。 As described above, the image processing apparatus 2800 according to the present embodiment includes the acquisition unit 2801 and the segmentation processing unit 2804. The acquisition unit 2801 acquires an input image that is a tomographic image of a predetermined part of the subject. The segmentation processing unit 2804 generates an area label image, which is area information capable of identifying an anatomical area, from the input image using an image segmentation engine including the learned model. The image segmentation engine includes a learned model in which tomographic images under various conditions and region label images are used as learning data. The image segmentation engine receives a tomographic image as input and outputs a region label image.
　また、画像処理装置２８００は、評価部２８０５を更に備える。評価部２８０５は、解剖学的特徴を用いた知識ベースの評価エンジンを用いて領域ラベル画像を評価し、評価の結果に応じて領域ラベル画像を出力するか否かを判断する。 The image processing device 2800 further includes an evaluation unit 2805. The evaluation unit 2805 evaluates the area label image using a knowledge-based evaluation engine using anatomical features, and determines whether to output the area label image according to the evaluation result.
　当該構成により、本実施例に係る画像処理装置２８００は、学習済モデルを含むセグメンテーションエンジンを用いて、画像診断や画像解析に利用可能なＲＯＩやＶＯＩを特定するために用いる領域情報として領域ラベル画像を生成することができる。そのため、従来のセグメンテーション処理について悪条件な入力画像に対しても、精度の高い領域ラベル画像を出力することができ、画像診断や画像解析に利用可能なＲＯＩやＶＯＩを提供できる。 With this configuration, the image processing apparatus 2800 according to the present embodiment uses the segmentation engine including the learned model to generate a region label image as region information used for specifying ROIs and VOIs that can be used for image diagnosis and image analysis. Can be generated. Therefore, it is possible to output a highly accurate region label image even for an input image having a bad condition in the conventional segmentation processing, and to provide an ROI or VOI that can be used for image diagnosis or image analysis.
　さらに、画像処理装置２８００は、評価部２８０５によって領域ラベル画像が尤もらしい画像か否か評価することで、画像診断や画像解析に不適切な領域ラベル画像が用いられることを防止することができる。 {Circle around (2)} The image processing apparatus 2800 can prevent the use of an inappropriate area label image for image diagnosis or image analysis by evaluating whether or not the area label image is a likely image by the evaluation unit 2805.
　また、画像処理装置２８００は、撮影条件取得部２８０２と、処理可否判定部２８０３とを更に備える。撮影条件取得部２８０２は、入力画像の撮影部位、撮影方式、撮影画角、及び画像サイズの少なくとも一つを含む撮影条件を取得する。処理可否判定部２８０３は、画像セグメンテーションエンジンを用いて入力画像から領域ラベル画像を生成可能であるか否かを判定する。処理可否判定部２８０３は、入力画像の撮影条件に基づいて当該判定を行う。 The image processing apparatus 2800 further includes a photographing condition acquisition unit 2802 and a processing availability determination unit 2803. The imaging condition acquisition unit 2802 acquires an imaging condition including at least one of an imaging part, an imaging method, an imaging angle of view, and an image size of an input image. The processing availability determination unit 2803 determines whether an area label image can be generated from an input image using an image segmentation engine. The processing availability determination unit 2803 makes the determination based on the shooting conditions of the input image.
　当該構成により、本実施例に係る画像処理装置２８００は、セグメンテーション処理部２８０４が処理できない入力画像を画像セグメンテーション処理から省くことができ、画像処理装置２８００の処理負荷やエラーの発生を低減させることができる。 With this configuration, the image processing apparatus 2800 according to the present embodiment can omit an input image that cannot be processed by the segmentation processing unit 2804 from the image segmentation processing, and reduce the processing load and the occurrence of errors of the image processing apparatus 2800. it can.
　さらに、画像処理装置２８００は、領域情報である領域ラベル画像を用いて、入力画像の画像解析を行う解析部２８０６を更に備える。当該構成により、画像処理装置２８００は、セグメンテーション処理部２８０４によって生成された精度の高い領域ラベル画像を用いて画像解析を行うことができ、精度の高い解析結果を得ることができる。 {Circle around (2)} The image processing apparatus 2800 further includes an analysis unit 2806 that performs an image analysis of the input image using the area label image as the area information. With this configuration, the image processing device 2800 can perform image analysis using the highly accurate region label image generated by the segmentation processing unit 2804, and can obtain a highly accurate analysis result.
　本実施例においては、処理可否判定部２８０３が、画像セグメンテーションエンジンによって画像セグメンテーション処理可能な入力画像であるか否かを判定する。その後、処理可否判定部２８０３が、セグメンテーション処理部２８０４によって処理可能な入力画像であると判定した場合に、セグメンテーション処理部２８０４が画像セグメンテーション処理を行う。これに対し、撮影装置２８１０によって、画像セグメンテーション処理可能な撮影条件でのみ撮影が行なわれる等の場合には、撮影装置２８１０から取得した画像に対し無条件に画像セグメンテーション処理を行ってもよい。この場合には、図３０に示すように、ステップＳ２９２０とステップＳ２９３０の処理を省き、ステップＳ２９１０の次にステップＳ２９４０を実施することができる。 In the present embodiment, the processing availability determination unit 2803 determines whether the input image can be subjected to image segmentation processing by the image segmentation engine. After that, when the processing availability determination unit 2803 determines that the input image can be processed by the segmentation processing unit 2804, the segmentation processing unit 2804 performs image segmentation processing. On the other hand, when the image capturing device 2810 performs image capturing only under image capturing conditions that allow image segmentation processing, the image acquired from the image capturing device 2810 may be unconditionally subjected to image segmentation processing. In this case, as shown in FIG. 30, the processing of steps S2920 and S2930 can be omitted, and step S2940 can be performed after step S2910.
　なお、本実施例においては、出力部２８０７（表示制御部）は、生成された領域ラベル画像や解析結果を表示部２８２０に表示させる構成としたが、出力部２８０７の動作はこれに限られない。例えば、出力部２８０７は、領域ラベル画像や解析結果を撮影装置２８１０や画像処理装置２８００に接続される他の装置に出力することもできる。このため、領域ラベル画像や解析結果は、これらの装置のユーザーインターフェースに表示されたり、任意の記録装置に保存されたり、任意の画像解析に利用されたり、画像管理システムに送信されたりすることができる。 In the present embodiment, the output unit 2807 (display control unit) displays the generated area label image and the analysis result on the display unit 2820. However, the operation of the output unit 2807 is not limited to this. . For example, the output unit 2807 can output the region label image and the analysis result to another device connected to the imaging device 2810 or the image processing device 2800. For this reason, the region label image and the analysis result can be displayed on the user interface of these devices, saved in any recording device, used for arbitrary image analysis, and transmitted to the image management system. it can.
　また、本実施例においては、出力部２８０７が、表示部２８２０に領域ラベル画像や画像解析結果を表示させる構成とした。しかしながら、出力部２８０７は、検者からの指示に応じて、領域ラベル画像や画像解析結果を表示部２８２０に表示させてもよい。例えば、出力部２８０７は、検者が表示部２８２０のユーザーインターフェース上の任意のボタンを押すことに応じて、領域ラベル画像や画像解析結果を表示部２８２０に表示させてもよい。この場合、出力部２８０７は、入力画像と切り替えて領域ラベル画像を表示させてもよい。また、出力部２８０７は、図３１のように入力画像ＵＩ３１１０と並べて領域ラベル画像ＵＩ３１２０を表示させてもよいし、図３２のＵＩ３２１０～ＵＩ３２４０のいずれかのように入力画像と半透明化した領域ラベル画像とを重畳して表示させてもよい。なお、半透明化の方法は既知の任意のものであってよく、例えば、領域ラベル画像の透明度を所望の値に設定することで、領域ラベル画像を半透明化することができる。 In the present embodiment, the output unit 2807 displays the area label image and the image analysis result on the display unit 2820. However, the output unit 2807 may cause the display unit 2820 to display the region label image or the image analysis result according to an instruction from the examiner. For example, the output unit 2807 may cause the display unit 2820 to display an area label image or an image analysis result in response to the examiner pressing an arbitrary button on the user interface of the display unit 2820. In this case, the output unit 2807 may display the region label image by switching to the input image. The output unit 2807 may display the area label image UI3120 side by side with the input image UI3110 as shown in FIG. 31, or may display the translucent area label with the input image as any one of UI3210 to UI3240 in FIG. The image may be superimposed and displayed. The translucent method may be any known method. For example, the area label image can be made translucent by setting the transparency of the area label image to a desired value.
　また、出力部２８０７は、領域ラベル画像が学習済モデルを用いて生成されたものである旨や学習済モデルを用いて生成された領域ラベル画像に基づいて行われた画像解析の結果である旨を表示部２８２０に表示させてもよい。さらに、出力部２８０７は、学習済モデルがどのような教師データによって学習を行ったものであるかを示す表示を表示部２８２０に表示させてもよい。当該表示としては、教師データの入力データと出力データの種類の説明や、入力データと出力データに含まれる撮影部位等の教師データに関する任意の表示を含んでよい。 Also, the output unit 2807 indicates that the region label image is generated using the learned model and that the output unit 2807 is a result of image analysis performed based on the region label image generated using the learned model. May be displayed on the display unit 2820. Furthermore, the output unit 2807 may cause the display unit 2820 to display a display indicating what kind of teacher data the learned model has learned. The display may include an explanation of the types of the input data and the output data of the teacher data, and an arbitrary display related to the teacher data such as an imaging part included in the input data and the output data.
　本実施例では、処理可否判定部２８０３が、画像セグメンテーションエンジンによって入力画像が対処可能であると判断したら、処理がステップＳ２９４０に移行して、セグメンテーション処理部２８０４による画像セグメンテーション処理が開始された。これに対し、出力部２８０７が処理可否判定部２８０３による判定結果を表示部２８２０に表示させ、セグメンテーション処理部２８０４が検者からの指示に応じて画像セグメンテーション処理を開始してもよい。この際、出力部２８０７は、判定結果とともに、入力画像や入力画像について取得した撮影部位等の撮影条件を表示部２８２０に表示させることもできる。この場合には、検者によって判定結果が正しいか否か判断された上で、画像セグメンテーション処理が行われるため、誤判定に基づく画像セグメンテーション処理を低減させることができる。 In the present embodiment, when the processing availability determination unit 2803 determines that the input image can be handled by the image segmentation engine, the process proceeds to step S2940, and the image segmentation process by the segmentation processing unit 2804 is started. On the other hand, the output unit 2807 may cause the display unit 2820 to display the determination result by the processing availability determination unit 2803, and the segmentation processing unit 2804 may start the image segmentation process according to an instruction from the examiner. At this time, the output unit 2807 can also display, on the display unit 2820, the input image and the imaging conditions such as the imaging region acquired for the input image, along with the determination result. In this case, the image segmentation process is performed after the examiner determines whether or not the determination result is correct, so that the image segmentation process based on the erroneous determination can be reduced.
　また、これに関連して、処理可否判定部２８０３による判定を行わず、出力部２８０７が入力画像や入力画像について取得した撮影部位等の撮影条件を表示部２８２０に表示させてもよい。この場合には、セグメンテーション処理部２８０４が検者からの指示に応じて画像セグメンテーション処理を開始することができる。 Also, in connection with this, the output unit 2807 may cause the display unit 2820 to display imaging conditions such as an input image and an imaging part acquired for the input image without performing the determination by the processing availability determination unit 2803. In this case, the segmentation processing unit 2804 can start the image segmentation processing according to the instruction from the examiner.
　さらに、本実施例では、評価部２８０５は、知識ベースの評価方法を採用する領域ラベル画像評価エンジンを用いて、セグメンテーション処理部２８０４によって生成された領域ラベル画像を評価した。これに対し、評価部２８０５は、領域ラベル画像と所定の評価手法による画像評価指数を教師データとしてトレーニングを行った学習済モデルを含む領域ラベル画像評価エンジンを用いて、領域ラベル画像が尤もらしい画像か否かを評価してもよい。 Further, in the present embodiment, the evaluation unit 2805 evaluated the area label image generated by the segmentation processing unit 2804 by using an area label image evaluation engine that employs a knowledge-based evaluation method. On the other hand, the evaluation unit 2805 uses an area label image evaluation engine including a trained model in which training is performed using the area label image and an image evaluation index based on a predetermined evaluation method as teacher data, and the area label image is likely to be an image. It may be evaluated whether or not.
　この場合には、領域ラベル画像評価エンジンに含まれる機械学習モデルの教師データは、領域ラベル画像や領域ラベル画像らしい偽画像を入力データとし、各画像に対する画像評価指数を出力データとする。画像評価指数としては、入力データが適切な領域ラベル画像である場合には真値、偽画像である場合には偽値とする。なお、偽画像の生成方法としては、適切でない条件を設定した領域ラベル画像の任意のジェネレーターを用いる方法や、適切な領域ラベル画像に意図的に不適切な領域ラベルを上書きして生成する方法等を採用してよい。 In this case, as the teacher data of the machine learning model included in the region label image evaluation engine, a region label image or a fake image like a region label image is used as input data, and an image evaluation index for each image is used as output data. The image evaluation index is a true value when the input data is an appropriate area label image, and a false value when the input data is a false image. As a method of generating a fake image, a method of using an arbitrary generator of an area label image in which inappropriate conditions are set, a method of intentionally overwriting an appropriate area label image with an inappropriate area label, and the like May be adopted.
　評価部２８０５が、このような学習を行った学習済モデルを含む領域ラベル画像評価エンジンを用いて、領域ラベル画像が尤もらしい画像か否かを評価した場合でも、画像診断や画像解析に不適切な領域ラベル画像が用いられることを防止することができる。 Even when the evaluation unit 2805 evaluates whether or not the region label image is a plausible image using the region label image evaluation engine including the trained model that has performed such learning, it is not suitable for image diagnosis or image analysis. It is possible to prevent a region label image from being used.
　なお、処理可否判定部２８０３によって画像セグメンテーション処理が不可能であると判定された場合において、領域ラベル無し画像は、出力部２８０７によって生成されてもよいし、処理可否判定部２８０３によって生成されてもよい。また、評価部２８０５によって画像セグメンテーションが適切に行われていないと判断された場合には、ステップＳ２９７０において出力部２８０７が、画像セグメンテーションを適切に行えなかった旨を表示部２８２０に表示させてもよい。 Note that when the image segmentation processing is not possible by the processing availability determination unit 2803, the image without region label may be generated by the output unit 2807 or may be generated by the processing availability determination unit 2803. Good. If the evaluation unit 2805 determines that image segmentation has not been performed properly, the output unit 2807 may display on the display unit 2820 that the image segmentation has not been performed properly in step S2970. .
（実施例９）
　次に、図２８及び図３３を参照して、実施例９に係る画像処理装置について説明する。実施例８では、セグメンテーション処理部２８０４は、一つの画像セグメンテーションエンジンを備えていた。これに対して、本実施例では、セグメンテーション処理部は、異なる教師データを用いて機械学習を行ったそれぞれの学習済モデルを含む複数の画像セグメンテーションエンジンを用いて、入力画像に対して複数の領域ラベル画像を生成する。 (Example 9)
Next, an image processing apparatus according to a ninth embodiment will be described with reference to FIGS. In the eighth embodiment, the segmentation processing unit 2804 includes one image segmentation engine. On the other hand, in the present embodiment, the segmentation processing unit uses a plurality of image segmentation engines including respective learned models that have performed machine learning using different teacher data, and uses a plurality of image segmentation engines with respect to the input image. Generate a label image.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例８に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例８に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は、実施例８に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
　本実施例に係るセグメンテーション処理部２８０４は、異なる教師データを用いて機械学習が行われたそれぞれの学習済モデルを含む二つ以上の画像セグメンテーションエンジンを用いて、入力画像に対して画像セグメンテーション処理を行う。 The segmentation processing unit 2804 according to the present embodiment performs image segmentation processing on an input image using two or more image segmentation engines including respective trained models that have been machine-learned using different teacher data. Do.
　ここで、本実施例に係る教師データ群の作成方法について説明する。具体的には、まず、様々な撮影部位が撮影された、入力データとしての画像と出力データとしての領域ラベル画像とのペア群を用意する。次に、撮影部位毎にペア群をグルーピングすることで、教師データ群を作成する。例えば、第一の撮影部位を撮影して取得されたペア群で構成される第一の教師データ、第二の撮影部位を撮影して取得されたペア群で構成される第二の教師データというように、教師データ群を作成する。 Here, a method of creating the teacher data group according to the present embodiment will be described. Specifically, first, a group of pairs of an image as input data and an area label image as output data, in which various imaging parts are imaged, is prepared. Next, a teacher data group is created by grouping a pair group for each imaging region. For example, first teacher data composed of a pair group acquired by imaging the first imaging region, and second teacher data composed of a pair group acquired by imaging the second imaging region. Thus, a teacher data group is created.
　その後、各教師データを用いて、別々の画像セグメンテーションエンジンに含まれる機械学習モデルに機械学習を行わせる。例えば、第一の教師データでトレーニングされた学習済モデルを含む第一の画像セグメンテーションエンジンを用意する。加えて、第二の教師データでトレーニングされた学習済モデルを含む第二の画像セグメンテーションエンジンを用意する、というように画像セグメンテーションエンジン群を用意する。 Then, using each of the teacher data, the machine learning model included in the separate image segmentation engine performs machine learning. For example, a first image segmentation engine including a trained model trained with first teacher data is provided. In addition, an image segmentation engine group is prepared, such as preparing a second image segmentation engine including a trained model trained with the second teacher data.
　このような画像セグメンテーションエンジンは、それぞれが含む学習済モデルのトレーニングに用いた教師データが異なる。そのため、このような画像セグメンテーションエンジンは、画像セグメンテーションエンジンに入力される画像の撮影条件によって、入力画像を画像セグメンテーション処理できる程度が異なる。具体的には、第一の画像セグメンテーションエンジンは、第一の撮影部位を撮影して取得された入力画像に対しては画像セグメンテーション処理の精度が高く、第二の撮影部位を撮影して取得された画像に対しては画像セグメンテーション処理の精度が低い。同様に、第二の画像セグメンテーションエンジンは、第二の撮影部位を撮影して取得された入力画像に対しては画像セグメンテーション処理の精度が高く、第一の撮影部位を撮影して取得された画像に対しては画像セグメンテーション処理の精度が低い。 画像 Such image segmentation engines differ in the teacher data used for training the learned models included in each. Therefore, such an image segmentation engine differs in the degree to which the input image can be subjected to the image segmentation process depending on the imaging conditions of the image input to the image segmentation engine. Specifically, the first image segmentation engine has a high accuracy of the image segmentation process for the input image obtained by imaging the first imaging region, and is obtained by imaging the second imaging region. The accuracy of the image segmentation process is low for images that have been corrupted. Similarly, the second image segmentation engine has a high accuracy of the image segmentation process for an input image obtained by imaging the second imaging region, and obtains an image obtained by imaging the first imaging region. , The accuracy of the image segmentation process is low.
　教師データのそれぞれが撮影部位によってグルーピングされたペア群で構成されることにより、該ペア群を構成する画像群の画質傾向が似る。このため、画像セグメンテーションエンジンは対応する撮影部位であれば、実施例８に係る画像セグメンテーションエンジンよりも精度良く画像セグメンテーション処理を行うことができる。なお、教師データのペアをグルーピングするための撮影条件は、撮影部位に限られず、撮影画角であったり、画像の解像度であったり、これらのうちの二つ以上の組み合わせであったりしてもよい。 (4) Since each of the teacher data is constituted by a group of pairs grouped by the imaged region, the image quality tendencies of the images constituting the group of pairs are similar. For this reason, if the image segmentation engine is a corresponding imaging part, the image segmentation processing can be performed with higher accuracy than the image segmentation engine according to the eighth embodiment. Note that the imaging conditions for grouping pairs of teacher data are not limited to imaging regions, and may be imaging angles of view, image resolutions, or a combination of two or more of these. Good.
　図３３を参照して、本実施例に係る一連の画像処理について説明する。図３３は、本実施例に係る一連の画像処理のフローチャートである。なお、ステップＳ３３１０及びステップＳ３３２０の処理は、実施例８に係るステップＳ２９１０及びステップＳ２９２０と同様であるため、説明を省略する。なお、入力画像に対して、無条件で画像セグメンテーション処理する場合には、ステップＳ３３２０の処理の後に、ステップＳ３３３０の処理を省き、処理をステップＳ３３４０に移行してよい。 A series of image processing according to the present embodiment will be described with reference to FIG. FIG. 33 is a flowchart of a series of image processing according to the present embodiment. Note that the processing in steps S3310 and S3320 is the same as that in steps S2910 and S2920 according to the eighth embodiment, and a description thereof will not be repeated. When the image segmentation process is unconditionally performed on the input image, the process of step S3330 may be omitted after the process of step S3320, and the process may proceed to step S3340.
　ステップＳ３３２０において入力画像の撮影条件が取得されると、処理はステップＳ３３３０に移行する。ステップＳ３３３０においては、処理可否判定部２８０３が、ステップＳ３３２０において取得した撮影条件群を用いて、前述の画像セグメンテーションエンジン群のいずれかが、入力画像を対処可能であるか否かを判定する。 If the shooting condition of the input image is obtained in step S3320, the process proceeds to step S3330. In step S3330, the processing availability determination unit 2803 determines whether any of the above-described image segmentation engine groups can deal with the input image using the shooting condition group acquired in step S3320.
　処理可否判定部２８０３が、画像セグメンテーションエンジン群のいずれも入力画像を対処不可能であると判定した場合には、処理はステップＳ３３８０に移行する。一方で、処理可否判定部２８０３が、画像セグメンテーションエンジン群のいずれかが入力画像を対処可能であると判定した場合には、処理はステップＳ３３４０に移行する。なお、画像処理装置２８００の設定や実装形態によっては、実施例８と同様に、画像セグメンテーションエンジンによって一部の撮影条件が対処不可能であると判定されたとしても、ステップＳ３３４０を実施してもよい。 If the processing availability determination unit 2803 determines that none of the image segmentation engine groups can deal with the input image, the process moves to step S3380. On the other hand, if the process availability determination unit 2803 determines that any of the image segmentation engine groups can handle the input image, the process proceeds to step S3340. Note that, depending on the settings and the implementation form of the image processing apparatus 2800, as in the eighth embodiment, even if the image segmentation engine determines that some shooting conditions cannot be dealt with, even if step S3340 is performed. Good.
　ステップＳ３３４０においては、セグメンテーション処理部２８０４が、ステップＳ３３２０で取得した入力画像の撮影条件及び画像セグメンテーションエンジン群の教師データの情報に基づいて、処理を行う画像セグメンテーションエンジンを選択する。具体的には、例えば、ステップＳ３３２０において取得した撮影条件群のうちの撮影部位に対して、同撮影部位又は周囲の撮影部位に関する教師データの情報を有し、画像セグメンテーション処理の精度が高い画像セグメンテーションエンジンを選択する。上述の例では、撮影部位が第一の撮影部位である場合には、セグメンテーション処理部２８０４は第一の画像セグメンテーションエンジンを選択する。 In step S3340, the segmentation processing unit 2804 selects an image segmentation engine to be processed based on the shooting conditions of the input image acquired in step S3320 and the information of the teacher data of the image segmentation engine group. More specifically, for example, the image segmentation information for the imaging region in the imaging condition group acquired in step S3320 has information of teacher data regarding the imaging region or surrounding imaging regions, and the image segmentation processing has high accuracy. Select an engine. In the above example, when the imaging region is the first imaging region, the segmentation processing unit 2804 selects the first image segmentation engine.
　ステップＳ３３５０では、セグメンテーション処理部２８０４が、ステップＳ３３４０において選択した画像セグメンテーションエンジンを用いて、入力画像を画像セグメンテーション処理した領域ラベル画像を生成する。ステップＳ３３６０及びＳ３３７０は、実施例８におけるステップＳ２９５０及びＳ２９６０と同様であるため、説明を省略する。 In step S3350, the segmentation processing unit 2804 uses the image segmentation engine selected in step S3340 to generate an area label image obtained by subjecting the input image to image segmentation processing. Steps S3360 and S3370 are the same as steps S2950 and S2960 in the eighth embodiment, and a description thereof will be omitted.
　ステップＳ３３７０において画像解析が行われたら処理はステップＳ３３８０に移行する。ステップＳ３３８０では、出力部２８０７は、評価部２８０５によって領域ラベル画像を出力すると判断されたら、領域ラベル画像及び解析結果を出力して、表示部２８２０に表示させる。なお、出力部２８０７は、領域ラベル画像を表示部２８２０に表示させる際、セグメンテーション処理部２８０４によって選択された画像セグメンテーションエンジンを用いて生成された領域ラベル画像である旨を表示させてもよい。なお、出力部２８０７は、領域ラベル画像及び解析結果のいずれか一方のみを出力してもよい。 If the image analysis is performed in step S3370, the process proceeds to step S3380. In step S3380, when the evaluation unit 2805 determines that the area label image is to be output, the output unit 2807 outputs the area label image and the analysis result, and causes the display unit 2820 to display the image. When displaying the region label image on the display unit 2820, the output unit 2807 may display a message indicating that the region label image is a region label image generated by using the image segmentation engine selected by the segmentation processing unit 2804. Note that the output unit 2807 may output only one of the area label image and the analysis result.
　一方、ステップＳ３３３０において画像セグメンテーション処理が不可能であるとされていた場合には、出力部２８０７は、領域ラベル画像の一種である領域ラベル無し画像を出力し、表示部２８２０に表示させる。なお、領域ラベル無し画像を生成する代わりに、撮影装置２８１０に対して、画像セグメンテーション処理が不可能であったことを示す信号を送信してもよい。 On the other hand, if it is determined in step S3330 that the image segmentation process cannot be performed, the output unit 2807 outputs an image without an area label, which is a type of area label image, and causes the display unit 2820 to display the image. Note that, instead of generating an image without a region label, a signal indicating that the image segmentation processing has been impossible may be transmitted to the imaging device 2810.
　また、ステップＳ３３６０において画像セグメンテーション処理を適切に行えなかったと判断された場合にも、出力部２８０７は、領域ラベル画像の一種である領域ラベル無し画像を出力し、表示部２８２０に表示させる。この場合にも、出力部２８０７は、領域ラベル無し画像を生成する代わりに、撮影装置２８１０に対して、画像セグメンテーション処理を適切に行えなかったことを示す信号を送信してもよい。ステップＳ３３８０における出力処理が終了すると、一連の画像処理が終了する。 Also, when it is determined in step S3360 that the image segmentation process has not been properly performed, the output unit 2807 outputs an image without an area label, which is a type of area label image, and causes the display unit 2820 to display the image. In this case as well, the output unit 2807 may transmit a signal indicating that the image segmentation processing has not been properly performed to the imaging device 2810 instead of generating an image without an area label. When the output processing in step S3380 ends, a series of image processing ends.
　上記のように、本実施例に係るセグメンテーション処理部２８０４は、異なる学習データを用いて学習を行ったそれぞれの学習済モデルを含む複数の画像セグメンテーションエンジンの少なくとも一つを用いて、領域ラベル画像を生成する。本実施例では、複数の画像セグメンテーションエンジンの各々は、それぞれ撮影部位、撮影画角、及び画像の解像度のうちの少なくとも一つについての異なる学習データを用いて学習を行った学習済モデルを含む。セグメンテーション処理部２８０４は、入力画像の撮影部位、撮影画角、及び画像の解像度のうちの少なくとも一つに応じた画像セグメンテーションエンジンを用いて、領域ラベル画像を生成する。 As described above, the segmentation processing unit 2804 according to the present embodiment uses at least one of a plurality of image segmentation engines including each trained model that has performed learning using different learning data to generate a region label image. Generate. In this embodiment, each of the plurality of image segmentation engines includes a trained model that has been trained using different learning data for at least one of the imaging region, imaging angle of view, and image resolution. The segmentation processing unit 2804 generates an area label image using an image segmentation engine corresponding to at least one of the imaging region of the input image, the imaging angle of view, and the resolution of the image.
　このような構成により、本実施例に係る画像処理装置２８００は、撮影条件に合わせて、より精度の高い画像セグメンテーション処理を実施することができる。 With such a configuration, the image processing device 2800 according to the present embodiment can perform more accurate image segmentation processing in accordance with the shooting conditions.
　本実施例では、セグメンテーション処理部２８０４が、入力画像の撮影条件に基づいて画像セグメンテーション処理に用いる画像セグメンテーションエンジンを選択したが、画像セグメンテーションエンジンの選択処理はこれに限られない。例えば、出力部２８０７が、取得した入力画像の撮影条件と画像セグメンテーションエンジン群を表示部２８２０のユーザーインターフェースに表示させてもよい。さらに、検者からの指示に応じて、セグメンテーション処理部２８０４が画像セグメンテーション処理に用いる画像セグメンテーションエンジンを選択してもよい。 In the present embodiment, the segmentation processing unit 2804 selects the image segmentation engine used for the image segmentation processing based on the shooting conditions of the input image, but the selection processing of the image segmentation engine is not limited to this. For example, the output unit 2807 may cause the user interface of the display unit 2820 to display the shooting conditions of the acquired input image and the group of image segmentation engines. Further, the segmentation processing unit 2804 may select an image segmentation engine to be used for the image segmentation process according to an instruction from the examiner.
　なお、出力部２８０７は、画像セグメンテーションエンジン群とともに各画像セグメンテーションエンジンの学習に用いた教師データの情報を表示部２８２０に表示させてもよい。なお、画像セグメンテーションエンジンの学習に用いた教師データの情報の表示態様は任意であってよく、例えば、学習に用いた教師データに関連する名称を用いて画像セグメンテーションエンジン群を表示してもよい。 Note that the output unit 2807 may cause the display unit 2820 to display information of teacher data used for learning of each image segmentation engine together with the image segmentation engine group. The display mode of the information of the teacher data used for the learning of the image segmentation engine may be arbitrary. For example, the group of the image segmentation engines may be displayed using the name related to the teacher data used for the learning.
　また、出力部２８０７が、セグメンテーション処理部２８０４によって選択された画像セグメンテーションエンジンを表示部２８２０のユーザーインターフェースに表示させ、検者からの指示を受け付けてもよい。この場合、セグメンテーション処理部２８０４は、検者からの指示に応じて、当該画像セグメンテーションエンジンを画像セグメンテーション処理に用いる画像セグメンテーションエンジンとして最終的に選択するか否かを判断してもよい。 The output unit 2807 may display the image segmentation engine selected by the segmentation processing unit 2804 on the user interface of the display unit 2820, and may receive an instruction from the examiner. In this case, the segmentation processing unit 2804 may determine whether or not to finally select the image segmentation engine as an image segmentation engine to be used for the image segmentation process, according to an instruction from the examiner.
　なお、出力部２８０７は、実施例８と同様に、生成された領域ラベル画像や評価結果を撮影装置２８１０や画像処理装置２８００に接続される他の装置に出力してもよい。また、出力部２８０７は、画像処理装置２８００の設定や実装形態によっては、これらを撮影装置２８１０や他の装置が利用可能なように加工したり、画像管理システム等に送信可能なようにデータ形式を変換したりしてもよい。 Note that the output unit 2807 may output the generated region label image and the evaluation result to another device connected to the imaging device 2810 or the image processing device 2800, as in the eighth embodiment. The output unit 2807 may process the image processing device 2800 so that it can be used by the image capturing device 2810 or another device, or may transmit the data to an image management system or the like, depending on the setting or implementation form of the image processing device 2800. May be converted.
（実施例１０）
　次に、図２８及び図３３を参照して、実施例１０に係る画像処理装置について説明する。実施例８及び９では、撮影条件取得部２８０２は、入力画像のデータ構造等から撮影条件群を取得する。これに対して、本実施例では、撮影条件取得部は、撮影箇所推定エンジンを用いて、入力画像の撮影部位又は撮影領域を入力画像に基づいて推定する。 (Example 10)
Next, an image processing apparatus according to the tenth embodiment will be described with reference to FIGS. In the eighth and ninth embodiments, the imaging condition acquisition unit 2802 acquires the imaging condition group from the data structure of the input image and the like. On the other hand, in the present embodiment, the imaging condition acquisition unit estimates the imaging region or the imaging region of the input image based on the input image using the imaging position estimation engine.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例９に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例９に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は、実施例８及び９に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the ninth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the ninth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatuses according to the eighth and ninth embodiments, the configuration illustrated in FIG. I do.
　本実施例に係る撮影条件取得部２８０２は、撮影箇所推定エンジン（推定エンジン）を用いて、取得部２８０１が取得した入力画像に描画されている撮影部位又は撮影領域を推定し、取得する。本実施例に係る撮影箇所推定エンジンの備える撮影箇所の推定手法では、機械学習アルゴリズムを用いた推定処理を行う。 The imaging condition acquisition unit 2802 according to the present embodiment estimates and acquires an imaging part or an imaging region drawn on the input image acquired by the acquisition unit 2801 using an imaging location estimation engine (estimation engine). In the method for estimating a shooting location included in the shooting location estimation engine according to the present embodiment, an estimation process using a machine learning algorithm is performed.
　本実施例では、機械学習アルゴリズムを用いた撮影箇所推定手法に係る学習済モデルのトレーニングには、画像である入力データと、入力データに対応する撮影部位ラベルである出力データとのペア群で構成された教師データを用いる。ここで、入力データとは、処理対象（入力画像）として想定される特定の撮影条件を持つ画像のことである。入力データとしては、撮影装置２８１０と同じ画質傾向を持つ撮影装置から取得された画像であることが好ましく、撮影装置２８１０と同じ設定をされた同じ機種であるとより良い。出力データである撮影部位ラベルの種類は、入力データに少なくとも一部が含まれている撮影部位であってよい。出力データである撮影部位ラベルの種類は、例えば、“黄斑部”、“視神経乳頭部”、“黄斑部及び視神経乳頭部”、並びに“その他”等であってよい。 In the present embodiment, the training of the learned model according to the imaging location estimation method using a machine learning algorithm includes a pair group of input data which is an image and output data which is an imaging part label corresponding to the input data. Use the obtained teacher data. Here, the input data is an image having a specific shooting condition assumed as a processing target (input image). The input data is preferably an image acquired from a photographing device having the same image quality tendency as the photographing device 2810, and more preferably the same model having the same settings as the photographing device 2810. The type of the imaging region label that is the output data may be an imaging region in which at least a part is included in the input data. The type of the imaging region label that is the output data may be, for example, “macular part”, “optic nerve head”, “macular and optic nerve head”, and “other”.
　本実施例に係る撮影箇所推定エンジンは、このような教師データを用いた学習を行った学習済モデルを含むことにより、入力された画像に描出されている撮影部位や撮影領域がどこであるかを出力することができる。また、撮影箇所推定エンジンは、必要な詳細レベルの撮影部位ラベルや撮影領域ラベル毎に、該撮影部位や撮影領域である確率を出力することもできる。 The imaging location estimation engine according to the present embodiment includes a learned model that has performed learning using such teacher data to determine where the imaging region and the imaging region depicted in the input image are. Can be output. In addition, the imaging location estimation engine can also output, for each required imaging level label or imaging area label at a required level of detail, the probability of being the imaging area or imaging area.
　撮影箇所推定エンジンを用いることで、撮影条件取得部２８０２は、入力画像に基づいて、入力画像の撮影部位や撮影領域を推定し、入力画像についての撮影条件として取得することができる。なお、撮影箇所推定エンジンが撮影部位ラベルや撮影領域ラベル毎に、該撮影部位や撮影領域である確率を出力する場合には、撮影条件取得部２８０２は、最も確率の高い撮影部位や撮影領域を入力画像の撮影条件として取得する。 By using the imaging location estimation engine, the imaging condition acquiring unit 2802 can estimate the imaging region and the imaging region of the input image based on the input image, and can acquire it as the imaging condition for the input image. When the imaging location estimation engine outputs, for each imaging region label or imaging region label, the probability of the imaging region or imaging region, the imaging condition acquisition unit 2802 determines the imaging region or imaging region with the highest probability. It is acquired as the shooting condition of the input image.
　次に、実施例９と同様に、図３３のフローチャートを参照して、本実施例に係る一連の画像処理について説明する。なお、本実施例に係るステップＳ３３１０、及びステップＳ３３３０～ステップＳ３３８０の処理は、実施例９におけるこれらの処理と同様であるため、説明を省略する。なお、入力画像に対して、無条件で画像セグメンテーション処理する場合には、ステップＳ３３２０の処理の後に、ステップＳ３３３０の処理を省き、処理をステップＳ３３４０に移行してよい。 Next, similarly to the ninth embodiment, a series of image processing according to the present embodiment will be described with reference to the flowchart in FIG. Note that the processes of step S3310 and steps S3330 to S3380 according to the present embodiment are the same as these processes in the ninth embodiment, and thus description thereof will be omitted. When the image segmentation process is unconditionally performed on the input image, the process of step S3330 may be omitted after the process of step S3320, and the process may proceed to step S3340.
　ステップＳ３３１０において入力画像が取得されると、処理はステップＳ３３２０に移行する。ステップＳ３３２０では、撮影条件取得部２８０２が、ステップＳ３３１０において取得した入力画像の撮影条件群を取得する。 If the input image is obtained in step S3310, the process proceeds to step S3320. In step S3320, the imaging condition acquisition unit 2802 acquires the imaging condition group of the input image acquired in step S3310.
　具体的には、入力画像のデータ形式に応じて、入力画像を構成するデータ構造に保存された撮影条件群を取得する。また、撮影条件群に撮影部位や撮影領域に関する情報が含まれていない場合、撮影条件取得部２８０２は撮影箇所推定エンジンに入力画像を入力し、入力画像がどの撮影部位・撮影領域を撮影して取得されたものなのかを推定する。具体的には、撮影条件取得部２８０２は、撮影箇所推定エンジンに入力画像を入力し、撮影部位ラベル群又は撮影領域ラベル群のそれぞれに対して出力された確率を評価し、最も確率の高い撮影部位又は撮影領域を入力画像の撮影条件として設定・取得する。 Specifically, according to the data format of the input image, a group of photographing conditions stored in a data structure constituting the input image is obtained. If the imaging condition group does not include information on the imaging region or the imaging region, the imaging condition acquisition unit 2802 inputs the input image to the imaging location estimation engine, and captures the imaging region or imaging region of the input image. Estimate whether it was acquired. Specifically, the imaging condition acquisition unit 2802 inputs an input image to the imaging location estimation engine, evaluates the probability output for each of the imaging region label group or the imaging region label group, and determines the imaging probability with the highest probability. A part or an imaging region is set and acquired as an imaging condition of an input image.
　なお、入力画像に撮影部位や撮影領域以外の撮影条件が保存されていない場合には、撮影条件取得部２８０２は、撮影装置２８１０や不図示の画像管理システムから撮影情報群を取得することができる。以降の処理は、実施例９に係る一連の画像処理と同様であるため説明を省略する。 When the input image does not store the imaging conditions other than the imaging region and the imaging region, the imaging condition acquisition unit 2802 can acquire the imaging information group from the imaging device 2810 or an image management system (not shown). . Subsequent processing is the same as a series of image processing according to the ninth embodiment, and a description thereof will not be repeated.
　上記のように、本実施例に係る撮影条件取得部２８０２は、学習済モデルを含む撮影箇所推定エンジンを用いて、入力画像から撮影部位及び撮影領域のうちの少なくとも一方を推定する推定部として機能する。撮影条件取得部２８０２は、撮影部位や撮影領域のラベルが付けられた画像を学習データとした学習済モデルを含む撮影箇所推定エンジンに入力画像を入力することで、入力画像の撮影部位や撮影領域を推定する。 As described above, the imaging condition acquisition unit 2802 according to the present embodiment functions as an estimation unit that estimates at least one of an imaging region and an imaging region from an input image using the imaging location estimation engine including the learned model. I do. The imaging condition acquisition unit 2802 inputs an input image to an imaging location estimation engine including a learned model in which an image to which an imaging region and an imaging region are labeled is used as learning data, thereby obtaining an imaging region and an imaging region of the input image. Is estimated.
　これにより、本実施例に係る画像処理装置２８００は、入力画像の撮影部位や撮影領域についての撮影条件を入力画像に基づいて取得することができる。 Accordingly, the image processing apparatus 2800 according to the present embodiment can acquire the imaging conditions for the imaging region and the imaging region of the input image based on the input image.
　なお、本実施例では、撮影条件取得部２８０２は、撮影条件群に撮影部位や撮影領域に関する情報が含まれていない場合に撮影箇所推定エンジンを用いて入力画像の撮影部位や撮影領域について推定を行った。しかしながら、撮影箇所推定エンジンを用いて撮影部位や撮影領域について推定を行う状況はこれに限られない。撮影条件取得部２８０２は、入力画像のデータ構造に含まれる撮影部位や撮影領域についての情報が、必要な詳細レベルの情報として不足している場合にも、撮影箇所推定エンジンを用いて撮影部位や撮影領域について推定を行ってもよい。 In the present embodiment, the imaging condition acquisition unit 2802 estimates the imaging region and the imaging region of the input image using the imaging region estimation engine when the imaging condition group does not include information on the imaging region and the imaging region. went. However, the situation in which the imaging site and the imaging region are estimated using the imaging location estimation engine is not limited to this. The imaging condition acquisition unit 2802 can use the imaging location estimation engine to determine the imaging site and the imaging region even when the information on the imaging region and the imaging region included in the data structure of the input image is insufficient as the required level of detail. The estimation may be performed on the shooting area.
　また、入力画像のデータ構造に撮影部位や撮影領域についての情報が含まれているか否かとは無関係に、撮影条件取得部２８０２が撮影箇所推定エンジンを用いて入力画像の撮影部位や撮影領域を推定してもよい。この場合、出力部２８０７が、撮影箇所推定エンジンから出力された推定結果と入力画像のデータ構造に含まれる撮影部位や撮影領域についての情報を表示部２８２０に表示させ、撮影条件取得部２８０２が検者の指示に応じて、これらの撮影条件を決定してもよい。 Further, regardless of whether or not the data structure of the input image includes information on the imaging region and the imaging region, the imaging condition obtaining unit 2802 estimates the imaging region and the imaging region of the input image using the imaging region estimation engine. May be. In this case, the output unit 2807 causes the display unit 2820 to display the estimation result output from the imaging location estimation engine and information about the imaging region and the imaging region included in the data structure of the input image, and the imaging condition acquisition unit 2802 performs the inspection. These photographing conditions may be determined according to a user's instruction.
（実施例１１）
　次に、図２８、図２９、図３４及び図３５を参照して、実施例１１に係る画像処理装置について説明する。本実施例では、セグメンテーション処理部が、入力画像を画像セグメンテーションエンジンが対処可能な画像サイズになるように、入力画像を拡大又は縮小する。また、セグメンテーション処理部は、画像セグメンテーションエンジンからの出力画像を、入力画像の画像サイズになるように縮小又は拡大して領域ラベル画像を生成する。 (Example 11)
Next, an image processing apparatus according to the eleventh embodiment will be described with reference to FIGS. 28, 29, 34, and 35. In the present embodiment, the segmentation processing unit enlarges or reduces the input image so that the input image has an image size that can be handled by the image segmentation engine. Further, the segmentation processing unit generates an area label image by reducing or enlarging the output image from the image segmentation engine so as to have the image size of the input image.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例８に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例８に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は、実施例８に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
　本実施例に係るセグメンテーション処理部２８０４には、実施例８に係る画像セグメンテーションエンジンと同様の、画像セグメンテーションエンジンが備えられている。ただし、本実施例では、画像セグメンテーションエンジンが含む機械学習モデルの学習に用いる教師データが実施例８における教師データと異なる。具体的には、本実施例では、教師データとして、入力データの画像及び出力データの画像を一定の画像サイズになるように拡大又は縮小した画像群により構成した、入力データと出力データのペア群を用いている。 セ The segmentation processing unit 2804 according to the present embodiment includes an image segmentation engine similar to the image segmentation engine according to the eighth embodiment. However, in the present embodiment, the teacher data used for learning the machine learning model included in the image segmentation engine is different from the teacher data in the eighth embodiment. Specifically, in the present embodiment, a pair group of input data and output data is configured as teacher data by a group of images obtained by enlarging or reducing an image of input data and an image of output data so as to have a fixed image size. Is used.
　ここで、図３４を参照して、本実施例に係る画像セグメンテーションエンジンが含む学習済モデルの教師データについて説明する。図３４に示すように、例えば、教師データについて設定された一定の画像サイズより小さな入力画像Ｉｍ３４１０と領域ラベル画像Ｉｍ３４２０とがある場合を考える。この場合、教師データについて設定された一定の画像サイズとなるように、入力画像Ｉｍ３４１０及び領域ラベル画像Ｉｍ３４２０のそれぞれを拡大する。そして、拡大した画像Ｉｍ３４１１と拡大した領域ラベル画像Ｉｍ３４２１とをペアとして、当該ペアを教師データの一つとして用いる。 Here, with reference to FIG. 34, the teacher data of the learned model included in the image segmentation engine according to the present embodiment will be described. As shown in FIG. 34, for example, consider a case where there are an input image Im3410 and an area label image Im3420 smaller than a certain image size set for teacher data. In this case, each of the input image Im3410 and the region label image Im3420 is enlarged so as to have a fixed image size set for the teacher data. Then, the enlarged image Im3411 and the enlarged region label image Im3421 are paired, and the pair is used as one of the teacher data.
　なお、実施例８と同様に、教師データの入力データには、処理対象（入力画像）として想定される特定の撮影条件を持つ画像を用いるが、当該特定の撮影条件は、予め決定された撮影部位、撮影方式、及び撮影画角である。つまり、本実施例に係る当該特定の撮影条件には、実施例８と異なり、画像サイズは含まれない。 As in the eighth embodiment, the input data of the teacher data is an image having a specific shooting condition assumed as a processing target (input image), and the specific shooting condition is a predetermined shooting condition. The part, the imaging method, and the imaging angle of view. That is, unlike the eighth embodiment, the specific photographing condition according to the present embodiment does not include the image size.
　本実施例に係るセグメンテーション処理部２８０４は、このような教師データで学習が行われた画像セグメンテーションエンジンを用いて、入力画像を画像セグメンテーション処理して領域ラベル画像を生成する。この際、セグメンテーション処理部２８０４は、入力画像を教師データについて設定された一定の画像サイズになるように拡大又は縮小した変形画像を生成し、変形画像を画像セグメンテーションエンジンに入力する。 セ The segmentation processing unit 2804 according to the present embodiment generates an area label image by performing image segmentation processing on an input image using an image segmentation engine that has been trained with such teacher data. At this time, the segmentation processing unit 2804 generates a deformed image in which the input image is enlarged or reduced so as to have a fixed image size set for the teacher data, and inputs the deformed image to the image segmentation engine.
　また、セグメンテーション処理部２８０４は、画像セグメンテーションエンジンからの出力画像を入力画像の画像サイズになるように縮小又は拡大し、領域ラベル画像を生成する。このため、本実施例に係るセグメンテーション処理部２８０４は、実施例８では対処できなかった画像サイズの入力画像であっても、画像セグメンテーションエンジンを用いて画像セグメンテーション処理して領域ラベル画像を生成することができる。 {Circle around (2)} The segmentation processing unit 2804 reduces or enlarges the output image from the image segmentation engine so as to have the image size of the input image, and generates an area label image. For this reason, the segmentation processing unit 2804 according to the present embodiment generates an area label image by performing an image segmentation process using an image segmentation engine even for an input image having an image size that cannot be dealt with in the eighth embodiment. Can be.
　次に、図２９及び図３５を参照して、本実施例に係る一連の画像処理について説明する。図３５は、本実施例に係るセグメンテーション処理のフローチャートである。なお、本実施例に係るステップＳ２９１０、ステップＳ２９２０、及びステップＳ２９５０～ステップＳ２９７０の処理は、実施例８におけるこれらの処理と同様であるため、説明を省略する。なお、入力画像に対して、画像サイズ以外の撮影条件について無条件で画像セグメンテーション処理する場合には、ステップＳ２９２０の処理の後に、ステップＳ２９３０の処理を省き、処理をステップＳ２９４０に移行してよい。 Next, a series of image processing according to the present embodiment will be described with reference to FIGS. FIG. 35 is a flowchart of the segmentation process according to the embodiment. Note that the processes in steps S2910, S2920, and steps S2950 to S2970 according to the present embodiment are the same as those in the eighth embodiment, and thus description thereof will be omitted. If the input image is subjected to the image segmentation processing unconditionally for the photographing conditions other than the image size, the processing in step S2930 may be omitted after the processing in step S2920, and the processing may proceed to step S2940.
　ステップＳ２９２０において、実施例８と同様に、撮影条件取得部２８０２が入力画像の撮影条件群を取得したら処理はステップＳ２９３０に移行する。ステップＳ２９３０では、処理可否判定部２８０３が、取得された撮影条件群を用いて、画像セグメンテーションエンジンによって入力画像を対処可能であるか否かを判定する。具体的には、処理可否判定部２８０３は、入力画像の撮影条件について、画像セグメンテーションエンジンが対処可能な、撮影部位、撮影方式、及び撮影画角であるか否かを判定する。処理可否判定部２８０３は、実施例８と異なり、画像サイズは判定しない。 In step S2920, similarly to the eighth embodiment, when the shooting condition obtaining unit 2802 obtains the shooting condition group of the input image, the process proceeds to step S2930. In step S2930, the processing possibility determination unit 2803 determines whether the input image can be handled by the image segmentation engine using the acquired shooting condition group. More specifically, the processing availability determination unit 2803 determines whether or not the imaging condition of the input image is an imaging region, an imaging method, and an imaging angle of view that can be handled by the image segmentation engine. Unlike the eighth embodiment, the processing availability determination unit 2803 does not determine the image size.
　処理可否判定部２８０３が、撮影部位、撮影方式、及び撮影画角について判定し、入力画像が対処可能と判定された場合には、処理はステップＳ２９４０に移行する。一方、処理可否判定部２８０３が、これら撮影条件に基づいて、画像セグメンテーションエンジンが入力画像を対処不可能であると判定した場合には、処理はステップＳ２９７０に移行する。なお、画像処理装置２８００の設定や実装形態によっては、撮影部位、撮影方式、及び撮影画角のうちの一部に基づいて入力画像が処理不可能であると判定されたとしても、ステップＳ２９４０における画像セグメンテーション処理が実施されてもよい。 (4) The processing availability determination unit 2803 determines the imaging region, the imaging method, and the imaging angle of view, and if it is determined that the input image can be handled, the process proceeds to step S2940. On the other hand, when the processability determination unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970. Note that, depending on the setting and the implementation form of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on a part of the imaging part, the imaging method, and the imaging angle of view, it is determined in step S2940 that the input image cannot be processed. An image segmentation process may be performed.
　処理がステップＳ２９４０に移行すると、図３５に示される本実施例に係る画像セグメンテーション処理が開始される。本実施例に係る画像セグメンテーション処理では、まず、ステップＳ３５１０において、セグメンテーション処理部２８０４が、入力画像を教師データについて設定された一定の画像サイズに拡大又は縮小し、変形画像を生成する。 When the process proceeds to step S2940, the image segmentation process according to the embodiment shown in FIG. 35 is started. In the image segmentation process according to this embodiment, first, in step S3510, the segmentation processing unit 2804 enlarges or reduces the input image to a fixed image size set for the teacher data, and generates a deformed image.
　次に、ステップＳ３５２０において、セグメンテーション処理部２８０４は、生成した変形画像を画像セグメンテーションエンジンに入力し、画像セグメンテーション処理された第一の領域ラベル画像を取得する。 Next, in step S3520, the segmentation processing unit 2804 inputs the generated deformed image to the image segmentation engine, and acquires a first region label image that has been subjected to image segmentation.
　その後、ステップＳ３５３０において、セグメンテーション処理部２８０４は、第一の領域ラベル画像を入力画像の画像サイズに縮小又は拡大し、最終的な領域ラベル画像を生成する。セグメンテーション処理部２８０４がステップＳ３５３０において最終的な領域ラベル画像を生成したら、本実施例に係る画像セグメンテーション処理は終了し、処理はステップＳ２９５０に移行する。ステップＳ２９５０以降の処理は、実施例８のステップＳ２９５０以降の処理と同様であるため説明を省略する。 Then, in step S3530, the segmentation processing unit 2804 reduces or enlarges the first region label image to the image size of the input image, and generates a final region label image. When the segmentation processing unit 2804 generates the final area label image in step S3530, the image segmentation processing according to the present embodiment ends, and the processing shifts to step S2950. The processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, and a description thereof will not be repeated.
　上記のように、本実施例に係るセグメンテーション処理部２８０４は、入力画像の画像サイズを、画像セグメンテーションエンジンが対処可能な画像サイズに調整して画像セグメンテーションエンジンに入力する。また、セグメンテーション処理部２８０４は、画像セグメンテーションエンジンからの出力画像を入力画像の元の画像サイズに調整することで領域ラベル画像を生成する。これにより、本実施例の画像処理装置２８００は、実施例８では対応しなかった画像サイズの入力画像についても画像セグメンテーション処理して、画像診断や画像解析に利用可能なＲＯＩやＶＯＩの情報を含む領域ラベル画像を生成することができる。 As described above, the segmentation processing unit 2804 according to the present embodiment adjusts the image size of the input image to an image size that can be handled by the image segmentation engine, and inputs the image size to the image segmentation engine. In addition, the segmentation processing unit 2804 generates an area label image by adjusting the output image from the image segmentation engine to the original image size of the input image. As a result, the image processing apparatus 2800 according to the present embodiment also performs image segmentation processing on an input image having an image size not supported in the eighth embodiment, and includes information on ROIs and VOIs that can be used for image diagnosis and image analysis. An area label image can be generated.
（実施例１２）
　次に、図２８、図２９、図３６及び図３７を参照して、実施例１２に係る画像処理装置について説明する。本実施例では、セグメンテーション処理部が、画像セグメンテーションエンジンによる一定の解像度を基準とした画像セグメンテーション処理により領域ラベル画像を生成する。 (Example 12)
Next, an image processing apparatus according to the twelfth embodiment will be described with reference to FIGS. 28, 29, 36, and 37. In the present embodiment, the segmentation processing unit generates an area label image by image segmentation processing based on a certain resolution by the image segmentation engine.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例８に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例８に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は、実施例８に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
　本実施例に係るセグメンテーション処理部２８０４には、実施例８と同様の、画像セグメンテーションエンジンが備えられている。ただし、本実施例では、画像セグメンテーションエンジンが含む機械学習モデルの学習に用いる教師データが実施例８における教師データと異なる。具体的には、教師データの入力データと出力データとのペア群を構成する画像群の解像度が一定の解像度となるような画像サイズに当該画像群を拡大又は縮小した後、十分に大きい一定の画像サイズとなるようにパディングしている。ここで、画像群の解像度とは、例えば、撮影装置の空間分解能や撮影領域に対する解像度をいう。 セ The segmentation processing unit 2804 according to the present embodiment includes an image segmentation engine similar to that of the eighth embodiment. However, in the present embodiment, the teacher data used for learning the machine learning model included in the image segmentation engine is different from the teacher data in the eighth embodiment. Specifically, after enlarging or reducing the image group to an image size such that the resolution of the image group forming the pair group of the input data and the output data of the teacher data becomes a constant resolution, a sufficiently large constant Padded to the image size. Here, the resolution of the image group refers to, for example, the spatial resolution of the imaging device and the resolution for the imaging region.
　ここで、図３６を参照して、本実施例に係る画像セグメンテーションエンジンの教師データについて説明する。図３６に示すように、例えば、教師データについて設定された一定の解像度より低い解像度を持つ画像Ｉｍ３６１０と領域ラベル画像Ｉｍ３６２０とがある場合を考える。この場合、教師データについて設定された一定の解像度となるように、画像Ｉｍ３６１０と領域ラベル画像Ｉｍ３６２０のそれぞれを拡大する。さらに、拡大された画像Ｉｍ３６１０と領域ラベル画像Ｉｍ３６２０のそれぞれに対して、教師データについて設定された一定の画像サイズとなるようにパディングする。そして、拡大及びパディングが行われた画像Ｉｍ３６１１と領域ラベル画像Ｉｍ３６２１とをペアとし、当該ペアを教師データの一つとして用いる。 Here, with reference to FIG. 36, the teacher data of the image segmentation engine according to the present embodiment will be described. As shown in FIG. 36, for example, consider a case where there are an image Im3610 and a region label image Im3620 having a resolution lower than a certain resolution set for teacher data. In this case, each of the image Im3610 and the region label image Im3620 is enlarged so as to have a fixed resolution set for the teacher data. Furthermore, padding is performed on each of the enlarged image Im3610 and the region label image Im3620 so as to have a fixed image size set for the teacher data. Then, the image Im3611 that has been enlarged and padded is paired with the region label image Im3621, and the pair is used as one of the teacher data.
　なお、教師データについて設定された一定の画像サイズとは、処理対象（入力画像）として想定される画像を一定の解像度となるように拡大又は縮小したときの最大となりうる画像サイズである。当該一定の画像サイズが十分に大きくない場合には、画像セグメンテーションエンジンに入力された画像を拡大したときに、学習済モデルが対処不可能な画像サイズとなる可能性がある。 The fixed image size set for the teacher data is the maximum image size when an image assumed as a processing target (input image) is enlarged or reduced to have a certain resolution. If the certain image size is not large enough, when the image input to the image segmentation engine is enlarged, there is a possibility that the learned model has an image size that cannot be dealt with.
　また、パディングが行われる領域は、効果的に画像セグメンテーション処理できるように学習済モデルの特性に合わせて、一定の画素値で埋めたり、近傍画素値で埋めたり、ミラーパディングしたりする。なお、実施例８と同様に、入力データには、処理対象として想定される特定の撮影条件を持つ画像を用いるが、当該特定の撮影条件は、予め決定された撮影部位、撮影方式、撮影画角である。つまり、本実施例に係る当該特定の撮影条件には、実施例８と異なり、画像サイズは含まれない。 領域 In addition, the area where padding is performed is padded with a fixed pixel value, padded with neighboring pixel values, or mirror-padded according to the characteristics of the learned model so that image segmentation processing can be performed effectively. Note that, similarly to the eighth embodiment, an image having a specific imaging condition assumed as a processing target is used as input data, and the specific imaging condition includes a predetermined imaging part, imaging method, and imaging image. Is the corner. That is, unlike the eighth embodiment, the specific photographing condition according to the present embodiment does not include the image size.
　本実施例に係るセグメンテーション処理部２８０４は、このような教師データで学習が行われた学習済モデルを含む画像セグメンテーションエンジンを用いて、入力画像を画像セグメンテーション処理して領域ラベル画像を生成する。この際、セグメンテーション処理部２８０４は、入力画像を教師データについて設定された一定の解像度になるように拡大又は縮小した変形画像を生成する。また、セグメンテーション処理部２８０４は、変形画像について、教師データについて設定された一定の画像サイズとなるようにパディングを行ってパディング画像を生成し、パディング画像を画像セグメンテーションエンジンに入力する。 セ The segmentation processing unit 2804 according to the present embodiment performs an image segmentation process on an input image using an image segmentation engine including a trained model trained with such teacher data to generate a region label image. At this time, the segmentation processing unit 2804 generates a deformed image obtained by enlarging or reducing the input image so as to have a fixed resolution set for the teacher data. In addition, the segmentation processing unit 2804 performs padding on the deformed image so as to have a fixed image size set for the teacher data, generates a padded image, and inputs the padded image to the image segmentation engine.
　また、セグメンテーション処理部２８０４は、画像セグメンテーションエンジンから出力された第一の領域ラベル画像について、パディングを行った領域分だけトリミングし、第二の領域ラベル画像を生成する。その後、セグメンテーション処理部２８０４は、生成した第二の領域ラベル画像を入力画像の画像サイズになるように縮小又は拡大し、最終的な領域ラベル画像を生成する。 {Circle around (2)} The segmentation processing unit 2804 trims the first region label image output from the image segmentation engine by the padded region to generate a second region label image. After that, the segmentation processing unit 2804 reduces or enlarges the generated second region label image to have the image size of the input image, and generates a final region label image.
　このため、本実施例に係るセグメンテーション処理部２８０４は、実施例８では対処できなかった画像サイズの入力画像であっても、画像セグメンテーションエンジンによって画像セグメンテーション処理して領域ラベル画像を生成することができる。 For this reason, the segmentation processing unit 2804 according to the present embodiment can generate an area label image by performing an image segmentation process using an image segmentation engine even for an input image having an image size that cannot be dealt with in the eighth embodiment. .
　次に、図２９及び図３７を参照して、本実施例に係る一連の画像処理について説明する。図３７は、本実施例に係る画像セグメンテーション処理のフローチャートである。なお、本実施例に係るステップＳ２９１０、ステップＳ２９２０、及びステップＳ２９５０～ステップＳ２９７０の処理は、実施例８におけるこれらの処理と同様であるため、説明を省略する。なお、入力画像に対して、画像サイズ以外の撮影条件について無条件で画像セグメンテーション処理する場合には、ステップＳ２９２０の処理の後に、ステップＳ２９３０の処理を省き、処理をステップＳ２９４０に移行してよい。 Next, a series of image processing according to the present embodiment will be described with reference to FIGS. FIG. 37 is a flowchart of the image segmentation process according to the embodiment. Note that the processes in steps S2910, S2920, and steps S2950 to S2970 according to the present embodiment are the same as those in the eighth embodiment, and thus description thereof will be omitted. When the image segmentation process is performed on the input image unconditionally for the photographing conditions other than the image size, the process of step S2930 may be omitted after the process of step S2920, and the process may proceed to step S2940.
　ステップＳ２９２０において、実施例８と同様に、撮影条件取得部２８０２が入力画像の撮影条件群を取得したら、処理はステップＳ２９３０に移行する。ステップＳ２９３０では、処理可否判定部２８０３が、取得された撮影条件群を用いて、画像セグメンテーションエンジンによって入力画像を対処可能であるか否かを判定する。具体的には、処理可否判定部２８０３は、入力画像の撮影条件について、画像セグメンテーションエンジンが対処可能な、撮影部位、撮影方式、及び撮影画角であるか否かを判定する。処理可否判定部２８０３は、実施例８と異なり、画像サイズは判定しない。 In step S2920, as in the eighth embodiment, when the shooting condition obtaining unit 2802 obtains the shooting condition group of the input image, the process proceeds to step S2930. In step S2930, the processing possibility determination unit 2803 determines whether the input image can be handled by the image segmentation engine using the acquired shooting condition group. More specifically, the processing availability determination unit 2803 determines whether or not the imaging condition of the input image is an imaging region, an imaging method, and an imaging angle of view that can be handled by the image segmentation engine. Unlike the eighth embodiment, the processing availability determination unit 2803 does not determine the image size.
　処理可否判定部２８０３が、撮影部位、撮影方式、及び撮影画角について判定し、入力画像が対処可能と判定された場合には、処理はステップＳ２９４０に移行する。一方、処理可否判定部２８０３が、これら撮影条件に基づいて、画像セグメンテーションエンジンが入力画像を対処不可能であると判定した場合には、処理はステップＳ２９７０に移行する。なお、画像処理装置２８００の設定や実装形態によっては、撮影部位、撮影方式、及び撮影画角のうちの一部に基づいて入力画像が処理不可能であると判定されたとしても、ステップＳ２９４０における画像セグメンテーション処理が実施されてもよい。 (4) The processing availability determination unit 2803 determines the imaging region, the imaging method, and the imaging angle of view, and if it is determined that the input image can be handled, the process proceeds to step S2940. On the other hand, when the processability determination unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970. Note that, depending on the setting and the implementation form of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on a part of the imaging part, the imaging method, and the imaging angle of view, it is determined in step S2940 that the input image cannot be processed. An image segmentation process may be performed.
　処理がステップＳ２９４０に移行すると、図３７に示される本実施例に係る画像セグメンテーション処理が開始される。本実施例に係る画像セグメンテーション処理では、まず、ステップＳ３７１０において、セグメンテーション処理部２８０４が、入力画像を教師データについて設定された一定の解像度となるように拡大又は縮小し、変形画像を生成する。 When the process proceeds to step S2940, the image segmentation process according to the embodiment shown in FIG. 37 is started. In the image segmentation processing according to the present embodiment, first, in step S3710, the segmentation processing unit 2804 enlarges or reduces the input image to have a fixed resolution set for the teacher data, and generates a deformed image.
　次に、ステップＳ３７２０において、セグメンテーション処理部２８０４は、生成した変形画像について、教師データについて設定された画像サイズとなるように、パディングを行ってパディング画像を生成する。この際、セグメンテーション処理部２８０４は、パディングを行う領域について、効果的に画像セグメンテーション処理できるように学習済モデルの特性に合わせて、一定の画素値で埋めたり、近傍画素値で埋めたり、ミラーパディングしたりする。 Next, in step S3720, the segmentation processing unit 2804 generates a padding image by performing padding on the generated deformed image so as to have the image size set for the teacher data. At this time, the segmentation processing unit 2804 fills the area to be padded with a fixed pixel value, a neighboring pixel value, or a mirror padding in accordance with the characteristics of the learned model so that the image segmentation processing can be performed effectively. Or
　ステップＳ３７３０では、セグメンテーション処理部２８０４がパディング画像を画像セグメンテーションエンジンに入力し画像セグメンテーション処理された第一の領域ラベル画像を取得する。 In step S3730, the segmentation processing unit 2804 inputs the padding image to the image segmentation engine, and obtains a first region label image that has been subjected to image segmentation processing.
　次に、ステップＳ３７４０において、セグメンテーション処理部２８０４は、第一の領域ラベル画像について、ステップＳ３７２０でパディングを行った領域分だけトリミングを行い、第二の領域ラベル画像を生成する。 Next, in step S3740, the segmentation processing unit 2804 performs trimming on the first area label image by the area padded in step S3720 to generate a second area label image.
　その後、ステップＳ３７５０において、セグメンテーション処理部２８０４は、第二の領域ラベル画像を入力画像の画像サイズに縮小又は拡大し、最終的な領域ラベル画像を生成する。セグメンテーション処理部２８０４がステップＳ３７５０において最終的な領域ラベル画像を生成したら、本実施例に係る画像セグメンテーション処理は終了し、処理はステップＳ２９５０に移行する。ステップＳ２９５０以降の処理は、実施例８のステップＳ２９５０以降の処理と同様であるため説明を省略する。 Then, in step S3750, the segmentation processing unit 2804 reduces or enlarges the second region label image to the image size of the input image, and generates a final region label image. When the segmentation processing unit 2804 generates the final area label image in step S3750, the image segmentation processing according to the present embodiment ends, and the processing shifts to step S2950. The processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, and a description thereof will not be repeated.
　上記のように、本実施例によるセグメンテーション処理部２８０４は、入力画像の解像度が所定の解像度となるように、入力画像の画像サイズを調整する。また、セグメンテーション処理部２８０４は、画像サイズが調整された入力画像について、画像セグメンテーションエンジンによって対処可能な画像サイズとなるように、パディングを行った画像を生成し、画像セグメンテーションエンジンに入力する。その後、セグメンテーション処理部２８０４は、画像セグメンテーションエンジンからの出力画像について、パディングを行った領域分だけトリミングを行う。そして、セグメンテーション処理部２８０４は、トリミングが行われた画像の画像サイズを、入力画像の元の画像サイズに調整することで領域ラベル画像を生成する。 As described above, the segmentation processing unit 2804 according to the present embodiment adjusts the image size of the input image so that the resolution of the input image becomes a predetermined resolution. In addition, the segmentation processing unit 2804 generates a padded image of the input image whose image size has been adjusted so that the image size can be handled by the image segmentation engine, and inputs the image to the image segmentation engine. Thereafter, the segmentation processing unit 2804 trims the output image from the image segmentation engine by the padded area. Then, the segmentation processing unit 2804 generates an area label image by adjusting the image size of the trimmed image to the original image size of the input image.
　これにより、本実施例のセグメンテーション処理部２８０４は、実施例８では対処できなかった画像サイズの入力画像であっても、画像セグメンテーションエンジンによって画像セグメンテーション処理して領域ラベル画像を生成することができる。また、解像度を基準とした教師データで学習した画像セグメンテーションエンジンを用いることで、同一画像サイズの画像を処理する実施例１０に係る画像セグメンテーションエンジンより効率よく入力画像を画像セグメンテーション処理できる場合がある。 Accordingly, the segmentation processing unit 2804 of this embodiment can generate an area label image by performing an image segmentation process using an image segmentation engine even for an input image having an image size that cannot be dealt with in the eighth embodiment. In addition, by using an image segmentation engine that has been trained with teacher data based on the resolution, the input image may be more efficiently subjected to image segmentation processing than the image segmentation engine according to the tenth embodiment that processes images of the same image size.
（実施例１３）
　次に、図２８、図２９、及び図３８乃至図４０を参照して、実施例１３に係る画像処理装置について説明する。本実施例では、セグメンテーション処理部が、入力画像を一定の画像サイズの領域毎に画像セグメンテーション処理することにより領域ラベル画像を生成する。 (Example 13)
Next, an image processing apparatus according to the thirteenth embodiment will be described with reference to FIGS. 28, 29, and 38 to 40. In the present embodiment, the segmentation processing unit generates an area label image by performing image segmentation processing on an input image for each area of a fixed image size.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例８に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例８に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は、実施例８に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
　本実施例に係るセグメンテーション処理部２８０４には、実施例８と同様の、画像セグメンテーションエンジンが備えられている。ただし、本実施例では、画像セグメンテーションエンジンが含む機械学習モデルの学習に用いる教師データが実施例８における教師データと異なる。具体的には、教師データを構成する、入力画像である入力データと、入力画像に対応する領域ラベル画像である出力データとのペア群を、入力画像と領域ラベル画像における、位置関係が対応する一定の画像サイズの矩形領域画像によって構成している。 セ The segmentation processing unit 2804 according to the present embodiment includes an image segmentation engine similar to that of the eighth embodiment. However, in the present embodiment, the teacher data used for learning the machine learning model included in the image segmentation engine is different from the teacher data in the eighth embodiment. Specifically, a pair group of input data, which is an input image, and output data, which is an area label image corresponding to the input image, which constitutes teacher data, has a positional relationship between the input image and the area label image. It consists of a rectangular area image of a fixed image size.
　ここで、図３８を参照して、本実施例に係る画像セグメンテーションエンジンの教師データについて説明する。図３８に示すように、教師データを構成するペア群の一つに関して、例えば、入力画像である画像Ｉｍ３８１０と、対応する領域ラベル画像である領域ラベル画像Ｉｍ３８２０があるとした場合を考える。この場合、実施例８においては、教師データの入力データを画像Ｉｍ３８１０、出力データを領域ラベル画像Ｉｍ３８２０とした。 Here, the teacher data of the image segmentation engine according to the present embodiment will be described with reference to FIG. As shown in FIG. 38, regarding one of the pairs forming the teacher data, it is assumed that, for example, there is an image Im3810 as an input image and an area label image Im3820 as a corresponding area label image. In this case, in the eighth embodiment, the input data of the teacher data is the image Im3810, and the output data is the area label image Im3820.
　これに対し、本実施例においては、画像Ｉｍ３８１０のうちの矩形領域画像Ｒ３８１１を入力データとし、領域ラベル画像Ｉｍ３８２０において矩形領域画像Ｒ３８１１と同じ（対応する）撮影領域である矩形領域画像Ｒ３８２１を出力データとする。そして、入力データである矩形領域画像Ｒ３８１１と出力データである矩形領域画像Ｒ３８２１によって教師データのペア（以下、第一の矩形領域画像ペア）を構成する。 In contrast, in the present embodiment, a rectangular area image R3811 of the image Im3810 is used as input data, and a rectangular area image R3821, which is the same (corresponding) shooting area as the rectangular area image R3811 in the area label image Im3820, is output data. And Then, a pair of teacher data (hereinafter, a first rectangular region image pair) is configured by the rectangular region image R3811 as input data and the rectangular region image R3821 as output data.
　ここで、矩形領域画像Ｒ３８１１と矩形領域画像Ｒ３８２１は、一定の画像サイズの画像とされる。なお、画像Ｉｍ３８１０と領域ラベル画像Ｉｍ３８２０は任意方法により位置合わせされてよい。また、矩形領域画像Ｒ３８１１と矩形領域画像Ｒ３８２１の対応する位置関係はテンプレートマッチングなどの任意方法によって特定されてよい。なお、画像セグメンテーションエンジンが含む機械学習モデルの設計によっては、入力データと出力データの、それぞれの画像サイズや次元数は異なっていてもよい。例えば、入力データがＢスキャン画像（二次元画像）の一部であり、出力データがＡスキャン画像（一次元）の一部とする場合である。 Here, the rectangular region image R3811 and the rectangular region image R3821 are images having a fixed image size. Note that the image Im3810 and the region label image Im3820 may be aligned by an arbitrary method. The positional relationship between the rectangular area image R3811 and the rectangular area image R3821 may be specified by an arbitrary method such as template matching. Note that, depending on the design of the machine learning model included in the image segmentation engine, the image size and the number of dimensions of the input data and the output data may be different. For example, there is a case where the input data is a part of the B-scan image (two-dimensional image) and the output data is a part of the A-scan image (one-dimensional).
　前述の一定の画像サイズは、例えば、処理対象（入力画像）として想定される画像の画像サイズ群について、対応する各次元の画素数群の公約数から決定することができる。この場合、画像セグメンテーションエンジンが出力する矩形領域画像群の位置関係が重なることを防いでもよい。 The above-mentioned fixed image size can be determined, for example, from a common divisor of a corresponding pixel number group of each dimension, for an image size group of an image assumed as a processing target (input image). In this case, it is possible to prevent the positional relationships of the rectangular area image groups output by the image segmentation engine from overlapping.
　具体的には、例えば、処理対象として想定される画像が二次元画像であり、画像サイズ群のうちの第一の画像サイズが幅５００画素、高さ５００画素であり、第二の画像サイズが幅１００画素、高さ１００画素である場合を考える。ここで、各辺の公約数から、矩形領域画像Ｒ３８１１，Ｒ３８２１に関する一定の画像サイズを選択する。この場合には、例えば、一定の画像サイズを、幅１００画素、高さ１００画素や、幅５０画素、高さ５０画素や、幅２５画素、高さ２５画素等から選択する。処理対象として想定される画像が三次元である場合には、幅、高さ、奥行きに関して画素数を決定する。 Specifically, for example, the image assumed as the processing target is a two-dimensional image, the first image size in the image size group is 500 pixels in width and 500 pixels in height, and the second image size is Consider a case where the width is 100 pixels and the height is 100 pixels. Here, a fixed image size for the rectangular area images R3811, R3821 is selected from the common divisor of each side. In this case, for example, a fixed image size is selected from a width of 100 pixels, a height of 100 pixels, a width of 50 pixels, a height of 50 pixels, a width of 25 pixels, a height of 25 pixels, and the like. If the image to be processed is three-dimensional, the number of pixels is determined for the width, height, and depth.
　なお、矩形領域は、入力データに対応する画像と出力データに対応する領域ラベル画像のペアの一つに対して、複数設定可能である。このため、例えば、画像Ｉｍ３８１０のうちの矩形領域画像Ｒ３８１２を入力データ、領域ラベル画像Ｉｍ３８２０において矩形領域画像Ｒ３８１２と同じ撮影領域である矩形領域画像Ｒ３８２２を出力データとする。そして、入力データである矩形領域画像Ｒ３８１２と出力データである矩形領域画像Ｒ３８２２によって教師データのペアを構成する。これにより、第一の矩形領域画像ペアとは別の矩形領域画像ペアを作成できる。 Note that a plurality of rectangular areas can be set for one of a pair of an area label image corresponding to input data and an area label image corresponding to output data. Therefore, for example, the rectangular area image R3812 of the image Im3810 is set as input data, and the rectangular area image R3822 which is the same shooting area as the rectangular area image R3812 in the area label image Im3820 is set as output data. Then, a pair of teacher data is formed by the rectangular area image R3812 as input data and the rectangular area image R3822 as output data. Thus, a rectangular area image pair different from the first rectangular area image pair can be created.
　なお、矩形領域の画像を異なる座標の画像に変えながら多数の矩形領域画像のペアを作成することで教師データを構成するペア群を充実させることができる。そして、当該教師データを構成するペア群を用いてトレーニングを行った画像セグメンテーションエンジンによって効率的な画像セグメンテーション処理が期待できる。ただし、学習済モデルの画像セグメンテーション処理に寄与しないペアは教師データに加えないようにすることができる。 By creating a large number of rectangular area image pairs while changing the rectangular area image into an image with different coordinates, the group of pairs forming the teacher data can be enhanced. Then, an efficient image segmentation process can be expected by the image segmentation engine that has been trained using the pair group forming the teacher data. However, pairs that do not contribute to the image segmentation processing of the trained model can be prevented from being added to the teacher data.
　なお、教師データの入力データ及び出力データとしては、一つの層や一つのラベルが付される領域を描画する画像を教師データとして用いることができる。また、教師データの入力データ及び出力データとして、複数の層、例えば２つの層、より好ましくは３つ以上の層を描画する領域の画像を用いることもできる。同様に、領域ラベル画像においてラベルが分けられる領域を複数描画する領域の画像を用いることもできる。これらの場合には、一つの層や一つのラベルが付される領域を描画する画像を教師データとして用いる場合と比べて、学習した層や領域についての位置関係から、学習済モデルを用いてより適切に画像セグメンテーション処理を行えるようになることが期待できる。 Note that, as input data and output data of the teacher data, an image that draws one layer or an area to which one label is attached can be used as the teacher data. Further, as input data and output data of the teacher data, an image of an area where a plurality of layers, for example, two layers, and more preferably three or more layers are drawn, can be used. Similarly, an image of an area in which a plurality of areas into which labels are divided in the area label image is drawn can be used. In these cases, compared to the case of using an image that draws one layer or an area to which one label is attached as teacher data, the position of the learned layer or area is more easily used by using the trained model. It can be expected that image segmentation processing can be performed appropriately.
　さらに、例えば、ペアである、入力画像から作成した矩形領域画像と領域ラベル画像から作成した矩形領域画像とに描画される撮影対象の構造や位置が大きく異なる場合を考える。この場合には、そのような教師データを用いて学習を行った画像セグメンテーションエンジンが精度の低い領域ラベル画像を出力してしまう可能性がある。そのため、このようなペアを教師データから取り除くこともできる。 {Circle around (2)} Further, for example, consider a case where the structures and positions of the imaging targets drawn in the rectangular area image created from the input image and the rectangular area image created from the area label image, which are a pair, are significantly different. In this case, the image segmentation engine that has learned using such teacher data may output a region label image with low accuracy. Therefore, such a pair can be removed from the teacher data.
　なお、実施例８と同様に、教師データの入力データには、処理対象として想定される特定の撮影条件を持つ画像を用いるが、当該特定の撮影条件は、予め決定された撮影部位、撮影方式、及び撮影画角である。つまり、本実施例に係る当該特定の撮影条件には、実施例８と異なり、画像サイズは含まれない。 As in the eighth embodiment, the input data of the teacher data uses an image having a specific imaging condition assumed as a processing target, and the specific imaging condition is determined by a predetermined imaging part and imaging method. , And the shooting angle of view. That is, unlike the eighth embodiment, the specific photographing condition according to the present embodiment does not include the image size.
　本実施例に係るセグメンテーション処理部２８０４は、このような教師データで学習が行われた画像セグメンテーションエンジンを用いて、入力画像を画像セグメンテーション処理して領域ラベル画像を生成する。この際、セグメンテーション処理部２８０４は、入力された画像を、隙間なく連続する、教師データについて設定された一定の画像サイズの矩形領域画像群に分割する。セグメンテーション処理部２８０４は、画像セグメンテーションエンジンを用いて、分割した矩形領域画像群のそれぞれを画像セグメンテーション処理し、分割された領域ラベル画像群を生成する。その後、セグメンテーション処理部２８０４は、生成した分割された領域ラベル画像群を、入力画像の位置関係に応じて配置して結合し、最終的な領域ラベル画像を生成する。 セ The segmentation processing unit 2804 according to the present embodiment generates an area label image by performing image segmentation processing on an input image using an image segmentation engine that has been trained with such teacher data. At this time, the segmentation processing unit 2804 divides the input image into a continuous rectangular area image group having a fixed image size set for the teacher data without any gap. The segmentation processing unit 2804 performs an image segmentation process on each of the divided rectangular area image groups using an image segmentation engine, and generates a divided area label image group. Thereafter, the segmentation processing unit 2804 arranges the generated divided area label images according to the positional relationship of the input images and combines them to generate a final area label image.
　このように、本実施例のセグメンテーション処理部２８０４は、入力された画像を矩形領域単位で画像セグメンテーション処理し、画像セグメンテーション処理した画像を結合する。これにより、実施例８では対応しなかった画像サイズの画像をも画像セグメンテーション処理して領域ラベル画像を生成することができる。 As described above, the segmentation processing unit 2804 of the present embodiment performs an image segmentation process on an input image in units of a rectangular area, and combines the images that have been subjected to the image segmentation process. As a result, it is possible to generate an area label image by performing image segmentation processing on an image having an image size not supported in the eighth embodiment.
　次に、図２９、図３９及び図４０を参照して、本実施例に係る一連の画像処理について説明する。図３９は、本実施例に係る画像セグメンテーション処理のフローチャートである。なお、本実施例に係るステップＳ２９１０、ステップＳ２９２０、及びステップＳ２９５０～ステップＳ２９７０の処理は、実施例８におけるこれらの処理と同様であるため、説明を省略する。なお、入力画像に対して、画像サイズ以外の撮影条件について無条件で画像セグメンテーション処理する場合には、ステップＳ２９２０の処理の後に、ステップＳ２９３０の処理を省き、処理をステップＳ２９４０に移行してよい。 Next, a series of image processing according to the present embodiment will be described with reference to FIGS. 29, 39, and 40. FIG. 39 is a flowchart of the image segmentation process according to the embodiment. Note that the processes in steps S2910, S2920, and steps S2950 to S2970 according to the present embodiment are the same as those in the eighth embodiment, and thus description thereof will be omitted. If the input image is subjected to the image segmentation processing unconditionally for the photographing conditions other than the image size, the processing in step S2930 may be omitted after the processing in step S2920, and the processing may proceed to step S2940.
　ステップＳ２９２０において、実施例８と同様に、撮影条件取得部２８０２が入力画像の撮影条件群を取得したら、処理はステップＳ２９３０に移行する。ステップＳ２９３０では、処理可否判定部２８０３が、取得された撮影条件群を用いて、画像セグメンテーションエンジンにより入力画像を対処可能であるか否かを判定する。具体的には、処理可否判定部２８０３は、入力画像の撮影条件について、画像セグメンテーションエンジンが対処可能な、撮影部位、撮影方式、及び撮影画角であるか否かを判定する。処理可否判定部２８０３は、実施例８と異なり、画像サイズは判定しない。 In step S2920, as in the eighth embodiment, when the shooting condition obtaining unit 2802 obtains the shooting condition group of the input image, the process proceeds to step S2930. In step S2930, the processing availability determination unit 2803 determines whether the input image can be handled by the image segmentation engine using the acquired shooting condition group. More specifically, the processing availability determination unit 2803 determines whether or not the imaging condition of the input image is an imaging region, an imaging method, and an imaging angle of view that can be handled by the image segmentation engine. Unlike the eighth embodiment, the processing availability determination unit 2803 does not determine the image size.
　処理可否判定部２８０３が、撮影部位、撮影方式、及び撮影画角について判定し、入力画像が対処可能と判定された場合には、処理はステップＳ２９４０に移行する。一方、処理可否判定部２８０３が、これら撮影条件に基づいて、画像セグメンテーションエンジンが入力画像を対処不可能であると判定した場合には、処理はステップＳ２９７０に移行する。なお、画像処理装置２８００の設定や実装形態によっては、撮影部位、撮影方式、及び撮影画角のうちの一部に基づいて入力画像が処理不可能であると判定されたとしても、ステップＳ２９４０における画像セグメンテーション処理が実施されてもよい。 (4) The processing availability determination unit 2803 determines the imaging region, the imaging method, and the imaging angle of view, and if it is determined that the input image can be handled, the process proceeds to step S2940. On the other hand, when the processability determination unit 2803 determines that the image segmentation engine cannot handle the input image based on these shooting conditions, the process proceeds to step S2970. Note that, depending on the setting and the implementation form of the image processing apparatus 2800, even if it is determined that the input image cannot be processed based on a part of the imaging part, the imaging method, and the imaging angle of view, it is determined in step S2940 that the input image cannot be processed. An image segmentation process may be performed.
　処理がステップＳ２９４０に移行すると、図３９に示される本実施例に係る画像セグメンテーション処理が開始される。本実施例に係る画像セグメンテーション処理では、まず、ステップＳ３９１０において、図４０に示すように、入力画像を隙間なく連続する、教師データについて設定された一定の画像サイズの矩形領域画像群に分割する。ここで、図４０は、入力画像Ｉｍ４０１０を一定の画像サイズの矩形領域画像Ｒ４０１１～Ｒ４０２６群に分割した一例を示す。なお、画像セグメンテーションエンジンが含む機械学習モデルの設計によっては、入力画像と出力画像の、それぞれの画像サイズや次元数が異なってもよい。その場合には、ステップＳ３９２０において生成される結合された領域ラベル画像に欠損が無いよう、前述の入力画像の分割位置を重複させたり、分離させたりして、調整する。 When the process proceeds to step S2940, the image segmentation process according to the embodiment shown in FIG. 39 is started. In the image segmentation processing according to the present embodiment, first, in step S3910, as shown in FIG. 40, the input image is divided into a group of rectangular area images that are continuous without gaps and have a fixed image size set for teacher data. Here, FIG. 40 shows an example in which the input image Im4010 is divided into groups of rectangular area images R4011 to R4026 having a fixed image size. Note that, depending on the design of the machine learning model included in the image segmentation engine, the image size and the number of dimensions of the input image and the output image may be different. In that case, the above-described division positions of the input image are adjusted by overlapping or separating so that the combined area label image generated in step S3920 has no loss.
　次に、ステップＳ３９２０において、セグメンテーション処理部２８０４は、矩形領域画像Ｒ４０１１～Ｒ４０２６群のそれぞれを画像セグメンテーションエンジンにより画像セグメンテーション処理し、分割された領域ラベル画像群を生成する。 Next, in step S3920, the segmentation processing unit 2804 performs image segmentation processing on each of the rectangular area images R4011 to R4026 by the image segmentation engine to generate a divided area label image group.
　そして、ステップＳ３９３０において、セグメンテーション処理部２８０４は、生成した分割された領域ラベル画像群のそれぞれを、入力画像について分割した矩形領域画像Ｒ４０１１～Ｒ４０２６群のそれぞれと同じ位置関係に配置して結合する。これにより、セグメンテーション処理部２８０４は領域ラベル画像を生成することができる。 Then, in step S3930, the segmentation processing unit 2804 arranges and combines the generated divided region label image groups in the same positional relationship as the divided rectangular region images R4011 to R4026 group of the input image. Accordingly, the segmentation processing unit 2804 can generate an area label image.
　セグメンテーション処理部２８０４がステップＳ３９３０において領域ラベル画像を生成したら、本実施例に係る画像セグメンテーション処理は終了し、処理はステップＳ２９５０に移行する。ステップＳ２９５０以降の処理は、実施例８のステップＳ２９５０以降の処理と同様であるため説明を省略する。 When the segmentation processing unit 2804 generates the area label image in step S3930, the image segmentation processing according to the present embodiment ends, and the processing shifts to step S2950. The processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, and a description thereof will not be repeated.
　上記のように、本実施例に係るセグメンテーション処理部２８０４は、入力画像を所定の画像サイズの複数の矩形領域画像Ｒ４０１１～Ｒ４０２６に分割する。その後、セグメンテーション処理部２８０４は、分割した複数の矩形領域画像Ｒ４０１１～Ｒ４０２６を画像セグメンテーションエンジンに入力して複数の分割領域ラベル画像を生成し、複数の分割領域ラベル画像を統合することで、領域ラベル画像を生成する。なお、統合時に矩形領域群間で位置関係が重なる場合には、該矩形領域群の画素値群を統合したり、上書きしたりする。 As described above, the segmentation processing unit 2804 according to the present embodiment divides an input image into a plurality of rectangular area images R4011 to R4026 having a predetermined image size. After that, the segmentation processing unit 2804 inputs the plurality of divided rectangular area images R4011 to R4026 to the image segmentation engine to generate a plurality of divided area label images, and integrates the plurality of divided area label images to obtain the area label. Generate an image. If the positional relationship between the rectangular area groups overlaps during integration, the pixel value groups of the rectangular area groups are integrated or overwritten.
　これにより、本実施例のセグメンテーション処理部２８０４は、実施例８では対処できなかった画像サイズの入力画像であっても、画像セグメンテーションエンジンを用いて画像セグメンテーション処理して領域ラベル画像を生成することができる。また、教師データを、所定の画像サイズに分割した複数の画像から作成すると、少ない画像から多くの教師データを作成することができる。そのため、この場合には、教師データを作成するための入力画像と領域ラベル画像の数を少なくすることができる。 Accordingly, the segmentation processing unit 2804 according to the present embodiment can generate an area label image by performing image segmentation processing using an image segmentation engine even for an input image having an image size that cannot be dealt with in the eighth embodiment. it can. In addition, when the teacher data is created from a plurality of images divided into a predetermined image size, a large number of teacher data can be created from a small number of images. Therefore, in this case, the number of input images and area label images for creating teacher data can be reduced.
　また、本実施例に係る画像セグメンテーションエンジンが含む学習済モデルは、２つ以上の層を含む断層画像を入力データとし、該断層画像に対応する領域ラベル画像を出力データとして学習を行ったモデルである。このため、一つの層や一つのラベルが付される領域を描画する画像を教師データとして用いる場合と比べて、学習した層や領域についての位置関係から、学習済モデルを用いてより適切に画像セグメンテーション処理を行えるようになることが期待できる。 The trained model included in the image segmentation engine according to the present embodiment is a model in which a tomographic image including two or more layers is used as input data, and a region label image corresponding to the tomographic image is used as output data to perform learning. is there. For this reason, compared to the case where an image that draws one layer or one label-attached area is used as teacher data, the image is more appropriately used using the trained model based on the positional relationship of the learned layers or areas. It can be expected that segmentation processing will be performed.
（実施例１４）
　次に、図２８、図４１及び図４２を参照して、実施例１４に係る画像処理装置について説明する。本実施例では、評価部が、検者の指示に応じて、複数の画像セグメンテーションエンジンから出力された複数の領域ラベル画像のうち最も精度の高い領域ラベル画像を選択する。 (Example 14)
Next, an image processing apparatus according to embodiment 14 will be described with reference to FIGS. In the present embodiment, the evaluator selects the most accurate area label image from among the plurality of area label images output from the plurality of image segmentation engines in accordance with the instruction of the examiner.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例８に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例８に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は、実施例８に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
　本実施例に係るセグメンテーション処理部２８０４は、異なる教師データを用いて機械学習が行われたそれぞれの学習済モデルを含む二つ以上の画像セグメンテーションエンジンを用いて、入力画像について画像セグメンテーション処理を行う。 セ The segmentation processing unit 2804 according to the present embodiment performs image segmentation processing on an input image using two or more image segmentation engines including respective learned models that have been machine-learned using different teacher data.
　ここで、本実施例に係る教師データ群の作成方法について説明する。具体的には、まず、様々な撮影条件によって撮影された画像である入力データと領域ラベル画像である出力データのペア群を用意する。次に、任意の撮影条件の組み合わせによってペア群をグルーピングすることで、教師データ群を作成する。例えば、第一の撮影条件の組み合わせによって取得されたペア群で構成される第一の教師データ、第二の撮影条件の組み合わせによって取得されたペア群で構成される第二の教師データというように、教師データ群として作成する。 Here, a method of creating the teacher data group according to the present embodiment will be described. Specifically, first, a pair group of input data, which is an image photographed under various photographing conditions, and output data, which is an area label image, is prepared. Next, a teacher data group is created by grouping the pair groups according to an arbitrary combination of imaging conditions. For example, first teacher data composed of a pair group acquired by a combination of first imaging conditions, second teacher data composed of a pair group acquired by a combination of second imaging conditions, and so on. Is created as a teacher data group.
　その後、各教師データを用いて、別々の画像セグメンテーションエンジンに含まれる機械学習モデルに機械学習を行わせる。例えば、第一の教師データでトレーニングされた学習済モデルを含む第一の画像セグメンテーションエンジンを用意する。加えて、第二の教師データでトレーニングされた学習済モデルに対応する第二の画像セグメンテーションエンジンを用意する、というように画像セグメンテーションエンジン群を用意する。 Then, using each of the teacher data, the machine learning model included in the separate image segmentation engine performs machine learning. For example, a first image segmentation engine including a trained model trained with first teacher data is provided. In addition, a group of image segmentation engines is prepared, such as preparing a second image segmentation engine corresponding to the trained model trained with the second teacher data.
　このような画像セグメンテーションエンジンは、それぞれが含む学習済モデルのトレーニングに用いた教師データが異なる。そのため、このような画像セグメンテーションエンジンは、画像セグメンテーションエンジンに入力される画像の撮影条件によって、入力画像を画像セグメンテーション処理できる精度が異なる。具体的には、第一の画像セグメンテーションエンジンは、第一の撮影条件の組み合わせで撮影して取得された入力画像に対しては画像セグメンテーション処理の精度が高い。一方で、第一の画像セグメンテーションエンジンは、第二の撮影条件の組み合わせで撮影して取得された画像に対しては画像セグメンテーション処理の精度が低い。同様に、第二の画像セグメンテーションエンジンは、第二の撮影条件の組み合わせで撮影して取得された入力画像に対しては画像セグメンテーション処理の精度が高い。一方で、第二の画像セグメンテーションエンジンは、第一の撮影条件の組み合わせで撮影して取得された画像に対しては画像セグメンテーション処理の精度が低い。 画像 Such image segmentation engines differ in the teacher data used for training the learned models included in each. Therefore, in such an image segmentation engine, the accuracy with which the input image can be subjected to the image segmentation process differs depending on the imaging conditions of the image input to the image segmentation engine. Specifically, the first image segmentation engine has high accuracy of the image segmentation processing on an input image obtained by shooting under a combination of the first shooting conditions. On the other hand, the first image segmentation engine has a low accuracy of the image segmentation process for an image captured and acquired under the combination of the second capturing conditions. Similarly, the second image segmentation engine has high accuracy of the image segmentation process for an input image captured and acquired under the combination of the second capturing conditions. On the other hand, the second image segmentation engine has a low accuracy of the image segmentation processing for an image captured and acquired under the combination of the first capturing conditions.
　教師データのそれぞれが撮影条件の組み合わせによってグルーピングされたペア群で構成されることにより、該ペア群を構成する画像群の画質傾向が似る。このため、画像セグメンテーションエンジンは対応する撮影条件の組み合わせであれば、実施例８に係る画像セグメンテーションエンジンよりも精度良く画像セグメンテーション処理を行うことができる。なお、教師データのペアをグルーピングするための撮影条件の組み合わせは、任意であってよく、例えば、撮影部位、撮影画角、及び画像の解像度のうちの二つ以上の組み合わせであってよい。また、教師データのグルーピングを、実施例９と同様に、一つの撮影条件に基づいて行ってもよい。 (4) Since each of the teacher data is constituted by a pair group grouped by a combination of the photographing conditions, the image quality tendency of the image group constituting the pair group is similar. Therefore, if the image segmentation engine is a combination of the corresponding shooting conditions, the image segmentation processing can be performed more accurately than the image segmentation engine according to the eighth embodiment. The combination of the imaging conditions for grouping the pair of teacher data may be arbitrary, and may be, for example, a combination of two or more of the imaging region, the imaging angle of view, and the image resolution. Further, the grouping of the teacher data may be performed based on one shooting condition, as in the ninth embodiment.
　評価部２８０５は、セグメンテーション処理部２８０４が、複数の画像セグメンテーションエンジンを用いて生成した複数の領域ラベル画像について、実施例８と同様に評価を行う。その後、評価部２８０５は、評価結果が真値である領域ラベル画像が複数ある場合、検者の指示に応じて、当該複数の領域ラベル画像のうち最も精度の高い領域ラベル画像を選択し、出力すべき領域ラベル画像として判断する。なお、評価部２８０５は、実施例８と同様に、学習済モデルを含む領域ラベル画像評価エンジンを用いて評価を行ってもよいし、知識ベースの領域ラベル画像評価エンジンを用いて評価を行ってもよい。 The evaluation unit 2805 evaluates a plurality of region label images generated by the segmentation processing unit 2804 using the plurality of image segmentation engines in the same manner as in the eighth embodiment. After that, when there are a plurality of area label images for which the evaluation result is a true value, the evaluator 2805 selects an area label image with the highest accuracy among the plurality of area label images according to the instruction of the examiner, and outputs the selected area label image. It is determined as an area label image to be performed. The evaluation unit 2805 may perform evaluation using an area label image evaluation engine including a learned model, or may perform evaluation using a knowledge-based area label image evaluation engine, as in the eighth embodiment. Is also good.
　解析部２８０６は、評価部２８０５によって出力すべき領域ラベル画像として判断された領域ラベル画像と入力画像を用いて、入力画像について実施例８と同様に画像解析処理を行う。出力部２８０７は、出力すべき領域ラベル画像として判断された領域ラベル画像や解析結果を、実施例８と同様に、表示部２８２０に表示させたり、他の装置に出力したりすることができる。なお、出力部２８０７は、評価結果が真値である複数の領域ラベル画像を表示部２８２０に表示させることができ、評価部２８０５は、表示部２８２０を確認した検者からの指示に応じて最も精度の高い領域ラベル画像を選択することができる。 The analysis unit 2806 performs an image analysis process on the input image in the same manner as in the eighth embodiment using the input image and the area label image determined as the area label image to be output by the evaluation unit 2805. The output unit 2807 can display the region label image determined as the region label image to be output and the analysis result on the display unit 2820 or output to another device, as in the eighth embodiment. Note that the output unit 2807 can display a plurality of area label images whose evaluation results are true values on the display unit 2820, and the evaluation unit 2805 responds to the instruction from the examiner who has checked the display unit 2820 most frequently. A highly accurate area label image can be selected.
　これにより、画像処理装置２８００は、複数の画像セグメンテーションエンジンを用いて生成された複数の領域ラベル画像のうち、検者の指示に応じた最も精度の高い領域ラベル画像を出力することができる。 Accordingly, the image processing apparatus 2800 can output the most accurate area label image corresponding to the instruction of the examiner among the plurality of area label images generated using the plurality of image segmentation engines.
　以下、図４１及び図４２を参照して、本実施例に係る一連の画像処理について説明する。図４１は、本実施例に係る一連の画像処理のフローチャートである。なお、本実施例に係るステップＳ４１１０及びステップＳ４１２０の処理は、実施例８におけるステップＳ２９１０及びステップＳ２９２０での処理と同様であるため、説明を省略する。なお、入力画像に対して、撮影条件について無条件で画像セグメンテーション処理する場合には、ステップＳ４１２０の処理の後に、ステップＳ４１３０の処理を省き、処理をステップＳ４１４０に移行してよい。 Hereinafter, a series of image processing according to the present embodiment will be described with reference to FIGS. FIG. 41 is a flowchart of a series of image processing according to the present embodiment. Note that the processing in steps S4110 and S4120 according to the present embodiment is the same as the processing in steps S2910 and S2920 in the eighth embodiment, and a description thereof will not be repeated. When the image segmentation process is performed on the input image unconditionally with respect to the shooting conditions, the process of step S4130 may be omitted after the process of step S4120, and the process may proceed to step S4140.
　ステップＳ４１２０において、実施例８と同様に、撮影条件取得部２８０２が入力画像の撮影条件群を取得したら、処理はステップＳ４１３０に移行する。ステップＳ４１３０では、処理可否判定部２８０３が、取得された撮影条件群を用いて、セグメンテーション処理部２８０４が用いる画像セグメンテーションエンジン群のいずれかが入力画像を対処可能であるか否かを判定する。 In step S4120, similarly to the eighth embodiment, when the shooting condition obtaining unit 2802 obtains the shooting condition group of the input image, the process proceeds to step S4130. In step S4130, the processing possibility determination unit 2803 determines whether any of the image segmentation engine groups used by the segmentation processing unit 2804 can handle the input image, using the acquired imaging condition group.
　処理可否判定部２８０３が、画像セグメンテーションエンジン群のいずれも入力画像を対処不可能であると判定した場合には、処理はステップＳ４１８０に移行する。一方で、処理可否判定部２８０３が、画像セグメンテーションエンジン群のいずれかが入力画像を対処可能であると判定した場合には、処理はステップＳ４１４０に移行する。なお、画像処理装置２８００の設定や実装形態によっては、実施例８と同様に、画像セグメンテーションエンジンによって一部の撮影条件が対処不可能であると判定されたとしても、ステップＳ４１４０を実施してもよい。 If the processing availability determination unit 2803 determines that none of the image segmentation engine groups can deal with the input image, the process moves to step S4180. On the other hand, if the process determination unit 2803 determines that any of the image segmentation engine groups can handle the input image, the process proceeds to step S4140. Note that, depending on the settings and the implementation form of the image processing apparatus 2800, as in the eighth embodiment, even if the image segmentation engine determines that some shooting conditions cannot be dealt with, even if step S4140 is executed. Good.
　ステップＳ４１４０においては、セグメンテーション処理部２８０４が、画像セグメンテーションエンジン群のそれぞれにステップＳ４１１０において取得した入力画像を入力し、領域ラベル画像群を生成する。なお、セグメンテーション処理部２８０４は、処理可否判定部２８０３によって入力画像について対処可能であると判定された画像セグメンテーションエンジンにのみ入力画像を入力してもよい。 In step S4140, the segmentation processing unit 2804 inputs the input image obtained in step S4110 to each of the image segmentation engine groups, and generates an area label image group. Note that the segmentation processing unit 2804 may input the input image only to the image segmentation engine that has been determined by the processability determination unit 2803 to be able to handle the input image.
　ステップＳ４１５０では、評価部２８０５が、実施例８と同様に、領域ラベル画像評価エンジンを用いて、ステップＳ４１４０において生成された領域ラベル画像群を評価する。ステップＳ４１６０では、評価結果（画像評価指数）が真値である領域ラベル画像が複数ある場合には、評価部２８０５は検者の指示に応じて、出力すべき領域ラベル画像を選択／判断する。 In step S4150, the evaluation unit 2805 evaluates the area label image group generated in step S4140 using the area label image evaluation engine, as in the eighth embodiment. In step S4160, when there are a plurality of region label images for which the evaluation result (image evaluation index) is a true value, the evaluation unit 2805 selects / determines the region label image to be output according to the instruction of the examiner.
　この場合には、まず、出力部２８０７が、評価結果が真値である領域ラベル画像群を、表示部２８２０のユーザーインターフェースに表示させる。ここで、図４２に当該インターフェースの一例を示す。当該インターフェースには、入力画像ＵＩ４２１０、及び評価結果が真値であった領域ラベル画像ＵＩ４２２０，ＵＩ４２３０，ＵＩ４２４０，ＵＩ４２５０のそれぞれが表示される。検者は不図示の任意の入力装置を操作して、画像群（領域ラベル画像ＵＩ４２２０～ＵＩ４２５０）のうち、最も精度の高い領域ラベル画像を指示する。評価部２８０５は、検者によって指示された領域ラベル画像を出力すべき領域ラベル画像として選択する。 In this case, first, the output unit 2807 causes the user interface of the display unit 2820 to display an area label image group whose evaluation result is a true value. Here, FIG. 42 shows an example of the interface. The interface displays the input image UI4210 and the region label images UI4220, UI4230, UI4240, and UI4250 for which the evaluation result is a true value. The examiner operates an arbitrary input device (not shown) to specify an area label image with the highest accuracy among the image group (area label images UI4220 to UI4250). The evaluation unit 2805 selects the area label image specified by the examiner as the area label image to be output.
　なお、評価結果が真値である領域ラベル画像が１つである場合には、当該領域ラベル画像を出力すべき領域ラベル画像として選択／判断する。また、評価結果が真値である領域ラベル画像がない場合には、評価部２８０５は、セグメンテーション処理部２８０４によって生成された領域ラベル画像を出力しないと判断し、領域ラベル無し画像を生成して出力／選択し、処理をステップＳ４１７０に進める。 If there is only one area label image whose evaluation result is a true value, the area label image is selected / determined as an area label image to be output. If there is no region label image for which the evaluation result is a true value, the evaluation unit 2805 determines that the region label image generated by the segmentation processing unit 2804 is not output, and generates and outputs an image without a region label. / Selection, and the process proceeds to step S4170.
　ステップＳ４１７０では、実施例８と同様に、解析部２８０６が、評価部２８０５によって出力すべき領域ラベル画像と判断された領域ラベル画像と入力画像を用いて、入力画像について画像解析処理を行う。なお、評価部２８０５によって領域ラベル無し画像が出力された場合には、解析部２８０６は画像解析処理を行わずに処理をステップＳ４１８０に進める。 In step S4170, as in the eighth embodiment, the analysis unit 2806 performs an image analysis process on the input image using the area label image determined to be the area label image to be output by the evaluation unit 2805 and the input image. If the evaluation unit 2805 outputs an image without a region label, the analysis unit 2806 advances the process to step S4180 without performing the image analysis process.
　ステップＳ４１８０においては、出力部２８０７が、出力すべき領域ラベル画像として判断された領域ラベル画像や画像解析結果を表示部２８２０に表示させる。なお、出力部２８０７は、表示部２８２０に領域ラベル画像及び画像解析結果を表示させるのに代えて、撮影装置２８１０や他の装置にこれらを表示させたり、記憶させたりしてもよい。また、出力部２８０７は、画像処理装置２８００の設定や実装形態によっては、これらを撮影装置２８１０や他の装置が利用可能なように加工したり、画像管理システム等に送信可能なようにデータ形式を変換したりしてもよい。また、出力部２８０７は、領域ラベル画像及び画像解析結果の両方を出力する構成に限られず、これらのうちのいずれか一方のみを出力してもよい。 In step S4180, the output unit 2807 causes the display unit 2820 to display the region label image determined as the region label image to be output and the image analysis result. Note that the output unit 2807 may display or store these on the imaging device 2810 or another device instead of displaying the region label image and the image analysis result on the display unit 2820. The output unit 2807 may process the image processing device 2800 so that it can be used by the image capturing device 2810 or another device, or may transmit the data to an image management system or the like, depending on the setting or implementation form of the image processing device 2800. May be converted. Also, the output unit 2807 is not limited to a configuration that outputs both the area label image and the image analysis result, and may output only one of them.
　一方、ステップＳ４１３０において画像セグメンテーション処理が不可能であるとされていた場合には、出力部２８０７は、領域ラベル無し画像を出力し、表示部２８２０に表示させる。なお、領域ラベル無し画像を出力する代わりに、撮影装置２８１０に対して、画像セグメンテーション処理が不可能であったことを示す信号を送信してもよい。 On the other hand, if it is determined in step S4130 that the image segmentation process cannot be performed, the output unit 2807 outputs an image without an area label and causes the display unit 2820 to display the image. Note that, instead of outputting the image without the region label, a signal indicating that the image segmentation processing has been impossible may be transmitted to the imaging device 2810.
　また、ステップＳ４１５０において、画像セグメンテーション処理が適切に行えなかった（生成された領域ラベル画像を出力しない）と判断された場合にも、出力部２８０７は、領域ラベル無し画像を出力し、表示部２８２０に表示させる。この場合にも、出力部２８０７は、領域ラベル無し画像を出力する代わりに、撮影装置２８１０に対して、画像セグメンテーション処理が適切に行えなかったことを示す信号を送信してもよい。ステップＳ４１７０における出力処理が終了すると、一連の画像処理が終了する。 Also, in step S4150, when it is determined that the image segmentation process has not been properly performed (the generated region label image is not output), the output unit 2807 outputs an image without a region label, and the display unit 2820 To be displayed. Also in this case, the output unit 2807 may transmit a signal indicating that the image segmentation processing has not been properly performed to the imaging device 2810 instead of outputting an image without an area label. When the output processing in step S4170 ends, a series of image processing ends.
　上記のように、本実施例に係るセグメンテーション処理部２８０４は、それぞれ異なる学習済モデルを含む複数の画像セグメンテーションエンジンを用いて、入力画像から複数の領域ラベル画像を生成する。さらに、評価部２８０５は、検者（ユーザー）の指示に応じて、複数の領域情報を評価して出力すべきと判断した複数の領域情報のうちの少なくとも１つを選択する。より具体的には、評価部２８０５は、画像評価指数が真値である領域ラベル画像が複数ある場合に、検者の指示に応じて、最も精度の高い領域ラベル画像を出力すべき領域ラベル画像として選択／判断する。これにより、画像処理装置２８００は、複数の画像セグメンテーションエンジンを用いて生成された複数の領域ラベル画像のうち、検者の指示に応じた精度の高い領域ラベル画像を出力することができる。 As described above, the segmentation processing unit 2804 according to the present embodiment generates a plurality of region label images from an input image using a plurality of image segmentation engines each including a different learned model. Further, the evaluator 2805 selects at least one of the plurality of pieces of area information determined to be evaluated and output according to an instruction of the examiner (user). More specifically, when there are a plurality of area label images whose image evaluation indices are true values, the evaluation unit 2805 outputs the area label image with the highest accuracy according to the instruction of the examiner. Is selected / determined. Accordingly, the image processing apparatus 2800 can output a highly accurate area label image corresponding to the instruction of the examiner among the plurality of area label images generated by using the plurality of image segmentation engines.
　本実施例では、評価部２８０５が、検者の指示に応じて、最も精度の高い領域ラベル画像を出力すべき領域ラベル画像として選択／判断した。これに対して、評価部２８０５は、検者の指示に応じて、評価結果が真値である複数の領域ラベル画像を出力すべき領域ラベル画像として選択／判断してもよい。この場合には、解析部２８０６は、出力すべき領域ラベル画像として選択された複数の領域ラベル画像について画像解析処理を行う。また、出力部２８０７は、選択された複数の領域ラベル画像や当該複数の領域ラベル画像の解析結果を出力する。 In the present embodiment, the evaluation unit 2805 selects / determines the most accurate area label image as the area label image to be output in accordance with the instruction of the examiner. On the other hand, the evaluation unit 2805 may select / determine a plurality of area label images whose evaluation results are true values as area label images to be output, in accordance with an instruction of the examiner. In this case, the analysis unit 2806 performs image analysis processing on a plurality of region label images selected as region label images to be output. The output unit 2807 outputs a plurality of selected region label images and analysis results of the plurality of region label images.
　また、本実施例では、評価部２８０５が、検者の指示に応じて、評価結果が真値である複数の領域ラベル画像から出力すべき領域ラベル画像を選択した。これに対して、出力部２８０７が、セグメンテーション処理部２８０４によって生成された全ての領域ラベル画像を表示部２８２０に表示させ、評価部２８０５が、検者からの指示に応じて、当該複数の領域ラベル画像から出力すべき領域ラベル画像を選択してもよい。この場合にも、評価部２８０５は、検者からの指示に応じて、複数の領域ラベル画像を出力すべき領域ラベル画像として選択／判断してもよい。 In the present embodiment, the evaluator 2805 selects an area label image to be output from a plurality of area label images for which the evaluation result is a true value, according to the instruction of the examiner. On the other hand, the output unit 2807 causes the display unit 2820 to display all the region label images generated by the segmentation processing unit 2804, and the evaluation unit 2805 causes the plurality of region label images to be displayed according to an instruction from the examiner. An area label image to be output from the image may be selected. Also in this case, the evaluation unit 2805 may select / determine a plurality of area label images as area label images to be output according to an instruction from the examiner.
（実施例１５）
　次に、図２８及び図４１を参照して、実施例１５に係る画像処理装置について説明する。実施例１４に係る画像処理装置では、評価部２８０５による評価結果が真値である複数の領域ラベル画像について、評価部２８０５が検者の指示に応じて、出力すべき画像を選択／判断した。これに対し、本実施例では、評価部は、所定の選択基準に基づいて、評価結果が真値である複数の領域ラベル画像のうちから出力すべき領域ラベル画像を選択／判断する。 (Example 15)
Next, an image processing apparatus according to a fifteenth embodiment will be described with reference to FIGS. In the image processing apparatus according to the fourteenth embodiment, the evaluation unit 2805 selects / determines an image to be output according to an instruction from the examiner, for a plurality of area label images whose evaluation results by the evaluation unit 2805 are true values. In contrast, in the present embodiment, the evaluation unit selects / determines an area label image to be output from a plurality of area label images whose evaluation results are true values, based on a predetermined selection criterion.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例１４に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例１４に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は、実施例８及び１４に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 限 り Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the fourteenth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the fourteenth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatuses according to the eighth and fourteenth embodiments, the configuration shown in FIG. I do.
　本実施例に係る評価部２８０５は、領域ラベル画像評価エンジンを用いて、セグメンテーション処理部２８０４によって生成された複数の領域ラベル画像を評価し、画像評価指数及び所定の選択基準に応じて出力すべき領域ラベル画像を選択する。 The evaluation unit 2805 according to the present embodiment evaluates a plurality of region label images generated by the segmentation processing unit 2804 using the region label image evaluation engine, and outputs the image according to the image evaluation index and a predetermined selection criterion. Select an area label image.
　以下、図４１を参照して、本実施例に係る一連の画像処理について説明する。なお、本実施例に係るステップＳ４１６０以外の処理は、実施例１４における処理と同様であるため、説明を省略する。なお、入力画像に対して、撮影条件について無条件で画像セグメンテーション処理する場合には、ステップＳ４１２０の処理の後に、ステップＳ４１３０の処理を省き、処理をステップＳ４１４０に移行してよい。 Hereinafter, a series of image processing according to the present embodiment will be described with reference to FIG. Note that processing other than step S4160 according to the present embodiment is the same as the processing according to the fourteenth embodiment, and a description thereof will not be repeated. When the image segmentation process is performed on the input image unconditionally with respect to the shooting conditions, the process of step S4130 may be omitted after the process of step S4120, and the process may proceed to step S4140.
　ステップＳ４１６０では、評価部２８０５は、ステップＳ４１５０における評価結果が真値である領域ラベル画像が複数ある場合、所定の選択基準に応じて、当該複数の領域ラベル画像のうち出力すべき領域ラベル画像を選択／判断する。評価部２８０５は、例えば、時系列的に最初に評価結果として真値が出力された領域ラベル画像を選択する。なお、選択基準はこれに限られず、所望の構成に応じて任意に設定されてよい。評価部２８０５は、例えば、評価結果が真値である領域ラベル画像のうち、入力画像の撮影条件群と学習データの撮影条件の組み合わせが最も近い（合致する）画像セグメンテーションエンジンにより生成された領域ラベル画像を選択／判断してもよい。 In step S4160, when there are a plurality of area label images for which the evaluation result in step S4150 is a true value, the evaluation unit 2805 determines an area label image to be output among the plurality of area label images in accordance with a predetermined selection criterion. Select / judge. The evaluation unit 2805 selects, for example, an area label image to which a true value has been output as an evaluation result first in time series. The selection criterion is not limited to this, and may be set arbitrarily according to a desired configuration. The evaluation unit 2805 may generate, for example, an area label generated by an image segmentation engine in which the combination of the shooting condition group of the input image and the shooting condition of the learning data is the closest (matched) among the area label images whose evaluation results are true values. An image may be selected / determined.
　また、全ての領域ラベル画像に対しての評価結果が偽値であった場合には、評価部２８０５は、画像セグメンテーション処理が適切に行えなかったと判断し、領域無しラベル画像を生成し、出力／選択する。ステップＳ４１７０以降の処理は、実施例１４のステップＳ４１７０以降の処理と同様であるため説明を省略する。 If the evaluation results for all the region label images are false values, the evaluation unit 2805 determines that the image segmentation processing has not been properly performed, generates a region-less label image, and outputs / select. The processing after step S4170 is the same as the processing after step S4170 of the fourteenth embodiment, and a description thereof will not be repeated.
　上記のように、本実施例に係る画像処理装置２８００では、セグメンテーション処理部２８０４は、複数の画像セグメンテーションエンジンを用いて、入力画像から複数の領域ラベル画像を生成する。評価部２８０５は、所定の選択基準に基づいて、出力すべきと評価された領域ラベル画像のうちの少なくとも１つ、又は領域ラベル無し画像を選択する。出力部２８０７は、評価部２８０５によって選択された領域ラベル画像を出力する。 As described above, in the image processing apparatus 2800 according to the present embodiment, the segmentation processing unit 2804 generates a plurality of region label images from an input image using a plurality of image segmentation engines. The evaluation unit 2805 selects, based on a predetermined selection criterion, at least one of the area label images evaluated to be output or an image without an area label. The output unit 2807 outputs the area label image selected by the evaluation unit 2805.
　これにより、本実施例に係る画像処理装置２８００では、領域ラベル画像評価エンジンの出力に基づいて、画像セグメンテーション処理に失敗した領域ラベル画像を出力することを防ぐことができる。また、領域ラベル画像評価エンジンが出力する画像評価指数が真値である領域ラベル画像が複数ある場合に、自動的にその中の一つを選択して表示又は出力することができる。 Accordingly, in the image processing apparatus 2800 according to the present embodiment, it is possible to prevent the output of the area label image in which the image segmentation processing has failed based on the output of the area label image evaluation engine. Further, when there are a plurality of area label images for which the image evaluation index outputted by the area label image evaluation engine is a true value, one of them can be automatically selected and displayed or output.
　なお、本実施例では、画像評価指数が真値である複数の領域ラベル画像のうちの少なくとも一つを選択して出力する構成としたが、画像評価指数が真値である複数の領域ラベル画像の全てを出力する構成としてもよい。この場合には、解析部２８０６は、評価部２８０５から出力された全ての領域ラベル画像について画像解析を行う。また、出力部２８０７は、評価部２８０５から出力された全ての領域ラベル画像及び対応する解析結果を全て表示部２８２０に表示させてもよいし、他の装置に出力してもよい。 In this embodiment, at least one of the plurality of area label images whose image evaluation index is a true value is selected and output. However, the plurality of area label images whose image evaluation index is a true value is selected. May be output. In this case, the analysis unit 2806 performs image analysis on all the region label images output from the evaluation unit 2805. Also, the output unit 2807 may cause the display unit 2820 to display all of the area label images and the corresponding analysis results output from the evaluation unit 2805, or may output them to another device.
（実施例１６）
　次に、図２８及び図２９を参照して、実施例１６に係る画像処理装置について説明する。本実施例では、まず、セグメンテーション処理部が三次元の入力画像を複数の二次元画像（二次元画像群）に分割する。次に、二次元画像群を画像セグメンテーションエンジンに入力し、セグメンテーション処理部が画像セグメンテーションエンジンからの出力画像群を結合することで三次元の領域ラベル画像を生成する。 (Example 16)
Next, an image processing apparatus according to the sixteenth embodiment will be described with reference to FIGS. In this embodiment, first, the segmentation processing unit divides a three-dimensional input image into a plurality of two-dimensional images (two-dimensional image group). Next, a two-dimensional image group is input to an image segmentation engine, and a segmentation processing unit combines the output images from the image segmentation engine to generate a three-dimensional region label image.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例８に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例８に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は、実施例８に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the eighth embodiment, the configuration illustrated in FIG. 28 is denoted by the same reference numeral, and description thereof is omitted.
　本実施例に係る取得部２８０１は、構造的に連続する二次元画像群で構成された、三次元画像を取得する。具体的には、三次元画像は、例えば、ＯＣＴのＢスキャン像（断層画像）群で構成された三次元ＯＣＴボリューム画像である。 取得 The acquisition unit 2801 according to this embodiment acquires a three-dimensional image composed of a structurally continuous two-dimensional image group. Specifically, the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a group of OCT B-scan images (tomographic images).
　セグメンテーション処理部２８０４は、本実施例に係る画像セグメンテーションエンジンを用いて、入力画像である三次元画像についてセグメンテーション処理を行い、複数の二次元の領域ラベル画像を生成する。本実施例に係る画像セグメンテーションエンジンの教師データである入力データと出力データのペア群は、二次元画像の画像群により構成されている。セグメンテーション処理部２８０４は、取得された三次元画像を複数の二次元画像に分割し、二次元画像毎に画像セグメンテーションエンジンに入力する。これにより、セグメンテーション処理部２８０４は、複数の二次元の領域ラベル画像を生成することができる。さらに、セグメンテーション処理部２８０４は、複数の二次元の領域ラベル画像を分割前の二次元画像の配置に並べて結合し、三次元の領域ラベル画像を生成する。 The segmentation processing unit 2804 performs a segmentation process on a three-dimensional image as an input image by using the image segmentation engine according to the present embodiment, and generates a plurality of two-dimensional region label images. A pair group of input data and output data, which are teacher data of the image segmentation engine according to the present embodiment, is configured by a two-dimensional image group. The segmentation processing unit 2804 divides the acquired three-dimensional image into a plurality of two-dimensional images, and inputs the two-dimensional images to the image segmentation engine. Accordingly, the segmentation processing unit 2804 can generate a plurality of two-dimensional region label images. Further, the segmentation processing unit 2804 generates a three-dimensional region label image by arranging and combining a plurality of two-dimensional region label images in an arrangement of the two-dimensional image before division.
　評価部２８０５は、領域ラベル画像評価エンジンを用いて、三次元の領域ラベル画像について尤もらしい領域ラベル画像か否かを判断する。評価結果が真値であった場合には、評価部２８０５は、当該三次元の領域ラベル画像を出力すべき領域ラベル画像として判断し出力する。一方、評価結果が偽値であった場合には、評価部２８０５は、三次元の領域ラベル無し画像を生成し、出力する。 The evaluation unit 2805 determines whether or not the three-dimensional region label image is a likely region label image by using the region label image evaluation engine. When the evaluation result is a true value, the evaluation unit 2805 determines and outputs the three-dimensional area label image as an area label image to be output. On the other hand, when the evaluation result is a false value, the evaluation unit 2805 generates and outputs a three-dimensional region label-less image.
　なお、領域ラベル画像評価エンジンが学習済モデルを含む場合には、当該学習済モデルの教師データとしては、三次元の領域ラベル画像と画像評価指数を用いることができる。また、評価部２８０５は、結合前の二次元の領域ラベル画像の各々について評価を行ってもよい。 When the region label image evaluation engine includes a learned model, a three-dimensional region label image and an image evaluation index can be used as teacher data of the learned model. The evaluation unit 2805 may evaluate each of the two-dimensional region label images before the combination.
　解析部２８０６は、評価部２８０５によって尤もらしい領域ラベル画像と判断された三次元の領域ラベル画像について画像解析処理を行う。なお、解析部２８０６は、結合前の二次元の領域ラベル画像の各々について画像解析処理を行ってもよい。また、評価部２８０５によって三次元の領域ラベル無し画像が出力された場合には、解析部２８０６は画像解析を行わない。 The analysis unit 2806 performs an image analysis process on the three-dimensional region label image determined as a likely region label image by the evaluation unit 2805. Note that the analysis unit 2806 may perform image analysis processing on each of the two-dimensional region label images before the combination. If the evaluation unit 2805 outputs a three-dimensional region label-less image, the analysis unit 2806 does not perform image analysis.
　出力部２８０７は、三次元の領域ラベル画像や解析結果を出力する。なお、出力部２８０７が、生成された三次元の領域ラベル画像を表示部２８２０に表示させる際における、三次元の領域ラベル画像の表示態様は任意であってよい。 The output unit 2807 outputs a three-dimensional region label image and an analysis result. When the output unit 2807 displays the generated three-dimensional region label image on the display unit 2820, the display mode of the three-dimensional region label image may be arbitrary.
　次に、図２９を参照して、本実施例に係る一連の画像処理について説明する。なお、本実施例に係るステップＳ２９１０～ステップＳ２９３０、及びステップＳ２９５０～ステップＳ２９７０の処理は、実施例８におけるこれらの処理と同様であるため、説明を省略する。ただし、ステップＳ２９１０では、取得部２８０１は三次元画像を取得する。なお、入力画像に対して、撮影条件について無条件で画像セグメンテーション処理する場合には、ステップＳ２９２０の処理の後に、ステップＳ２９３０の処理を省き、処理をステップＳ２９４０に移行してよい。 Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processes of steps S2910 to S2930 and steps S2950 to S2970 according to the present embodiment are the same as those of the eighth embodiment, and a description thereof will be omitted. However, in step S2910, the obtaining unit 2801 obtains a three-dimensional image. When the image segmentation process is performed on the input image unconditionally with respect to the shooting conditions, the process of step S2930 may be omitted after the process of step S2920, and the process may proceed to step S2940.
　ステップＳ２９３０において、処理可否判定部２８０３が、画像セグメンテーションエンジンによって入力画像を対処可能と判定した場合には、処理はステップＳ２９４０に移行する。ステップＳ２９４０では、セグメンテーション処理部２８０４が、取得された三次元画像を複数の二次元画像に分割する。セグメンテーション処理部２８０４は、分割した複数の二次元画像のそれぞれを画像セグメンテーションエンジンに入力し、複数の二次元の領域ラベル画像を生成する。セグメンテーション処理部２８０４は、取得した三次元画像に基づいて、生成した複数の二次元の領域ラベル画像を結合し、三次元の領域ラベル画像を生成する。ステップＳ２９５０以降の処理は、実施例８のステップＳ２９５０以降の処理と同様であるため説明を省略する。 In step S2930, if the processing availability determination unit 2803 determines that the input image can be handled by the image segmentation engine, the process proceeds to step S2940. In step S2940, the segmentation processing unit 2804 divides the obtained three-dimensional image into a plurality of two-dimensional images. The segmentation processing unit 2804 inputs each of the divided two-dimensional images to the image segmentation engine, and generates a plurality of two-dimensional region label images. The segmentation processing unit 2804 combines the generated two-dimensional region label images based on the acquired three-dimensional image, and generates a three-dimensional region label image. The processing after step S2950 is the same as the processing after step S2950 in the eighth embodiment, and a description thereof will not be repeated.
　上記のように、本実施例に係るセグメンテーション処理部２８０４は、入力画像を入力画像の次元よりも低い次元の複数の画像に分割し、分割した画像毎にセグメンテーションエンジン入力する。より具体的には、セグメンテーション処理部２８０４は、三次元の入力画像を複数の二次元の画像に分割して画像セグメンテーションエンジンに入力する。セグメンテーション処理部２８０４は、画像セグメンテーションエンジンから出力された複数の二次元の領域ラベル画像を結合し、三次元の領域ラベル画像を生成する。 As described above, the segmentation processing unit 2804 according to the present embodiment divides an input image into a plurality of images having dimensions lower than the dimensions of the input image, and inputs the divided images to the segmentation engine. More specifically, the segmentation processing unit 2804 divides a three-dimensional input image into a plurality of two-dimensional images and inputs the divided two-dimensional images to an image segmentation engine. The segmentation processing unit 2804 combines a plurality of two-dimensional region label images output from the image segmentation engine to generate a three-dimensional region label image.
　これにより、本実施例に係るセグメンテーション処理部２８０４は、二次元画像の教師データを用いて学習が行われた学習済モデルを含む画像セグメンテーションエンジンを用いて、三次元画像を画像セグメンテーション処理することができる。 Accordingly, the segmentation processing unit 2804 according to the present embodiment can perform the image segmentation processing on the three-dimensional image using the image segmentation engine including the trained model trained using the teacher data of the two-dimensional image. it can.
　なお、本実施例に係るセグメンテーション処理部２８０４は、三次元の入力画像を複数の二次元の画像に分割して、画像セグメンテーション処理を行った。しかしながら、当該分割に係る処理を行う対象は三次元の入力画像に限られない。例えば、セグメンテーション処理部２８０４は、二次元の入力画像を複数の一次元の画像に分割して、画像セグメンテーション処理を行ってもよい。また、セグメンテーション処理部２８０４は、四次元の入力画像を複数の三次元の画像や複数の二次元の画像に分割して、画像セグメンテーション処理を行ってもよい。 Note that the segmentation processing unit 2804 according to the present embodiment divides a three-dimensional input image into a plurality of two-dimensional images and performs image segmentation processing. However, the target for performing the processing related to the division is not limited to the three-dimensional input image. For example, the segmentation processing unit 2804 may divide the two-dimensional input image into a plurality of one-dimensional images and perform the image segmentation processing. In addition, the segmentation processing unit 2804 may divide the four-dimensional input image into a plurality of three-dimensional images or a plurality of two-dimensional images, and perform the image segmentation processing.
（実施例１７）
　次に、図２８及び図２９を参照して、実施例１７に係る画像処理装置について説明する。本実施例では、セグメンテーション処理部が三次元の入力画像を複数の二次元画像に分割し、複数の二次元画像を複数の画像セグメンテーションエンジンによって並列に画像セグメンテーション処理する。その後、セグメンテーション処理部が、各画像セグメンテーションエンジンからの出力画像を結合することで三次元の領域ラベル画像を生成する。 (Example 17)
Next, an image processing apparatus according to the seventeenth embodiment will be described with reference to FIGS. In the present embodiment, the segmentation processing unit divides the three-dimensional input image into a plurality of two-dimensional images, and performs the image segmentation processing on the two-dimensional images in parallel by the plurality of image segmentation engines. Then, the segmentation processing unit generates a three-dimensional region label image by combining the output images from the respective image segmentation engines.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例１６に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例１６に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は、実施例８及び１６に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the sixteenth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the sixteenth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatuses according to the eighth and sixteenth embodiments, the configuration illustrated in FIG. I do.
　本実施例に係るセグメンテーション処理部２８０４は、実施例１６と同様な複数の画像セグメンテーションエンジンを用いて、入力画像である三次元画像について画像セグメンテーション処理を行い、三次元の領域ラベル画像を生成する。なお、セグメンテーション処理部２８０４が用いる複数の画像セグメンテーションエンジン群は、回路やネットワークを介して、二つ以上の装置群に分散処理可能なように実装されていてもよいし、単一の装置に実装されていてもよい。 セ The segmentation processing unit 2804 according to the present embodiment performs image segmentation processing on a three-dimensional image as an input image using a plurality of image segmentation engines similar to those in the sixteenth embodiment, and generates a three-dimensional region label image. Note that the plurality of image segmentation engines used by the segmentation processing unit 2804 may be mounted so as to be distributed to two or more devices via a circuit or a network, or may be mounted on a single device. It may be.
　セグメンテーション処理部２８０４は、実施例１６と同様に、取得された三次元画像を複数の二次元画像に分割する。セグメンテーション処理部２８０４は、複数の二次元画像について複数の画像セグメンテーションエンジンを用いて、分担して（並列的に）画像セグメンテーション処理を行い、複数の二次元の領域ラベル画像を生成する。セグメンテーション処理部２８０４は、複数の画像セグメンテーションエンジンから出力された複数の二次元の領域ラベル画像を、処理対象である三次元画像に基づいて結合し、三次元の領域ラベル画像を生成する。より具体的には、セグメンテーション処理部２８０４は、複数の二次元の領域ラベル画像を分割前の二次元画像の配置に並べて結合し、三次元の領域ラベル画像を生成する。 The segmentation processing unit 2804 divides the obtained three-dimensional image into a plurality of two-dimensional images, as in the sixteenth embodiment. A segmentation processing unit 2804 performs image segmentation processing on a plurality of two-dimensional images by sharing (in parallel) using a plurality of image segmentation engines to generate a plurality of two-dimensional region label images. The segmentation processing unit 2804 combines a plurality of two-dimensional region label images output from the plurality of image segmentation engines based on a three-dimensional image to be processed, and generates a three-dimensional region label image. More specifically, the segmentation processing unit 2804 generates a three-dimensional region label image by arranging and combining a plurality of two-dimensional region label images in an arrangement of the two-dimensional image before division.
　次に、図２９を参照して、本実施例に係る一連の画像処理について説明する。なお、本実施例に係るステップＳ２９１０～ステップＳ２９３０、及びステップＳ２９５０～ステップＳ２９７０の処理は、実施例１６におけるこれらの処理と同様であるため、説明を省略する。なお、入力画像に対して、撮影条件について無条件で画像セグメンテーション処理する場合には、ステップＳ２９２０の処理の後に、ステップＳ２９３０の処理を省き、処理をステップＳ２９４０に移行してよい。 Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processes of steps S2910 to S2930 and steps S2950 to S2970 according to the present embodiment are the same as those of the sixteenth embodiment, and a description thereof will be omitted. When the image segmentation process is performed on the input image unconditionally with respect to the shooting conditions, the process of step S2930 may be omitted after the process of step S2920, and the process may proceed to step S2940.
　ステップＳ２９３０において、処理可否判定部２８０３が、画像セグメンテーションエンジンによって入力画像を対処可能と判定した場合には、処理はステップＳ２９４０に移行する。ステップＳ２９４０では、セグメンテーション処理部２８０４が、取得された三次元画像を複数の二次元画像に分割する。セグメンテーション処理部２８０４は、分割した複数の二次元画像のそれぞれを複数の画像セグメンテーションエンジンに入力し、並列的に画像セグメンテーション処理して、複数の二次元の領域ラベル画像を生成する。セグメンテーション処理部２８０４は、取得した三次元画像に基づいて、生成した複数の二次元の領域ラベル画像を結合し、三次元の領域ラベル画像を生成する。 In step S2930, if the processing availability determination unit 2803 determines that the input image can be handled by the image segmentation engine, the process proceeds to step S2940. In step S2940, the segmentation processing unit 2804 divides the obtained three-dimensional image into a plurality of two-dimensional images. The segmentation processing unit 2804 inputs each of the divided two-dimensional images to a plurality of image segmentation engines, performs image segmentation processing in parallel, and generates a plurality of two-dimensional region label images. The segmentation processing unit 2804 combines the generated two-dimensional region label images based on the acquired three-dimensional image, and generates a three-dimensional region label image.
　上記のように、本実施例に係るセグメンテーション処理部２８０４は、複数の画像セグメンテーションエンジンを含む。セグメンテーション処理部２８０４は、三次元の入力画像を複数の二次元の画像に分割し、分割した複数の二次元の画像について複数の画像セグメンテーションエンジンを並列的に用いて、複数の二次元の領域ラベル画像を生成する。セグメンテーション処理部２８０４は複数の二次元の領域ラベル画像を統合することで、三次元の領域ラベル画像を生成する。 As described above, the segmentation processing unit 2804 according to the present embodiment includes a plurality of image segmentation engines. A segmentation processing unit 2804 divides a three-dimensional input image into a plurality of two-dimensional images, and uses a plurality of image segmentation engines in parallel for the plurality of divided two-dimensional images to generate a plurality of two-dimensional region labels. Generate an image. The segmentation processing unit 2804 generates a three-dimensional region label image by integrating a plurality of two-dimensional region label images.
　これにより、本実施例に係るセグメンテーション処理部２８０４は、二次元画像の教師データを用いて学習が行われた学習済モデルを含む画像セグメンテーションエンジンを用いて、三次元画像を画像セグメンテーション処理することができる。また、実施例１６と比べて、より効率的に三次元画像を画像セグメンテーション処理することができる。 Accordingly, the segmentation processing unit 2804 according to the present embodiment can perform the image segmentation processing on the three-dimensional image using the image segmentation engine including the trained model trained using the teacher data of the two-dimensional image. it can. In addition, as compared with the sixteenth embodiment, a three-dimensional image can be more efficiently subjected to image segmentation processing.
　なお、実施例１６と同様に、セグメンテーション処理部２８０４による分割に係る処理を行う対象は三次元の入力画像に限られない。例えば、セグメンテーション処理部２８０４は、二次元の入力画像を複数の一次元の画像に分割して、画像セグメンテーション処理を行ってもよい。また、セグメンテーション処理部２８０４は、四次元の入力画像を複数の三次元の画像や複数の二次元の画像に分割して、画像セグメンテーション処理を行ってもよい。 Note that, similarly to the sixteenth embodiment, the target to be subjected to the division-related processing by the segmentation processing unit 2804 is not limited to the three-dimensional input image. For example, the segmentation processing unit 2804 may divide the two-dimensional input image into a plurality of one-dimensional images and perform the image segmentation processing. In addition, the segmentation processing unit 2804 may divide the four-dimensional input image into a plurality of three-dimensional images or a plurality of two-dimensional images, and perform the image segmentation processing.
　また、複数の画像セグメンテーションエンジンの教師データは、各画像セグメンテーションエンジンで処理を行う処理対象に応じて異なる教師データであってもよい。例えば、第一の画像セグメンテーションエンジンは第一の撮影領域についての教師データで学習を行い、第二の画像セグメンテーションエンジンは第二の撮影領域についての教師データで学習を行ってもよい。この場合には、それぞれの画像セグメンテーションエンジンが、より精度高く二次元画像の画像セグメンテーション処理を行うことができる。 The teacher data of the plurality of image segmentation engines may be different teacher data depending on a processing target to be processed by each image segmentation engine. For example, the first image segmentation engine may learn with teacher data for a first shooting region, and the second image segmentation engine may learn with teacher data for a second shooting region. In this case, each image segmentation engine can perform image segmentation processing of a two-dimensional image with higher accuracy.
　さらに、評価部２８０５が、セグメンテーション処理部２８０４と同様に、学習済モデルを含む複数の領域ラベル画像評価エンジンを用いて、三次元の領域ラベル画像を並列的に評価することもできる。この場合には、評価部２８０５は、セグメンテーション処理部２８０４によって生成された複数の二次元の領域ラベル画像について、複数の領域ラベル画像評価エンジンを並列的に用いて、評価を行う。 評 価 Furthermore, the evaluation unit 2805 can evaluate a three-dimensional region label image in parallel using a plurality of region label image evaluation engines including the learned model, similarly to the segmentation processing unit 2804. In this case, the evaluation unit 2805 evaluates a plurality of two-dimensional region label images generated by the segmentation processing unit 2804 using a plurality of region label image evaluation engines in parallel.
　その後、各二次元の領域ラベル画像についての画像評価指数が真値である場合には、評価部２８０５は、三次元の領域ラベル画像が尤もらしい領域ラベル画像であると判断して出力することができる。この場合、領域ラベル画像評価エンジンが含む学習済モデルの教師データは、二次元の領域ラベル画像と画像評価指数により構成することができる。なお、評価部２８０５は、各二次元の領域ラベル画像の一部についての画像評価指数が真値である場合に、三次元の領域ラベル画像が尤もらしい領域ラベル画像であると判断して出力することもできる。 Thereafter, when the image evaluation index of each two-dimensional region label image is a true value, the evaluation unit 2805 may determine that the three-dimensional region label image is a likely region label image and output it. it can. In this case, the teacher data of the trained model included in the area label image evaluation engine can be constituted by a two-dimensional area label image and an image evaluation index. When the image evaluation index for a part of each two-dimensional region label image is a true value, the evaluation unit 2805 determines that the three-dimensional region label image is a likely region label image and outputs it. You can also.
（実施例１８）
　次に、図２９及び図４３を参照して、実施例１８に係る画像処理装置について説明する。本実施例では、取得部が撮影装置ではなく画像管理システムから入力画像を取得する。 (Example 18)
Next, an image processing apparatus according to the eighteenth embodiment will be described with reference to FIGS. In the present embodiment, the acquisition unit acquires the input image from the image management system instead of the imaging device.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例８に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例８に係る画像処理装置との違いを中心として説明する。なお、本実施例に係る画像処理装置の構成は実施例８に係る画像処理装置２８００の構成と同様であるため、図２８に示す構成について同じ参照符号を用いて説明を省略する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus 2800 according to the eighth embodiment, the description of the configuration illustrated in FIG.
　図４３は、本実施例に係る画像処理装置２８００の概略的な構成を示す。本実施例に係る画像処理装置２８００は画像管理システム４３００、及び表示部２８２０と任意の回路やネットワークを介して接続されている。画像管理システム４３００は、任意の撮影装置によって撮影された画像や画像処理された画像を受信して保存する装置及びシステムである。また、画像管理システム４３００は、接続された装置の要求に応じて画像を送信したり、保存された画像に対して画像処理を行ったり、画像処理の要求を他の装置に要求したりすることができる。画像管理システム４３００は、例えば、画像保存通信システム（ＰＡＣＳ）を含むことができる。 FIG. 43 illustrates a schematic configuration of an image processing apparatus 2800 according to the present embodiment. The image processing apparatus 2800 according to the present embodiment is connected to the image management system 4300 and the display unit 2820 via an arbitrary circuit or a network. The image management system 4300 is a device and a system that receives and stores an image photographed by an arbitrary photographing device or an image processed image. Further, the image management system 4300 transmits an image in response to a request from a connected device, performs image processing on a stored image, and requests another device for a request for image processing. Can be. The image management system 4300 can include, for example, an image storage and communication system (PACS).
　本実施例に係る取得部２８０１は、画像処理装置２８００に接続される画像管理システム４３００から入力画像を取得することができる。また、出力部２８０７は、セグメンテーション処理部２８０４によって生成された領域ラベル画像を、画像管理システム４３００に出力することができる。なお、出力部２８０７は実施例８と同様に、領域ラベル画像を表示部２８２０に表示させることもできる The acquisition unit 2801 according to the embodiment can acquire an input image from the image management system 4300 connected to the image processing device 2800. The output unit 2807 can output the area label image generated by the segmentation processing unit 2804 to the image management system 4300. Note that the output unit 2807 can display an area label image on the display unit 2820 as in the eighth embodiment.
　次に、図２９を参照して、本実施例に係る一連の画像処理について説明する。なお、本実施例に係るステップＳ２９２０～ステップＳ２９６０の処理は、実施例８におけるこれらの処理と同様であるため、説明を省略する。なお、入力画像に対して、撮影条件について無条件で画像セグメンテーション処理する場合には、ステップＳ２９２０の処理の後に、ステップＳ２９３０の処理を省き、処理をステップＳ２９４０に移行してよい。 Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processing in steps S2920 to S2960 according to the present embodiment is the same as the processing in the eighth embodiment, and a description thereof will be omitted. When the image segmentation process is performed on the input image unconditionally with respect to the shooting conditions, the process of step S2930 may be omitted after the process of step S2920, and the process may proceed to step S2940.
　ステップＳ２９１０において、取得部２８０１は、回路やネットワークを介して接続された画像管理システム４３００から、画像管理システム４３００が保存している画像を入力画像として取得する。なお、取得部２８０１は、画像管理システム４３００からの要求に応じて、入力画像を取得してもよい。このような要求は、例えば、画像管理システム４３００が画像を保存した時や、保存した画像を他の装置に送信する前、保存された画像を表示部２８２０に表示する時に発行されてよい。また、当該要求は、例えば、画像管理システム４３００を利用者が操作して画像セグメンテーション処理の要求を行った時や、画像管理システム４３００が備える画像解析機能に領域ラベル画像を利用する時等に発行されてよい。 In step S2910, the acquiring unit 2801 acquires, as an input image, an image stored in the image management system 4300 from the image management system 4300 connected via a circuit or a network. Note that the acquisition unit 2801 may acquire an input image in response to a request from the image management system 4300. Such a request may be issued, for example, when the image management system 4300 stores the image, before transmitting the stored image to another device, or when displaying the stored image on the display unit 2820. The request is issued, for example, when a user operates the image management system 4300 to make a request for an image segmentation process, or when using an area label image for an image analysis function of the image management system 4300. May be.
　ステップＳ２９２０～ステップＳ２９６０の処理は、実施例８における処理と同様である。ステップＳ２９７０においては、出力部２８０７は、ステップＳ２９５０において評価部２８０５によって領域ラベル画像を出力すると判断されたら、領域ラベル画像を画像管理システム４３００に出力画像として出力する。なお、出力部２８０７は、画像処理装置２８００の設定や実装によっては、出力画像を画像管理システム４３００が利用可能なように加工したり、出力画像のデータ形式を変換したりしてもよい。また、出力部２８０７は、解析部２８０６による解析結果も画像管理システム４３００に出力することができる。 処理 The processing in steps S2920 to S2960 is the same as the processing in the eighth embodiment. In step S2970, the output unit 2807 outputs the area label image to the image management system 4300 as an output image when the evaluation unit 2805 determines in step S2950 to output the area label image. Note that the output unit 2807 may process the output image so that it can be used by the image management system 4300 or may convert the data format of the output image depending on the settings and implementation of the image processing device 2800. The output unit 2807 can also output the analysis result by the analysis unit 2806 to the image management system 4300.
　一方、ステップＳ２９５０において、評価部２８０５によって画像セグメンテーション処理を適切に行えなかったと判断したら、領域ラベル無し画像を画像管理システム４３００に出力画像として出力する。また、ステップＳ２９３０において、処理可否判定部２８０３が、入力画像を画像セグメンテーション処理不可能と判定した場合にも、出力部２８０７は、領域ラベル無し画像を画像管理システム４３００に出力する。 On the other hand, in step S2950, if the evaluation unit 2805 determines that the image segmentation process has not been properly performed, the image without the region label is output to the image management system 4300 as an output image. Also, in step S2930, when the processing availability determination unit 2803 determines that the input image cannot be subjected to the image segmentation process, the output unit 2807 outputs the image without the region label to the image management system 4300.
　上記のように、本実施例に係る取得部２８０１は、画像管理システム４３００から入力画像を取得する。このため、本実施例の画像処理装置２８００は、画像管理システム４３００が保存している画像を元に、画像診断に適した領域ラベル画像を、撮影者や被検者の侵襲性を高めたり、労力を増したりすることなく出力することができる。また、出力された領域ラベル画像や画像解析結果は画像管理システム４３００に保存されたり、画像管理システム４３００が備えるユーザーインターフェースに表示されたりすることができる。また、出力された領域ラベル画像は、画像管理システム４３００が備える画像解析機能に利用されたり、画像管理システム４３００に接続された他の装置に画像管理システム４３００を介して送信されたりすることができる。 As described above, the acquisition unit 2801 according to the present embodiment acquires an input image from the image management system 4300. For this reason, the image processing device 2800 of the present embodiment increases the invasiveness of the photographer or the subject based on the image stored in the image management system 4300, and generates an area label image suitable for image diagnosis. The output can be made without increasing labor. The output area label image and the image analysis result can be stored in the image management system 4300 or displayed on a user interface provided in the image management system 4300. Further, the output region label image can be used for an image analysis function provided in the image management system 4300, or transmitted to another device connected to the image management system 4300 via the image management system 4300. .
　なお、画像処理装置２８００や画像管理システム４３００、表示部２８２０は、不図示の他の装置と回路やネットワークを介して接続されていてもよい。また、これらの装置は本実施例では別個の装置とされているが、これらの装置の一部又は全部を一体的に構成してもよい。 The image processing device 2800, the image management system 4300, and the display unit 2820 may be connected to another device (not shown) via a circuit or a network. Although these devices are separate devices in this embodiment, a part or all of these devices may be integrally formed.
（実施例１９）
　次に、図４４及び図４５を参照して、実施例１９に係る画像処理装置について説明する。本実施例では、修正部が、領域ラベル画像修正エンジンを用いて、画像セグメンテーションエンジンから出力された領域ラベル画像に誤った領域ラベル値が設定されていれば修正する。 (Example 19)
Next, an image processing apparatus according to the nineteenth embodiment will be described with reference to FIGS. 44 and 45. In the present embodiment, the correction unit uses the area label image correction engine to correct an incorrect area label value in the area label image output from the image segmentation engine.
　特に明記しない限り、本実施例に係る画像処理装置の構成及び処理は、実施例８に係る画像処理装置２８００と同様である。そのため、以下では、本実施例に係る画像処理装置について、実施例８に係る画像処理装置との違いを中心として説明する。 構成 Unless otherwise specified, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 2800 according to the eighth embodiment. Therefore, hereinafter, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus according to the eighth embodiment.
　図４４は、本実施例に係る画像処理装置４４００の概略的な構成を示す。本実施例に係る画像処理装置４４００には、取得部２８０１、撮影条件取得部２８０２、処理可否判定部２８０３、セグメンテーション処理部２８０４、評価部２８０５、解析部２８０６及び出力部２８０７に加えて、修正部４４０８が設けられている。なお、画像処理装置４４００は、これら構成要素のうちの一部が設けられた複数の装置で構成されてもよい。ここで、本実施例に係る画像処理装置４４００における修正部４４０８以外の構成は、実施例８に係る画像処理装置の構成と同様であるため、図２８に示す構成について同一の参照符号を用いて示し、説明を省略する。 FIG. 44 shows a schematic configuration of an image processing device 4400 according to the present embodiment. The image processing apparatus 4400 according to the present embodiment includes an acquisition unit 2801, a shooting condition acquisition unit 2802, a processability determination unit 2803, a segmentation processing unit 2804, an evaluation unit 2805, an analysis unit 2806, and an output unit 2807, and a correction unit. 4408 are provided. Note that the image processing device 4400 may be configured by a plurality of devices in which some of these components are provided. Here, since the configuration of the image processing apparatus 4400 according to the present embodiment other than the correction unit 4408 is the same as the configuration of the image processing apparatus according to the eighth embodiment, the same reference numerals are used for the configuration shown in FIG. The description is omitted.
　また、画像処理装置４４００は、実施例８に係る画像処理装置２８００と同様に撮影装置２８１０、表示部２８２０及び不図示の他の装置と、任意の回路やネットワークを介して接続されてよい。また、これらの装置は、他の任意の装置と回路やネットワークを介して接続されてもよいし、他の任意の装置と一体的に構成されてもよい。なお、これらの装置は本実施例では別個の装置とされているが、これらの装置の一部又は全部を一体的に構成してもよい。 The image processing device 4400 may be connected to the image capturing device 2810, the display unit 2820, and other devices (not shown) via an arbitrary circuit or a network, similarly to the image processing device 2800 according to the eighth embodiment. These devices may be connected to any other device via a circuit or a network, or may be configured integrally with any other device. Although these devices are separate devices in this embodiment, a part or all of these devices may be integrally configured.
　本実施例に係る修正部４４０８には、入力された領域ラベル画像を修正する領域ラベル画像修正エンジンが備えられている。領域ラベル画像修正エンジンは、用語の説明において上述したような、解剖学的な知識ベース処理による領域ラベル値の修正を行う。なお、例えば、修正対象である領域ラベル値の連続領域を上書きする領域ラベル値については、該連続領域に接している画素の数が最も多い領域ラベル値で上書きするものとする。 修正 The correction unit 4408 according to the present embodiment includes an area label image correction engine that corrects an input area label image. The region label image correction engine corrects a region label value by anatomical knowledge base processing as described above in the description of terms. Note that, for example, an area label value overwriting a continuous area of an area label value to be corrected is overwritten with an area label value having the largest number of pixels in contact with the continuous area.
　次に図４５を参照して、本実施例に係る一連の画像処理について説明する。なお、本実施例に係るステップＳ４５１０～ステップＳ４５４０の処理は、それぞれ、実施例８におけるステップＳ２９１０～ステップＳ２９４０の処理と同様であるため、説明を省略する。なお、入力画像に対して、撮影条件について無条件で画像セグメンテーション処理する場合には、ステップＳ４５２０の処理の後に、ステップＳ４５３０の処理を省き、処理をステップＳ４５４０に移行してよい。 Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processing in steps S4510 to S4540 according to the present embodiment is the same as the processing in steps S2910 to S2940 in the eighth embodiment, and a description thereof will be omitted. When the image segmentation process is performed on the input image unconditionally with respect to the photographing conditions, the process of step S4530 may be omitted after the process of step S4520, and the process may proceed to step S4540.
　ステップＳ４５４０において、セグメンテーション処理部２８０４が、領域ラベル画像を生成したら、処理はステップＳ４５５０に移行する。ステップＳ４５５０において、評価部２８０５は、実施例８と同様に、領域ラベル画像評価エンジンを用いて、生成された領域ラベル画像を評価する。評価部２８０５は、評価結果が真値である場合には、当該領域ラベル画像を出力すべき領域ラベル画像として判断する。一方、本実施例に係る評価部２８０５は、評価結果が偽値である場合には、当該領域ラベル画像は修正が必要な領域ラベル画像であると判断する。 In step S4540, when the segmentation processing unit 2804 generates an area label image, the process proceeds to step S4550. In step S4550, the evaluation unit 2805 evaluates the generated region label image using the region label image evaluation engine, as in the eighth embodiment. When the evaluation result is a true value, the evaluation unit 2805 determines that the area label image is an area label image to be output. On the other hand, when the evaluation result is a false value, the evaluation unit 2805 according to the present embodiment determines that the area label image is an area label image that needs to be corrected.
　ステップＳ４５６０では、修正部４４０８が、ステップＳ４５４０において修正が必要な領域ラベル画像として判断された領域ラベル画像に対し、領域ラベル修正エンジンを用いて領域ラベル値の修正を行う。具体的には、修正部４４０８が、ステップＳ４５４０において修正が必要であると判断された領域ラベル画像を、領域ラベル画像修正エンジンに入力する。領域ラベル画像修正エンジンは、入力された領域ラベル画像について、解剖学的な知識ベース処理に従って、誤って設定された領域ラベル値を修正し、修正された領域ラベル画像を出力する。 In step S4560, the correction unit 4408 corrects the area label value of the area label image determined as the area label image requiring correction in step S4540 using the area label correction engine. Specifically, the correction unit 4408 inputs the region label image determined to need correction in step S4540 to the region label image correction engine. The region label image correction engine corrects an erroneously set region label value of the input region label image according to anatomical knowledge base processing, and outputs the corrected region label image.
　なお、ステップＳ４５５０において、生成された領域ラベル画像が出力すべき領域ラベル画像であると判断された場合には、修正部４４０８は、領域ラベル画像の修正を行わずに、処理を進める。 If it is determined in step S4550 that the generated area label image is an area label image to be output, the correction unit 4408 proceeds with the processing without correcting the area label image.
　ステップＳ４５７０では、解析部２８０６が、ステップＳ４５４０において出力すべき領域ラベル画像であると判断された領域ラベル画像、又は、ステップＳ４５５０において領域ラベルの修正が行われた領域ラベル画像を用いて、入力画像の画像解析処理を行う。画像解析処理の内容は実施例８と同様であってよいため、説明を省略する。 In step S4570, the analysis unit 2806 uses the area label image determined to be the area label image to be output in step S4540 or the area label image in which the area label has been corrected in step S4550, and Image analysis processing. The content of the image analysis processing may be the same as that of the eighth embodiment, and the description is omitted.
　ステップＳ４５８０においては、出力部２８０７が、出力すべき領域ラベル画像として判断された領域ラベル画像又は領域ラベルが修正された領域ラベル画像及び画像解析結果を表示部２８２０に表示させる。なお、出力部２８０７は、表示部２８２０に領域ラベル画像及び画像解析結果を表示させるのに代えて、撮影装置２８１０や他の装置にこれらを表示させたり、記憶させたりしてもよい。また、出力部２８０７は、画像処理装置４４００の設定や実装形態によっては、これらを撮影装置２８１０や他の装置が利用可能なように加工したり、画像管理システム等に送信可能なようにデータ形式を変換したりしてもよい。また、出力部２８０７は、領域ラベル画像及び画像解析結果の両方を出力する構成に限られず、これらのうちのいずれか一方のみを出力してもよい。 In step S4580, the output unit 2807 causes the display unit 2820 to display the area label image determined as the area label image to be output or the area label image with the corrected area label and the image analysis result. Note that the output unit 2807 may display or store these on the imaging device 2810 or another device instead of displaying the region label image and the image analysis result on the display unit 2820. The output unit 2807 may process the image processing device 4400 so that it can be used by the imaging device 2810 or another device, or may transmit the data to an image management system or the like, depending on the setting or the implementation form of the image processing device 4400. May be converted. Also, the output unit 2807 is not limited to a configuration that outputs both the area label image and the image analysis result, and may output only one of them.
　一方、ステップＳ４５３０において画像セグメンテーション処理が不可能であるとされていた場合には、出力部２８０７は、領域ラベル無し画像を出力し、表示部２８２０に表示させる。なお、領域ラベル無し画像を出力する代わりに、撮影装置２８１０に対して、画像セグメンテーション処理が不可能であったことを示す信号を送信してもよい。ステップＳ４５８０における出力処理が終了すると、一連の画像処理が終了する。 On the other hand, if it is determined in step S4530 that the image segmentation process cannot be performed, the output unit 2807 outputs an image without an area label and causes the display unit 2820 to display the image. Note that, instead of outputting the image without the region label, a signal indicating that the image segmentation processing has been impossible may be transmitted to the imaging device 2810. When the output processing in step S4580 ends, a series of image processing ends.
　上記のように、本実施例に係る画像処理装置４４００は、修正部４４０８を更に備える。修正部４４０８は、所定の修正手法による知識ベース処理を行う領域ラベル画像修正エンジンを用いて、セグメンテーション処理部２８０４によって生成された領域ラベル画像を修正する。出力部２８０７は、修正部４４０８によって修正された領域ラベル画像を出力する。 As described above, the image processing device 4400 according to the present embodiment further includes the correction unit 4408. The correction unit 4408 corrects the area label image generated by the segmentation processing unit 2804 using an area label image correction engine that performs a knowledge base process using a predetermined correction method. The output unit 2807 outputs the area label image corrected by the correction unit 4408.
　特に、本実施例に係る修正部４４０８は、評価部２８０５によって、画像セグメンテーション処理が適切に行えなかったと判断された領域ラベル画像について、領域ラベルを修正する。また、解析部２８０６は、領域ラベルが修正された領域ラベル画像について画像解析処理を行う。 In particular, the correction unit 4408 according to the present embodiment corrects the region label of the region label image determined by the evaluation unit 2805 that the image segmentation processing has not been properly performed. In addition, the analysis unit 2806 performs an image analysis process on the area label image whose area label has been corrected.
　これにより、本実施例に係る画像処理装置２８００では、領域ラベル画像修正エンジンによって画像セグメンテーション処理に失敗した領域ラベル画像の誤りを訂正し、出力することができる。 Accordingly, in the image processing apparatus 2800 according to the present embodiment, the error of the area label image for which the image segmentation processing has failed by the area label image correction engine can be corrected and output.
　なお、本実施例では、修正部４４０８は、評価部２８０５による評価結果が偽値である領域ラベル画像について領域ラベルを修正した。しかしながら、修正部４４０８の構成はこれに限られない。修正部４４０８は、評価部２８０５による評価結果が真値である領域ラベル画像について、領域ラベルを修正してもよい。この場合、解析部２８０６は、修正された領域ラベルを用いて入力画像の画像解析処理を行う。また、出力部２８０７は、修正された領域ラベル画像やその解析結果を出力する。 In the embodiment, the correction unit 4408 corrects the area label of the area label image whose evaluation result by the evaluation unit 2805 is a false value. However, the configuration of the correction unit 4408 is not limited to this. The correction unit 4408 may correct the region label for the region label image whose evaluation result by the evaluation unit 2805 is a true value. In this case, the analysis unit 2806 performs an image analysis process on the input image using the corrected area label. The output unit 2807 outputs the corrected area label image and the analysis result.
　さらに、この場合には、評価部２８０５は、評価結果が偽値である領域ラベル画像については修正部４４０８によって領域ラベルの修正を行わせないように、評価結果が偽値であったら領域ラベル無し画像を生成することもできる。修正部４４０８は、評価部２８０５によって領域ラベル無し画像が生成されたら、修正を行わずに処理を先に進めることができる。 Further, in this case, if the evaluation result is a false value, the evaluation unit 2805 determines that there is no area label so that the correction unit 4408 does not correct the area label for the area label image whose evaluation result is a false value. Images can also be generated. When the evaluation unit 2805 generates an image without an area label, the correction unit 4408 can proceed with the processing without performing the correction.
　評価部２８０５による評価結果が真値であるときに、修正部４４０８によって領域ラベル画像を修正する場合には、画像処理装置２８００は、より精度の高い領域ラベル画像や解析結果を出力できる。 (4) When the correction unit 4408 corrects the region label image when the evaluation result by the evaluation unit 2805 is a true value, the image processing apparatus 2800 can output a more accurate region label image or analysis result.
　上述の実施例８乃至１９では、セグメンテーション処理部２８０４が解剖学的な領域を識別可能な領域情報として領域ラベル画像を生成する構成を説明したが、セグメンテーション処理部２８０４が生成する領域情報はこれに限られない。セグメンテーション処理部が、画像セグメンテーションエンジンを用いて入力画像から生成する領域情報としては、各領域ラベルを有する画素の座標値等の数値データ群等であってもよい。 In the above-described embodiments 8 to 19, the configuration in which the segmentation processing unit 2804 generates the region label image as the region information capable of identifying the anatomical region has been described. However, the region information generated by the segmentation processing unit 2804 includes the region information. Not limited. The region information generated by the segmentation processing unit from the input image using the image segmentation engine may be a group of numerical data such as coordinate values of pixels having each region label.
　なお、画像セグメンテーションエンジンや、領域ラベル画像評価エンジン、撮影箇所推定エンジンに含まれるそれぞれ学習済モデルは、画像処理装置２８００，４４００に設けられることができる。学習済モデルは、例えば、ＣＰＵや、ＭＰＵ、ＧＰＵ、ＦＰＧＡ等のプロセッサーによって実行されるソフトウェアモジュール等で構成されてもよいし、ＡＳＩＣ等の特定の機能を果たす回路等によって構成されてもよい。また、学習済モデルは、画像処理装置２８００，４４００と接続される別のサーバの装置等に設けられてもよい。この場合には、画像処理装置２８００，４４００は、インターネット等の任意のネットワークを介して学習済モデルを備えるサーバ等に接続することで、学習済モデルを用いることができる。ここで、学習済モデルを備えるサーバは、例えば、クラウドサーバや、フォグサーバ、エッジサーバ等であってよい。 Note that each of the learned models included in the image segmentation engine, the region label image evaluation engine, and the shooting location estimation engine can be provided in the image processing devices 2800 and 4400. The learned model may be configured by, for example, a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may be configured by a circuit that performs a specific function such as an ASIC. Further, the learned model may be provided in a device of another server connected to the image processing devices 2800 and 4400 or the like. In this case, the image processing apparatuses 2800 and 4400 can use the learned model by connecting to a server or the like having the learned model via an arbitrary network such as the Internet. Here, the server including the learned model may be, for example, a cloud server, a fog server, an edge server, or the like.
　実施例８乃至１９によれば、従来の画像セグメンテーション処理よりも精度の高い画像セグメンテーション処理を実施できる。 According to the eighth to nineteenth embodiments, it is possible to perform image segmentation processing with higher accuracy than the conventional image segmentation processing.
（変形例１）
　実施例１乃至７及びそれらの変形例では、処理部２２２又は第一の処理部８２２が学習済モデルを用いて、断層画像から網膜層を検出し、境界画像を生成した。また、実施例９乃至１９では、セグメンテーション処理部２８０４が、学習済モデルを含む画像セグメンテーションエンジンを用いて、入力された画像に対応する領域ラベル画像を生成した。 (Modification 1)
In the first to seventh embodiments and their modifications, the processing unit 222 or the first processing unit 822 detects the retinal layer from the tomographic image using the learned model and generates a boundary image. In the ninth to nineteenth embodiments, the segmentation processing unit 2804 generates an area label image corresponding to the input image by using the image segmentation engine including the learned model.
　これに対し、学習済モデルを用いて検出された網膜層の情報や生成された境界画像又は領域ラベル画像は、操作者からの指示に応じて手動で修正されてもよい。例えば、操作者は、表示部５０，２８２０に表示された、網膜層の検出結果や境界画像又は領域ラベル画像の少なくとも一部を指定し、網膜層の位置やラベルを変更することができる。この場合、検出結果の修正や境界画像又は領域ラベル画像の修正は、処理部２２２や第一の処理部８２２、セグメンテーション処理部２８０４によって、操作者の指示に応じて行われてもよいし、これらとは別の修正部等の構成要素によって行われてもよい。このため、処理部２２２、第一の処理部８２２、セグメンテーション処理部２８０４、又は当該修正部は、操作者の指示に応じて、第一の処理部が検出した網膜層の構造を修正する修正部の一例として機能する。なお、当該修正部等は、ＣＰＵやＭＰＵ、ＧＰＵ、ＦＰＧＡ等のプロセッサーによって実行されるソフトウェアモジュール等で構成されてもよいし、ＡＳＩＣ等の特定の機能を果たす回路等によって構成されてもよい。 On the other hand, the information of the retinal layer detected using the learned model and the generated boundary image or region label image may be manually corrected according to an instruction from the operator. For example, the operator can change the position and label of the retinal layer by designating at least a part of the detection result of the retinal layer and the boundary image or the region label image displayed on the display units 50 and 2820. In this case, the correction of the detection result and the correction of the boundary image or the region label image may be performed by the processing unit 222, the first processing unit 822, and the segmentation processing unit 2804 in accordance with the instruction of the operator. It may be performed by a component such as a correction unit different from the above. Therefore, the processing unit 222, the first processing unit 822, the segmentation processing unit 2804, or the correction unit corrects the structure of the retinal layer detected by the first processing unit in accordance with an instruction from the operator. Function as an example. The correction unit or the like may be configured by a software module executed by a processor such as a CPU, an MPU, a GPU, and an FPGA, or may be configured by a circuit that performs a specific function such as an ASIC.
（変形例２）
　変形例１において手動で修正されたデータは、処理部２２２又は第一の処理部８２２が用いる学習済モデル、及びセグメンテーション処理部２８０４が用いる画像セグメンテーションエンジンに含まれる学習済モデルについての追加学習に用いられてもよい。具体的には、処理部２２２又は第一の処理部８２２が用いる学習済モデルについて、入力された断層画像を学習データの入力データとし、操作者からの指示に応じて位置が修正された網膜層（層境界）の情報を出力データ（正解データ）として追加学習を行う。なお、操作者からの指示に応じてラベルが修正された境界画像を出力データとしてもよい。また、画像セグメンテーションエンジンに含まれる学習済モデルについて、入力された画像を学習データの入力データとし、操作者からの指示に応じてラベルの位置が変更された領域ラベル画像を出力データとして追加学習を行う。 (Modification 2)
The data manually modified in the first modification is used for additional learning on the learned model used by the processing unit 222 or the first processing unit 822 and the learned model included in the image segmentation engine used by the segmentation processing unit 2804. You may be. Specifically, for the learned model used by the processing unit 222 or the first processing unit 822, the input tomographic image is used as input data of learning data, and the position of the retinal layer is corrected according to an instruction from the operator. Additional learning is performed using the information of (layer boundary) as output data (correct answer data). Note that a boundary image whose label has been corrected in accordance with an instruction from the operator may be used as output data. In addition, for the trained model included in the image segmentation engine, additional learning is performed using the input image as input data of learning data and the area label image whose label position has been changed according to the instruction from the operator as output data. Do.
　学習済モデルに対しこのような追加学習を行うことで、学習済モデルを用いた検出処理やセグメンテーション処理の精度を向上させられることが期待できる。また、このような処理を行うことで、学習データに関するラベル付け処理（アノテーション処理）を容易に行うことができ、より精度の高い学習データを容易に作成することができる。 By performing such additional learning on the trained model, it can be expected that the accuracy of detection processing and segmentation processing using the trained model can be improved. Further, by performing such processing, labeling processing (annotation processing) for learning data can be easily performed, and learning data with higher accuracy can be easily created.
（変形例３）
　変形例２で説明した追加学習は、操作者の指示に応じて行われてもよい。例えば、表示制御部２５又は出力部２８０７は、変形例１に係る操作者の指示に応じた修正が行われた場合に、修正された網膜層の検出結果や領域ラベル画像等を学習データとして用いるか否かを表示部５０，２８２０に表示させることができる。操作者は、表示部５０，２８２０に表示された選択肢を選択することで、追加学習の要否を指示することができる。これにより、画像処理装置２０，８０，２８００，４４００は、操作者の指示に応じて、追加学習の要否を決定することができる。 (Modification 3)
The additional learning described in Modification 2 may be performed according to an instruction from the operator. For example, the display control unit 25 or the output unit 2807 uses the corrected detection result of the retinal layer, the region label image, and the like as the learning data when the correction according to the instruction of the operator according to the first modification is performed. The presence or absence can be displayed on the display units 50 and 2820. The operator can instruct whether additional learning is necessary by selecting an option displayed on the display units 50 and 2820. Accordingly, the image processing apparatuses 20, 80, 2800, and 4400 can determine whether additional learning is necessary or not according to an instruction from the operator.
　なお、上述のように、学習済モデルはサーバ等の装置に設けられることもできる。この場合には、画像処理装置２０，８０，２８００，４４００は、追加学習を行うとする操作者の指示に応じて、入力された画像と上述の修正が行われた検出結果又は領域ラベル画像等を学習データのペアとして、当該サーバ等に送信・保存することができる。言い換えると、画像処理装置２０，８０，２８００，４４００は、操作者の指示に応じて、学習済モデルを備えるサーバ等の装置に追加学習の学習データを送信するか否かを決定することができる。 Note that, as described above, the learned model can be provided in a device such as a server. In this case, the image processing apparatuses 20, 80, 2800, and 4400 respond to the instruction of the operator to perform the additional learning by using the input image and the detection result or the region label image or the like in which the above correction has been performed. Can be transmitted and stored in the server or the like as a pair of learning data. In other words, the image processing apparatuses 20, 80, 2800, and 4400 can determine whether to transmit learning data for additional learning to a device such as a server including a learned model in accordance with an instruction from the operator. .
（変形例４）
　上述した様々な実施例及び変形例では、静止画について、網膜層の検出処理や領域ラベル画像等の生成処理を行う構成について説明した。これに対し、動画像について、上記実施例及び変形例に係る網膜層の検出処理や領域ラベル画像等の生成処理を繰り返し実行してもよい。一般に眼科装置では、本撮影を行う前に、装置の位置合わせ等のためプレビュー画像（動画像）を生成し、表示することが行われている。そのため、例えば、当該プレビュー画像である断層画像の動画像の少なくとも１つのフレーム毎に、上記実施例及び変形例に係る網膜層の検出処理や領域ラベル画像等の生成処理を繰り返し実行してもよい。 (Modification 4)
In the various embodiments and the modified examples described above, the configuration in which the processing of detecting the retinal layer and the processing of generating the region label image and the like are described for the still image. On the other hand, the process of detecting the retinal layer and the process of generating the region label image and the like according to the above-described embodiment and the modification may be repeatedly performed on the moving image. In general, an ophthalmologic apparatus generates and displays a preview image (moving image) for positioning of the apparatus or the like before performing actual imaging. Therefore, for example, for at least one frame of the moving image of the tomographic image as the preview image, the processing of detecting the retinal layer and the processing of generating the region label image and the like according to the embodiment and the modification may be repeatedly executed. .
　この場合には、表示制御部２５又は出力部２８０７は、プレビュー画像について検出された網膜層や領域ラベル画像等を表示部５０，２８２０に表示させることができる。また、画像処理装置２０，８０，２８００，４４００は、プレビュー画像について検出された網膜層や、網膜層のラベル付けがされた領域が、断層画像の表示領域における所定の位置に位置するようにＯＣＴ装置を制御することができる。より具体的には、画像処理装置２０，８０，２８００，４４００は、プレビュー画像について検出された網膜層や網膜層のラベル付けがされた領域が、断層画像の表示領域における所定の位置に位置するようにコヒーレンスゲート位置を変更する。なお、コヒーレンスゲート位置の調整は、例えば、駆動制御部２３によってコヒーレンスゲートステージ１４を駆動させる等により行われてよい。 In this case, the display control unit 25 or the output unit 2807 can cause the display units 50 and 2820 to display the retinal layer, the region label image, and the like detected for the preview image. Further, the image processing apparatuses 20, 80, 2800, and 4400 operate the OCT so that the retinal layer detected in the preview image and the labeled area of the retinal layer are located at predetermined positions in the display area of the tomographic image. The device can be controlled. More specifically, in the image processing apparatuses 20, 80, 2800, and 4400, the retinal layer detected in the preview image and the labeled area of the retinal layer are located at predetermined positions in the display area of the tomographic image. The coherence gate position as follows. The adjustment of the coherence gate position may be performed, for example, by driving the coherence gate stage 14 by the drive control unit 23 or the like.
　なお、当該コヒーレンスゲート位置の調整は、操作者による指示に応じて手動で行われてもよい。この場合、操作者は、表示部５０，２８２０に表示された、プレビュー画像について検出された網膜層や領域ラベル画像に基づいて、コヒーレンスゲート位置の調整量を画像処理装置２０，８０，２８００，４４００に入力することができる。 The adjustment of the coherence gate position may be manually performed according to an instruction from the operator. In this case, the operator adjusts the adjustment amount of the coherence gate position based on the retinal layer or the region label image detected for the preview image displayed on the display unit 50, 2820 by the image processing device 20, 80, 2800, 4400. Can be entered.
　このような処理によれば、学習済モデルを用いて検出された網膜層又は生成された領域ラベル画像に基づいて、被検眼に対するＯＣＴ装置の位置合わせ（アライメント）を、適切に行うことができる。 According to such processing, it is possible to appropriately perform alignment (alignment) of the OCT apparatus with respect to the eye to be inspected based on the retinal layer detected using the learned model or the generated region label image.
　なお、上記実施例及び変形例に係る網膜層の検出処理や領域ラベル画像等の生成処理を適用可能な動画像は、ライブ動画像に限らず、例えば、記憶部に記憶（保存）された動画像であってもよい。また、コヒーレンスゲート位置等の各種の調整中では、被検眼の網膜等の撮影対象がまだ上手く撮像できていない可能性がある。このため、学習済モデルに入力される医用画像と学習データとして用いられた医用画像との違いが大きいために、精度良く網膜層の検出結果や領域ラベル画像等が得られない可能性がある。そこで、断層画像（Ｂスキャン）の画質評価等の評価値が閾値を超えたら、学習済モデルを用いた網膜層の検出処理や領域ラベル画像等の生成処理を自動的に開始するように構成してもよい。また、断層画像（Ｂスキャン）の画質評価等の評価値が閾値を超えたら、学習済モデルを用いた画像セグメンテーション処理を指示するボタンを検者が指定可能な状態（アクティブ状態）に変更するように構成されてもよい。 The moving image to which the detection processing of the retinal layer and the generation processing of the region label image and the like according to the above-described embodiment and the modified example are applicable is not limited to a live moving image, and may be, for example, a moving image stored (saved) in a storage unit. It may be an image. Also, during various adjustments of the coherence gate position and the like, there is a possibility that the imaging target such as the retina of the eye to be inspected has not been imaged well yet. For this reason, since there is a large difference between the medical image input to the learned model and the medical image used as the learning data, there is a possibility that the detection result of the retinal layer, the region label image and the like cannot be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B-scan) exceeds the threshold value, the detection process of the retinal layer using the learned model and the generation process of the region label image are automatically started. You may. When the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the button for instructing the image segmentation process using the learned model is changed to a state (active state) that can be specified by the examiner. May be configured.
（変形例５）
　疾病眼では、疾病の種類に応じて画像特徴が異なる。そのため、上述した様々な実施例や変形例において用いられる学習済モデルは、疾病の種類毎又は異常部位毎にそれぞれ生成・用意されてもよい。この場合には、例えば、画像処理装置２０，８０，２８００，４４００は、操作者からの被検眼の疾病の種類や異常部位等の入力（指示）に応じて、処理に用いる学習済モデルを選択することができる。なお、疾病の種類や異常部位毎に用意される学習済モデルは、網膜層の検出や領域ラベル画像等の生成に用いられる学習済モデルに限られず、例えば、画像の評価用のエンジンや解析用のエンジン等で用いられる学習済モデルであってもよい。 (Modification 5)
A diseased eye has different image features depending on the type of disease. Therefore, the learned models used in the above-described various embodiments and modifications may be generated and prepared for each type of disease or each abnormal part. In this case, for example, the image processing apparatuses 20, 80, 2800, and 4400 select a learned model to be used for processing according to an input (instruction) of a disease type, an abnormal part, or the like of the eye to be inspected from the operator. can do. The trained model prepared for each type of disease or abnormal region is not limited to a trained model used for detecting a retinal layer or generating an area label image, and may be, for example, an image evaluation engine or an analysis engine. It may be a learned model used in an engine or the like.
　また、画像処理装置２０，８０，２８００，４４００は、別に用意された学習済モデルを用いて、画像から被検眼の疾病の種類や異常部位を識別してもよい。この場合には、画像処理装置２０，８０，２８００，４４００は、当該別に用意された学習済モデルを用いて識別された疾病の種類や異常部位に基づいて、上記処理に用いる学習済モデルを自動的に選択することができる。なお、当該被検眼の疾病の種類や異常部位を識別するための学習済モデルは、断層画像や眼底画像等を入力データとし、疾病の種類やこれら画像における異常部位を出力データとした学習データのペアを用いて学習を行ってよい。ここで、学習データの入力データとしては、断層画像や眼底画像等を単独で入力データとしてもよいし、これらの組み合わせを入力データとしてもよい。 The image processing apparatuses 20, 80, 2800, and 4400 may identify a disease type and an abnormal part of the subject's eye from the image using a separately prepared learned model. In this case, the image processing apparatuses 20, 80, 2800, and 4400 automatically generate the learned model used in the above processing based on the type of the disease and the abnormal part identified using the separately prepared learned model. Can be selected. The trained model for identifying the type of disease or abnormal part of the eye to be examined is a learning model in which a tomographic image or a fundus image is used as input data, and the type of disease or abnormal part in these images is used as output data. Learning may be performed using pairs. Here, as input data of the learning data, a tomographic image, a fundus image, or the like may be used alone as input data, or a combination thereof may be used as input data.
　また、異常部位を検出する場合には、敵対性生成ネットワーク（ＧＡＮ：Ｇｅｎｅｒａｔｉｖｅ　Ａｄｖｅｒｓａｒｉａｌ　Ｎｅｔｗｏｋｓ）や変分オートエンコーダ―（ＶＡＥ：Ｖａｒｉａｔｉｏｎａｌ　ａｕｔｏ－ｅｎｃｏｄｅｒ）を用いてもよい。例えば、断層画像の生成を学習して得た生成器と、生成器が生成した新たな断層画像と本物の眼底正面画像との識別を学習して得た識別器とからなるＤＣＧＡＮ（Ｄｅｅｐ　Ｃｏｎｖｏｌｕｔｉｏｎａｌ　ＧＡＮ）を機械学習モデルとして用いることができる。 In addition, when an abnormal site is detected, a hostility generation network (GAN: General Adversary Network) or a variational auto-encoder (VAE: Variational auto-encoder) may be used. For example, a DCGAN (Deep \ Convolutional \ GAN) including a generator obtained by learning generation of a tomographic image and a classifier obtained by learning identification of a new tomographic image generated by the generator and a real frontal fundus image. ) Can be used as a machine learning model.
　ＤＣＧＡＮを用いる場合には、例えば、識別器が入力された断層画像をエンコードすることで潜在変数にし、生成器が潜在変数に基づいて新たな断層画像を生成する。その後、入力された断層画像と生成された新たな断層画像との差分を異常部位として抽出することができる。また、ＶＡＥを用いる場合には、例えば、入力された断層画像をエンコーダーによりエンコードすることで潜在変数にし、潜在変数をデコーダーによりデコードすることで新たな断層画像を生成する。その後、入力された断層画像と生成された新たな断層画像像との差分を異常部位として抽出することができる。なお、入力データの例として断層画像を例として説明したが、眼底画像や前眼の正面画像等を用いてもよい。 In the case of using DCGAN, for example, the discriminator encodes the input tomographic image into a latent variable by encoding, and the generator generates a new tomographic image based on the latent variable. Thereafter, a difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part. When VAE is used, for example, an input tomographic image is encoded by an encoder to be a latent variable, and the latent variable is decoded by a decoder to generate a new tomographic image. Thereafter, a difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part. Although a tomographic image has been described as an example of the input data, a fundus image, a front image of the anterior eye, and the like may be used.
（変形例６）
　上述した様々な実施例及び変形例において、処理部２２２や、第一の処理部８２２、セグメンテーション処理部２８０４により学習済モデルを用いて、被検眼の領域を検出する場合には、検出した領域毎に所定の画像処理を施すこともできる。例えば、硝子体領域、網膜領域、及び脈絡膜領域のうちの少なくとも２つの領域を検出する場合を考える。この場合には、検出された少なくとも２つの領域に対してコントラスト調整等の画像処理を施す際に、それぞれ異なる画像処理のパラメータを用いることで、各領域に適した調整を行うことができる。各領域に適した調整が行われた画像を表示することで、操作者は領域毎の疾病等をより適切に診断することができる。なお、検出された領域毎に異なる画像処理のパラメータを用いる構成については、例えば、学習済モデルを用いずに被検眼の領域を検出する第二の処理部８２３によって検出された被検眼の領域について同様に適用されてもよい。 (Modification 6)
In the various embodiments and modifications described above, when the processing unit 222, the first processing unit 822, and the segmentation processing unit 2804 use the learned model to detect the region of the subject's eye, May be subjected to predetermined image processing. For example, consider a case where at least two of the vitreous, retinal, and choroidal regions are detected. In this case, when image processing such as contrast adjustment is performed on at least two detected areas, adjustments suitable for each area can be performed by using different image processing parameters. By displaying an image adjusted appropriately for each region, the operator can more appropriately diagnose a disease or the like for each region. Note that, for a configuration using different image processing parameters for each of the detected regions, for example, regarding the region of the subject's eye detected by the second processing unit 823 that detects the region of the subject's eye without using the learned model The same may be applied.
（変形例７）
　上述した様々な実施例及び変形例における表示制御部２５又は出力部２８０７は、表示画面のレポート画面において、所望の層の層厚や各種の血管密度等の解析結果を表示させてもよい。また、視神経乳頭部、黄斑部、血管領域、神経線維束、硝子体領域、黄斑領域、脈絡膜領域、強膜領域、篩状板領域、網膜層境界、網膜層境界端部、視細胞、血球、血管壁、血管内壁境界、血管外側境界、神経節細胞、角膜領域、隅角領域、シュレム管等の少なくとも１つを含む注目部位に関するパラメータの値（分布）を解析結果として表示させてもよい。このとき、例えば、各種のアーチファクトの低減処理が適用された医用画像を解析することで、精度の良い解析結果を表示させることができる。なお、アーチファクトは、例えば、血管領域等による光吸収により生じる偽像領域や、プロジェクションアーチファクト、被検眼の状態（動きや瞬き等）によって測定光の主走査方向に生じる正面画像における帯状のアーチファクト等であってもよい。また、アーチファクトは、例えば、被検者の所定部位の医用画像上に撮影毎にランダムに生じるような写損領域であれば、何でもよい。また、表示制御部２５又は出力部２８０７は、上述したような様々なアーチファクト（写損領域）の少なくとも１つを含む領域に関するパラメータの値（分布）を解析結果として表示部５０，２８２０に表示させてもよい。また、ドルーゼン、新生血管、白斑（硬性白斑）、及びシュードドルーゼン等の異常部位等の少なくとも１つを含む領域に関するパラメータの値（分布）を解析結果として表示させてもよい。 (Modification 7)
The display control unit 25 or the output unit 2807 in the various embodiments and modifications described above may display analysis results such as a desired layer thickness and various blood vessel densities on the report screen of the display screen. In addition, the optic papilla, macula, blood vessel region, nerve fiber bundle, vitreous region, macula region, choroid region, sclera region, cribriform region, retinal layer boundary, retinal layer boundary edge, photoreceptor cells, blood cells, The value (distribution) of a parameter relating to a site of interest including at least one of a blood vessel wall, a blood vessel inner wall boundary, a blood vessel outer boundary, a ganglion cell, a corneal region, a corner region, and Schlemm's canal may be displayed as an analysis result. At this time, for example, by analyzing a medical image to which various types of artifact reduction processing are applied, a highly accurate analysis result can be displayed. Note that the artifact is, for example, a false image region caused by light absorption by a blood vessel region or the like, a projection artifact, a band-like artifact in a front image generated in a main scanning direction of measurement light due to a state (movement, blink, etc.) of an eye to be inspected, or the like. There may be. In addition, the artifact may be anything as long as it is an image failure area that randomly appears on a medical image of a predetermined part of the subject at each imaging, for example. In addition, the display control unit 25 or the output unit 2807 causes the display units 50 and 2820 to display parameter values (distribution) regarding an area including at least one of the various artifacts (defect areas) as an analysis result. You may. Further, the values (distributions) of parameters relating to a region including at least one of abnormal sites such as drusen, new blood vessels, vitiligo (hard vitiligo), and pseudo drusen may be displayed as analysis results.
　また、解析結果は、解析マップや、各分割領域に対応する統計値を示すセクター等で表示されてもよい。なお、解析結果は、医用画像の解析結果を学習データとして学習して得た学習済モデル（解析結果生成エンジン、解析結果生成用の学習済モデル）を用いて生成されたものであってもよい。このとき、学習済モデルは、医用画像とその医用画像の解析結果とを含む学習データや、医用画像とその医用画像とは異なる種類の医用画像の解析結果とを含む学習データ等を用いた学習により得たものであってもよい。 The analysis result may be displayed as an analysis map, a sector indicating a statistical value corresponding to each divided region, or the like. The analysis result may be generated using a learned model (analysis result generation engine, a learned model for generating the analysis result) obtained by learning the analysis result of the medical image as learning data. . At this time, the trained model is a learning model using learning data including a medical image and an analysis result of the medical image, learning data including a medical image and an analysis result of a medical image of a type different from the medical image, and the like. May be obtained.
　また、学習データは、処理部２２２、又は第一の処理部８２２及び／若しくは第二の処理部８２３による網膜層の検出結果や、セグメンテーション処理部２８０４で生成された領域ラベル画像と、それらを用いた医用画像の解析結果とを含んだものでもよい。この場合、画像処理装置は、例えば、解析結果生成用の学習済モデルを用いて、第一の検出処理を実行して得た結果から、断層画像の解析結果を生成する、解析結果生成部の一例として機能することができる。 Further, the learning data includes the detection result of the retinal layer by the processing unit 222 or the first processing unit 822 and / or the second processing unit 823, the region label image generated by the segmentation processing unit 2804, and the learning data. It may include the analysis result of the medical image. In this case, for example, the image processing apparatus generates an analysis result of the tomographic image from a result obtained by executing the first detection process using the learned model for generating the analysis result. It can serve as an example.
　さらに、学習済モデルは、輝度正面画像及びモーションコントラスト正面画像のように、所定部位の異なる種類の複数の医用画像をセットとする入力データを含む学習データを用いた学習により得たものであってもよい。ここで、輝度正面画像は輝度のＥｎ－Ｆａｃｅ画像に対応し、モーションコントラスト正面画像はＯＣＴＡのＥｎ－Ｆａｃｅ画像に対応する。 Further, the learned model is obtained by learning using learning data including input data in which a plurality of medical images of different types of a predetermined part are set, such as a luminance front image and a motion contrast front image. Is also good. Here, the luminance front image corresponds to the luminance En-Face image, and the motion contrast front image corresponds to the OCTA En-Face image.
　また、高画質化用の学習済モデルを用いて生成された高画質画像を用いて得た解析結果が表示されるように構成されてもよい。この場合、学習データに含まれる入力データとしては、高画質化用の学習済モデルを用いて生成された高画質画像であってもよいし、低画質画像と高画質画像とのセットであってもよい。なお、学習データは、学習済モデルを用いて高画質化された画像について、手動又は自動で少なくとも一部に修正が施された画像であってもよい。 解析 Also, an analysis result obtained using a high-quality image generated using a learned model for improving image quality may be displayed. In this case, the input data included in the learning data may be a high-quality image generated using a trained model for high image quality, or a set of a low-quality image and a high-quality image. Is also good. Note that the learning data may be an image obtained by manually or automatically correcting at least a part of an image whose image quality has been improved using the learned model.
　また、学習データは、例えば、解析領域を解析して得た解析値（例えば、平均値や中央値等）、解析値を含む表、解析マップ、画像におけるセクター等の解析領域の位置等の少なくとも１つを含む情報を（教師あり学習の）正解データとして、入力データにラベル付けしたデータであってもよい。なお、操作者からの指示に応じて、解析結果生成用の学習済モデルを用いて得た解析結果が表示されるように構成されてもよい。 Further, the learning data includes, for example, at least an analysis value (for example, an average value or a median value) obtained by analyzing the analysis area, a table including the analysis value, an analysis map, and the position of the analysis area such as a sector in the image. Information including one may be data obtained by labeling input data as correct answer data (for supervised learning). Note that, in accordance with an instruction from the operator, an analysis result obtained using the learned model for generating the analysis result may be displayed.
　また、上述した実施例及び変形例における表示制御部２５及び出力部２８０７は、表示画面のレポート画面において、緑内障や加齢黄斑変性等の種々の診断結果を表示させてもよい。このとき、例えば、上述したような各種のアーチファクトの低減処理が適用された医用画像を解析することで、精度の良い診断結果を表示させることができる。また、診断結果は、特定された異常部位等の位置を画像上に表示されてもよいし、異常部位の状態等を文字等によって表示されてもよい。さらに、異常部位等の分類結果（例えば、カーティン分類）を診断結果として表示させてもよい。 In addition, the display control unit 25 and the output unit 2807 in the above-described embodiments and modified examples may display various diagnosis results such as glaucoma and age-related macular degeneration on the report screen of the display screen. At this time, for example, by analyzing the medical image to which the various artifact reduction processes described above are applied, a highly accurate diagnosis result can be displayed. In the diagnosis result, the position of the specified abnormal part or the like may be displayed on the image, or the state or the like of the abnormal part may be displayed by characters or the like. Furthermore, a classification result (for example, a Curtin classification) of an abnormal part or the like may be displayed as a diagnosis result.
　なお、診断結果は、医用画像の診断結果を学習データとして学習して得た学習済モデル（診断結果生成エンジン、診断結果生成用の学習済モデル）を用いて生成されたものであってもよい。また、学習済モデルは、医用画像とその医用画像の診断結果とを含む学習データや、医用画像とその医用画像とは異なる種類の医用画像の診断結果とを含む学習データ等を用いた学習により得たものであってもよい。 Note that the diagnosis result may be generated using a learned model (diagnosis result generation engine, a learned model for generating a diagnosis result) obtained by learning a diagnosis result of a medical image as learning data. . The learned model is obtained by learning using learning data including a medical image and a diagnosis result of the medical image, and learning data including a medical image and a diagnosis result of a medical image of a different type from the medical image. It may be obtained.
　また、学習データは、処理部２２２、又は第一の処理部８２２及び／若しくは第二の処理部８２３による網膜層の検出結果や、セグメンテーション処理部２８０４で生成された領域ラベル画像と、それらを用いた医用画像の診断結果とを含んだものでもよい。この場合、画像処理装置は、例えば、診断結果生成用の学習済モデルを用いて、第一の検出処理を実行して得た結果から、断層画像の診断結果を生成する、診断結果生成部の一例として機能することができる。 Further, the learning data includes the detection result of the retinal layer by the processing unit 222 or the first processing unit 822 and / or the second processing unit 823, the region label image generated by the segmentation processing unit 2804, and the learning data. And a diagnosis result of the medical image. In this case, for example, the image processing apparatus generates a diagnosis result of a tomographic image from a result obtained by executing the first detection processing using a learned model for generating a diagnosis result. It can serve as an example.
　さらに、高画質化用の学習済モデルを用いて生成された高画質画像を用いて得た診断結果が表示されるように構成されてもよい。この場合、学習データに含まれる入力データとしては、高画質化用の学習済モデルを用いて生成された高画質画像であってもよいし、低画質画像と高画質画像とのセットであってもよい。なお、学習データは、学習済モデルを用いて高画質化された画像について、手動又は自動で少なくとも一部に修正が施された画像であってもよい。 Furthermore, a diagnosis result obtained by using a high-quality image generated by using a learned model for improving image quality may be displayed. In this case, the input data included in the learning data may be a high-quality image generated using a learned model for improving the image quality, or a set of a low-quality image and a high-quality image. Is also good. Note that the learning data may be an image obtained by manually or automatically correcting at least a part of an image whose image quality has been improved using the learned model.
　また、学習データは、例えば、診断名、病変（異常部位）の種類や状態（程度）、画像における病変の位置、注目領域に対する病変の位置、所見（読影所見等）、診断名の根拠（肯定的な医用支援情報等）、診断名を否定する根拠（否定的な医用支援情報）等の少なくとも１つを含む情報を（教師あり学習の）正解データとして、入力データにラベル付けしたデータであってもよい。なお、検者からの指示に応じて、診断結果生成用の学習済モデルを用いて得た診断結果が表示されるように構成されてもよい。 The learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal part), the position of the lesion in the image, the position of the lesion with respect to the attention area, the findings (interpretation findings, etc.), Data that includes at least one of the following: information including at least one of grounds for negating the diagnosis name (negative medical support information) and the like (supervised learning). You may. In addition, according to the instruction from the examiner, a configuration may be adopted in which a diagnosis result obtained using the learned model for generating a diagnosis result is displayed.
　また、上述した様々な実施例及び変形例に係る表示制御部２５及び出力部２８０７は、表示画面のレポート画面において、上述したような注目部位、アーチファクト、及び異常部位等の物体認識結果（物体検出結果）やセグメンテーション結果を表示させてもよい。このとき、例えば、画像上の物体の周辺に矩形の枠等を重畳して表示させてもよい。また、例えば、画像における物体上に色等を重畳して表示させてもよい。なお、物体認識結果やセグメンテーション結果は、物体認識やセグメンテーションを示す情報を正解データとして医用画像にラベル付けした学習データを学習して得た学習済モデルを用いて生成されたものであってもよい。なお、上述した解析結果生成や診断結果生成は、上述した物体認識結果やセグメンテーション結果を利用することで得られたものであってもよい。例えば、物体認識やセグメンテーションの処理により得た注目部位に対して解析結果生成や診断結果生成の処理を行ってもよい。 In addition, the display control unit 25 and the output unit 2807 according to the various embodiments and the modifications described above, on the report screen of the display screen, recognize the object (for example, the object detection result such as the noted part, the artifact, and the abnormal part) as described above (object detection). Results) and the segmentation results may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, a color or the like may be superimposed and displayed on an object in an image. The object recognition result and the segmentation result may be generated using a learned model obtained by learning learning data obtained by labeling a medical image with information indicating the object recognition and the segmentation as correct data. . Note that the above-described generation of the analysis result and the generation of the diagnosis result may be obtained by using the above-described object recognition result and the segmentation result. For example, a process of generating an analysis result and a process of generating a diagnosis result may be performed on a region of interest obtained by the processing of object recognition and segmentation.
　また、特に診断結果生成用の学習済モデルは、被検者の所定部位の異なる種類の複数の医用画像をセットとする入力データを含む学習データにより学習して得た学習済モデルであってもよい。このとき、学習データに含まれる入力データとして、例えば、眼底のモーションコントラスト正面画像及び輝度正面画像（あるいは輝度断層画像）をセットとする入力データが考えられる。また、学習データに含まれる入力データとして、例えば、眼底の断層画像（Ｂスキャン画像）及びカラー眼底画像（あるいは蛍光眼底画像）をセットとする入力データ等も考えられる。また、異なる種類の複数の医療画像は、異なるモダリティ、異なる光学系、又は異なる原理等により取得されたものであれば何でもよい。 Further, in particular, the learned model for generating a diagnosis result may be a learned model obtained by learning using learning data including input data in which a plurality of different types of medical images of a predetermined part of the subject are set. Good. At this time, as input data included in the learning data, for example, input data in which a front image of a motion contrast of a fundus and a luminance front image (or a luminance tomographic image) are set can be considered. Further, as input data included in the learning data, for example, input data in which a tomographic image of a fundus (B-scan image) and a color fundus image (or a fluorescent fundus image) are set may be considered. Further, the plurality of different types of medical images may be any medical images obtained by different modalities, different optical systems, different principles, or the like.
　また、特に診断結果生成用の学習済モデルは、被検者の異なる部位の複数の医用画像をセットとする入力データを含む学習データにより学習して得た学習済モデルであってもよい。このとき、学習データに含まれる入力データとして、例えば、眼底の断層画像（Ｂスキャン画像）と前眼部の断層画像（Ｂスキャン画像）とをセットとする入力データが考えられる。また、学習データに含まれる入力データとして、例えば、眼底の黄斑の三次元ＯＣＴ画像（三次元断層画像）と眼底の視神経乳頭のサークルスキャン（又はラスタスキャン）断層画像とをセットとする入力データ等も考えられる。 In addition, in particular, the learned model for generating a diagnosis result may be a learned model obtained by learning using learning data including input data in which a plurality of medical images of different parts of the subject are set. At this time, as input data included in the learning data, for example, input data in which a tomographic image of the fundus (B-scan image) and a tomographic image of the anterior segment (B-scan image) are set can be considered. Further, as input data included in the learning data, for example, input data in which a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic papilla of the fundus are set. Is also conceivable.
　なお、学習データに含まれる入力データは、被検者の異なる部位及び異なる種類の複数の医用画像であってもよい。このとき、学習データに含まれる入力データは、例えば、前眼部の断層画像とカラー眼底画像とをセットとする入力データ等が考えられる。また、上述した学習済モデルは、被検者の所定部位の異なる撮影画角の複数の医用画像をセットとする入力データを含む学習データにより学習して得た学習済モデルであってもよい。また、学習データに含まれる入力データは、パノラマ画像のように、所定部位を複数領域に時分割して得た複数の医用画像を貼り合わせたものであってもよい。このとき、パノラマ画像のような広画角画像を学習データとして用いることにより、狭画角画像よりも情報量が多い等の理由から画像の特徴量を精度良く取得できる可能性があるため、処理の結果を向上することができる。また、学習データに含まれる入力データは、被検者の所定部位の異なる日時の複数の医用画像をセットとする入力データであってもよい。 The input data included in the learning data may be different parts of the subject and a plurality of different types of medical images. At this time, the input data included in the learning data may be, for example, input data that sets a tomographic image of the anterior ocular segment and a color fundus image. Further, the above-described learned model may be a learned model obtained by learning using learning data including input data in which a plurality of medical images of a predetermined part of the subject with different imaging angles of view are set. The input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a predetermined part into a plurality of regions, such as a panoramic image. At this time, by using a wide-angle image such as a panoramic image as learning data, there is a possibility that the feature amount of the image can be acquired with high accuracy because the amount of information is larger than that of the narrow-angle image. Can improve the result. The input data included in the learning data may be input data in which a plurality of medical images at different dates and times of a predetermined part of the subject are set.
　また、上述した解析結果と診断結果と物体認識結果とセグメンテーション結果とのうち少なくとも１つの結果が表示される表示画面は、レポート画面に限らない。このような表示画面は、例えば、撮影確認画面、経過観察用の表示画面、及び撮影前の各種調整用のプレビュー画面（各種のライブ動画像が表示される表示画面）等の少なくとも１つの表示画面に表示されてもよい。例えば、上述した学習済モデルを用いて得た上記少なくとも１つの結果を撮影確認画面に表示させることにより、操作者は、撮影直後であっても精度の良い結果を確認することができる。また、実施例７等で説明した低画質画像と高画質画像との表示の変更は、例えば、低画質画像の解析結果と高画質画像の解析結果との表示の変更であってもよい。 The display screen on which at least one of the analysis result, the diagnosis result, the object recognition result, and the segmentation result is displayed is not limited to the report screen. Such a display screen includes, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). May be displayed. For example, by displaying the at least one result obtained using the above-described learned model on a shooting confirmation screen, the operator can check the result with high accuracy even immediately after shooting. The change in the display of the low-quality image and the high-quality image described in the seventh embodiment and the like may be, for example, the change in the display of the analysis result of the low-quality image and the analysis result of the high-quality image.
　ここで、上述した様々な学習済モデルは、学習データを用いた機械学習により得ることができる。機械学習には、例えば、多階層のニューラルネットワークから成る深層学習（Ｄｅｅｐ　Ｌｅａｒｎｉｎｇ）がある。また、多階層のニューラルネットワークの少なくとも一部には、例えば、畳み込みニューラルネットワーク（ＣＮＮ：Ｃｏｎｖｏｌｕｔｉｏｎａｌ　Ｎｅｕｒａｌ　Ｎｅｔｗｏｒｋ）を用いることができる。また、多階層のニューラルネットワークの少なくとも一部には、オートエンコーダ（自己符号化器）に関する技術が用いられてもよい。また、学習には、バックプロパゲーション（誤差逆伝搬法）に関する技術が用いられてもよい。ただし、機械学習としては、深層学習に限らず、画像等の学習データの特徴量を学習によって自ら抽出（表現）可能なモデルを用いた学習であれば何でもよい。機械学習モデルは、例えば、カプセルネットワーク（Ｃａｐｓｕｌｅ　Ｎｅｔｗｏｒｋ；ＣａｐｓＮｅｔ）でもよい。ここで、一般的なニューラルネットワークでは、各ユニット（各ニューロン）はスカラー値を出力するように構成されることによって、例えば、画像における特徴間の空間的な位置関係（相対位置）に関する空間情報が低減されるように構成されている。これにより、例えば、画像の局所的な歪みや平行移動等の影響が低減されるような学習を行うことができる。一方、カプセルネットワークでは、各ユニット（各カプセル）は空間情報をベクトルとして出力するように構成されることよって、例えば、空間情報が保持されるように構成されている。これにより、例えば、画像における特徴間の空間的な位置関係が考慮されたような学習を行うことができる。 Here, the various learned models described above can be obtained by machine learning using learning data. The machine learning includes, for example, deep learning (Deep @ Learning) including a multi-layer neural network. For at least a part of the multi-layer neural network, for example, a convolutional neural network (CNN: Convolutional Neural Network) can be used. In addition, at least a part of the multi-layer neural network may use a technology related to an auto encoder (self-encoder). Further, a technology related to back propagation (error back propagation method) may be used for learning. However, the machine learning is not limited to the deep learning, but may be any learning using a model capable of extracting (representing) a feature amount of learning data such as an image by learning. The machine learning model may be, for example, a capsule network (Capsule @ Network; CapsNet). Here, in a general neural network, since each unit (each neuron) is configured to output a scalar value, for example, spatial information about a spatial positional relationship (relative position) between features in an image is obtained. It is configured to be reduced. Thereby, for example, it is possible to perform learning such that the influence of local distortion or parallel movement of the image is reduced. On the other hand, in the capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is held. Thus, for example, learning can be performed in which a spatial positional relationship between features in an image is considered.
　また、高画質化エンジン（高画質化用の学習済モデル）は、高画質化エンジンにより生成された少なくとも１つの高画質画像を含む学習データを追加学習して得た学習済モデルであってもよい。このとき、高画質画像を追加学習用の学習データとして用いるか否かを、検者からの指示により選択可能に構成されてもよい。 Further, the high-quality image engine (learned model for high-quality image) may be a learned model obtained by additionally learning learning data including at least one high-quality image generated by the high-quality image engine. Good. At this time, whether or not to use the high-quality image as learning data for additional learning may be configured to be selectable by an instruction from the examiner.
（変形例８）
　上述した様々な実施例及び変形例におけるプレビュー画面において、ライブ動画像の少なくとも１つのフレーム毎に上述した高画質化用の学習済モデルが用いられるように構成されてもよい。このとき、プレビュー画面において、異なる部位や異なる種類の複数のライブ動画像が表示されている場合には、各ライブ動画像に対応する学習済モデルが用いられるように構成されてもよい。これにより、例えば、ライブ動画像であっても、処理時間を短縮することができるため、検者は撮影開始前に精度の高い情報を得ることができる。このため、例えば、再撮影の失敗等を低減することができるため、診断の精度や効率を向上させることができる。なお、複数のライブ動画像は、例えば、ＸＹＺ方向のアライメントのための前眼部の動画像、及び眼底観察光学系のフォーカス調整やＯＣＴフォーカス調整のための眼底の正面動画像であってよい。また、複数のライブ動画像は、例えば、ＯＣＴのコヒーレンスゲート調整（測定光路長と参照光路長との光路長差の調整）のための眼底の断層動画像等であってもよい。 (Modification 8)
The preview screens in the various embodiments and modifications described above may be configured so that the above-described learned model for improving image quality is used for at least one frame of a live moving image. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the learned model corresponding to each live moving image may be used. Thus, for example, even for a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of imaging. For this reason, for example, failure in re-imaging can be reduced, so that the accuracy and efficiency of diagnosis can be improved. Note that the plurality of live moving images may be, for example, a moving image of the anterior segment for alignment in the XYZ directions and a front moving image of the fundus for focus adjustment of the fundus observation optical system and OCT focus adjustment. Further, the plurality of live video images may be, for example, tomographic video images of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length).
　また、上述した学習済モデルを適用可能な動画像は、ライブ動画像に限らず、例えば、記憶部に記憶（保存）された動画像であってもよい。このとき、例えば、記憶部に記憶（保存）された眼底の断層動画像の少なくとも１つのフレーム毎に位置合わせして得た動画像が表示画面に表示されてもよい。例えば、硝子体を好適に観察したい場合には、まず、フレーム上に硝子体ができるだけ存在する等の条件を基準とする基準フレームを選択してもよい。このとき、各フレームは、ＸＺ方向の断層画像（Ｂスキャン像）である。そして、選択された基準フレームに対して他のフレームがＸＺ方向に位置合わせされた動画像が表示画面に表示されてもよい。このとき、例えば、動画像の少なくとも１つのフレーム毎に高画質化用の学習済モデルを用いて順次生成された高画質画像（高画質フレーム）を連続表示させるように構成してもよい。 The moving image to which the learned model can be applied is not limited to a live moving image, and may be, for example, a moving image stored (saved) in a storage unit. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus stored (saved) in the storage unit may be displayed on the display screen. For example, when it is desired to appropriately observe the vitreous body, first, a reference frame may be selected based on the condition that the vitreous body exists on the frame as much as possible. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a configuration may be adopted in which high-quality images (high-quality frames) sequentially generated using a learned model for improving image quality are continuously displayed for at least one frame of a moving image.
　なお、上述したフレーム間の位置合わせの手法としては、Ｘ方向の位置合わせの手法とＺ方向（深度方向）の位置合わせの手法とは、同じ手法が適用されてもよいし、全て異なる手法が適用されてもよい。また、同一方向の位置合わせは、異なる手法で複数回行われてもよく、例えば、粗い位置合わせを行った後に、精密な位置合わせが行われてもよい。また、位置合わせの手法としては、例えば、断層画像（Ｂスキャン像）をセグメンテーション処理して得た網膜層境界を用いた（Ｚ方向の粗い）位置合わせがある。さらに、位置合わせの手法としては、例えば、断層画像を分割して得た複数の領域と基準画像との相関情報（類似度）を用いた（Ｘ方向やＺ方向の精密な）位置合わせもある。またさらに、位置合わせの手法としては、例えば、断層画像（Ｂスキャン像）毎に生成した一次元投影像を用いた（Ｘ方向の）位置合わせ、２次元正面画像を用いた（Ｘ方向の）位置合わせ等がある。また、ピクセル単位で粗く位置合わせが行われてから、サブピクセル単位で精密な位置合わせが行われるように構成されてもよい。 As the above-described method of positioning between frames, the same method may be applied to the method of positioning in the X direction and the method of positioning in the Z direction (depth direction), or all different methods may be used. May be applied. In addition, the alignment in the same direction may be performed a plurality of times by different methods. For example, after the rough alignment is performed, the precise alignment may be performed. Further, as a positioning method, for example, there is positioning (coarse in the Z direction) using a retinal layer boundary obtained by performing a segmentation process on a tomographic image (B-scan image). Further, as an alignment method, for example, there is an alignment (precise in the X and Z directions) using correlation information (similarity) between a plurality of regions obtained by dividing a tomographic image and a reference image. . Further, as a positioning method, for example, positioning (in the X direction) using a one-dimensional projection image generated for each tomographic image (B scan image), and using a two-dimensional front image (in the X direction) There is alignment, etc. In addition, the configuration may be such that, after coarse positioning is performed in pixel units, precise positioning is performed in subpixel units.
　ここで、各種の調整中では、被検眼の網膜等の撮影対象がまだ上手く撮像できていない可能性がある。このため、学習済モデルに入力される医用画像と学習データとして用いられた医用画像との違いが大きいために、精度良く高画質画像が得られない可能性がある。そこで、断層画像（Ｂスキャン）の画質評価等の評価値が閾値を超えたら、高画質動画像の表示（高画質フレームの連続表示）を自動的に開始するように構成してもよい。また、断層画像（Ｂスキャン）の画質評価等の評価値が閾値を超えたら、高画質化ボタンを検者が指定可能な状態（アクティブ状態）に変更するように構成されてもよい。 Here, during the various adjustments, there is a possibility that the imaging target such as the retina of the eye to be inspected has not been imaged well yet. For this reason, since there is a large difference between the medical image input to the learned model and the medical image used as the learning data, a high-quality image may not be obtained with high accuracy. Therefore, when an evaluation value such as the image quality evaluation of a tomographic image (B scan) exceeds a threshold, display of a high-quality moving image (continuous display of high-quality frames) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the image quality improvement button may be changed to a state in which the examiner can specify (active state).
　また、走査パターン等が異なる撮影モード毎に異なる高画質化用の学習済モデルを用意して、選択された撮影モードに対応する高画質化用の学習済モデルが選択されるように構成されてもよい。また、異なる撮影モードで得た様々な医用画像を含む学習データを学習して得た１つの高画質化用の学習済モデルが用いられてもよい。 In addition, a different learned model for improving image quality is prepared for each shooting mode having a different scanning pattern or the like, and a learned model for improving image quality corresponding to the selected shooting mode is selected. Is also good. Further, one learned model for improving image quality obtained by learning learning data including various medical images obtained in different imaging modes may be used.
（変形例９）
　上述した様々な実施例及び変形例においては、各種学習済モデルが追加学習中である場合、追加学習中の学習済モデル自体を用いて出力（推論・予測）することが難しい可能性がある。このため、追加学習中の学習済モデルに対する医用画像の入力を禁止することがよい。また、追加学習中の学習済モデルと同じ学習済モデルをもう一つ予備の学習済モデルとして用意してもよい。このとき、追加学習中には、予備の学習済モデルに対して医用画像の入力が実行できるようにすることがよい。そして、追加学習が完了した後に、追加学習後の学習済モデルを評価し、問題がなければ、予備の学習済モデルから追加学習後の学習済モデルに置き換えればよい。また、問題があれば、予備の学習済モデルが用いられるようにしてもよい。 (Modification 9)
In the above-described various embodiments and modified examples, when various learned models are undergoing additional learning, it may be difficult to output (inference / prediction) using the learned model itself undergoing additional learning. For this reason, it is preferable to prohibit the input of a medical image to the learned model during the additional learning. Further, the same learned model as the learned model during the additional learning may be prepared as another spare learned model. At this time, during the additional learning, it is preferable that a medical image can be input to the spare learned model. Then, after the additional learning is completed, the learned model after the additional learning is evaluated, and if there is no problem, the spare learned model may be replaced with the learned model after the additional learning. If there is a problem, a spare learned model may be used.
　また、撮影部位毎に学習して得た学習済モデルを選択的に利用できるようにしてもよい。具体的には、第一の撮影部位（肺、被検眼等）を含む学習データを用いて得た第一の学習済モデルと、第一の撮影部位とは異なる第二の撮影部位を含む学習データを用いて得た第二の学習済モデルと、を含む複数の学習済モデルを用意することができる。そして、制御部２００は、これら複数の学習済モデルのいずれかを選択する選択手段を有してもよい。このとき、制御部２００は、選択された学習済モデルに対して追加学習を実行する制御手段を有してもよい。制御手段は、検者からの指示に応じて、選択された学習済モデルに対応する撮影部位と該撮影部位の撮影画像とがペアとなるデータを検索し、検索して得たデータを学習データとする学習を、選択された学習済モデルに対して追加学習として実行することができる。なお、選択された学習済モデルに対応する撮影部位は、データのヘッダの情報から取得したり、検者により手動入力されたりしたものであってよい。また、データの検索は、例えば、病院や研究所等の外部施設のサーバ等からネットワークを介して行われてよい。これにより、学習済モデルに対応する撮影部位の撮影画像を用いて、撮影部位毎に効率的に追加学習することができる。 学習 Also, a learned model obtained by learning for each imaging region may be selectively used. Specifically, a first learned model obtained using learning data including a first imaging part (lung, eye to be examined, etc.) and a learning including a second imaging part different from the first imaging part A plurality of learned models including the second learned model obtained using the data can be prepared. Then, the control unit 200 may include a selection unit that selects any one of the plurality of learned models. At this time, the control unit 200 may include a control unit that executes additional learning on the selected learned model. The control means searches for data in which an imaging part corresponding to the selected learned model and an imaging image of the imaging part are paired in accordance with an instruction from the examiner, and retrieves the obtained data as learning data. Can be executed as additional learning on the selected learned model. The imaging part corresponding to the selected learned model may be obtained from the information in the header of the data or manually input by the examiner. The data search may be performed via a network from a server or the like of an external facility such as a hospital or a laboratory. Thus, additional learning can be efficiently performed for each imaging region using the imaging image of the imaging region corresponding to the learned model.
　なお、選択手段及び制御手段は、制御部２００のＣＰＵやＭＰＵ等のプロセッサーによって実行されるソフトウェアモジュールにより構成されてよい。また、選択手段及び制御手段は、ＡＳＩＣ等の特定の機能を果たす回路や独立した装置等によって構成されてもよい。 Note that the selection unit and the control unit may be configured by a software module executed by a processor such as a CPU or an MPU of the control unit 200. Further, the selection unit and the control unit may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like.
　また、追加学習用の学習データを、病院や研究所等の外部施設のサーバ等からネットワークを介して取得する際には、改ざんや、追加学習時のシステムトラブル等による信頼性低下を低減することが有用である。そこで、デジタル署名やハッシュ化による一致性の確認を行うことで、追加学習用の学習データの正当性を検出してもよい。これにより、追加学習用の学習データを保護することができる。このとき、デジタル署名やハッシュ化による一致性の確認した結果として、追加学習用の学習データの正当性が検出できなかった場合には、その旨の警告を行い、その学習データによる追加学習を行わないものとする。なお、サーバは、その設置場所を問わず、例えば、クラウドサーバ、フォグサーバ、エッジサーバ等のどのような形態でもよい。 Also, when acquiring learning data for additional learning from a server or the like at an external facility such as a hospital or research institute via a network, it is necessary to reduce the reduction in reliability due to tampering or system trouble during additional learning. Is useful. Therefore, the validity of the learning data for additional learning may be detected by confirming the matching by digital signature or hashing. Thereby, the learning data for additional learning can be protected. At this time, if the validity of the learning data for additional learning cannot be detected as a result of checking the consistency by digital signature or hashing, a warning to that effect is issued, and additional learning using the learning data is performed. Make it not exist. The server may be in any form, such as a cloud server, a fog server, an edge server, etc., regardless of the installation location.
（変形例１０）
　上述した様々な実施例及び変形例において、検者からの指示は、手動による指示（例えば、ユーザーインターフェース等を用いた指示）以外にも、音声等による指示であってもよい。このとき、例えば、機械学習により得た音声認識モデル（音声認識エンジン、音声認識用の学習済モデル）を含む機械学習モデルが用いられてもよい。また、手動による指示は、キーボードやタッチパネル等を用いた文字入力等による指示であってもよい。このとき、例えば、機械学習により得た文字認識モデル（文字認識エンジン、文字認識用の学習済モデル）を含む機械学習モデルが用いられてもよい。また、検者からの指示は、ジェスチャー等による指示であってもよい。このとき、機械学習により得たジェスチャー認識モデル（ジェスチャー認識エンジン、ジェスチャー認識用の学習済モデル）を含む機械学習モデルが用いられてもよい。 (Modification 10)
In the various embodiments and modifications described above, the instruction from the examiner may be an instruction by voice or the like in addition to a manual instruction (for example, an instruction using a user interface or the like). At this time, for example, a machine learning model including a speech recognition model (speech recognition engine, learned model for speech recognition) obtained by machine learning may be used. The manual instruction may be an instruction by character input using a keyboard, a touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (a character recognition engine, a learned model for character recognition) obtained by machine learning may be used. The instruction from the examiner may be an instruction by a gesture or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, learned model for gesture recognition) obtained by machine learning may be used.
　また、検者からの指示は、表示部５０，２８２０上の検者の視線検出結果等であってもよい。視線検出結果は、例えば、表示部５０，２８２０周辺から撮影して得た検者の動画像を用いた瞳孔検出結果であってもよい。このとき、動画像からの瞳孔検出は、上述したような物体認識エンジンを用いてもよい。また、検者からの指示は、脳波、体を流れる微弱な電気信号等による指示であってもよい。 The instruction from the examiner may be the result of detection of the examiner's line of sight on the display units 50 and 2820. The gaze detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from around the display units 50 and 2820. At this time, the pupil detection from the moving image may use the above-described object recognition engine. Further, the instruction from the examiner may be an instruction based on an electroencephalogram, a weak electric signal flowing through the body, or the like.
　このような場合、例えば、学習データとしては、上述したような種々の学習済モデルの処理による結果の表示の指示を示す文字データ又は音声データ（波形データ）等を入力データとし、種々の学習済モデルの処理による結果等を実際に表示部に表示させるための実行命令を正解データとする学習データであってもよい。また、学習データとしては、例えば、高画質化用の学習済モデルで得た高画質画像の表示の指示を示す文字データ又は音声データ等を入力データとし、高画質画像の表示の実行命令及び図２２Ａ及び図２２Ｂに示すようなボタン２２２０をアクティブ状態に変更するための実行命令を正解データとする学習データであってもよい。なお、学習データとしては、例えば、文字データ又は音声データ等が示す指示内容と実行命令内容とが互いに対応するものであれば何でもよい。また、音響モデルや言語モデル等を用いて、音声データから文字データに変換してもよい。また、複数のマイクで得た波形データを用いて、音声データに重畳しているノイズデータを低減する処理を行ってもよい。また、文字又は音声等による指示と、マウス又はタッチパネル等による指示とを、検者からの指示に応じて選択可能に構成されてもよい。また、文字又は音声等による指示のオン・オフを、検者からの指示に応じて選択可能に構成されてもよい。 In such a case, for example, as the learning data, character data or voice data (waveform data) indicating an instruction to display a result of processing of the various learned models as described above is used as input data, and various learned models are used. It may be learning data in which an execution instruction for actually displaying the result of the model processing on the display unit is correct data. As the learning data, for example, character data or audio data indicating an instruction to display a high-quality image obtained by a learned model for high-quality image input is used as input data, and an instruction to execute a high-quality image display The learning data may be the correct answer data as the execution instruction for changing the button 2220 to the active state as shown in FIG. 22A and FIG. 22B. The learning data may be any data as long as the instruction content indicated by the character data or the voice data and the execution instruction content correspond to each other. Moreover, you may convert audio | voice data into character data using an acoustic model or a language model. Further, a process of reducing noise data superimposed on audio data may be performed using waveform data obtained by a plurality of microphones. Further, an instruction using characters or voice and an instruction using a mouse or a touch panel may be configured to be selectable according to an instruction from the examiner. In addition, on / off of an instruction by a character, a voice, or the like may be configured to be selectable according to an instruction from the examiner.
　ここで、機械学習には、上述したような深層学習があり、また、多階層のニューラルネットワークの少なくとも一部には、例えば、再帰型ニューラルネットワーク（ＲＮＮ：Ｒｅｃｕｒｒｅｎｔ　Ｎｅｕｒａｌ　Ｎｅｔｗｏｒｋ）を用いることができる。ここで、本変形例に係る機械学習モデルの一例として、時系列情報を扱うニューラルネットワークであるＲＮＮに関して、図４６Ａ及び図４６Ｂを参照して説明する。また、ＲＮＮの一種であるＬｏｎｇ　ｓｈｏｒｔ－ｔｅｒｍ　ｍｅｍｏｒｙ（以下、ＬＳＴＭ）に関して、図４７Ａ及び図４７Ｂを参照して説明する。 Here, machine learning includes deep learning as described above, and a recurrent neural network (RNN: Recurrent Neural Network) can be used as at least a part of the multi-layer neural network, for example. Here, as an example of the machine learning model according to the present modification, an RNN that is a neural network that handles time-series information will be described with reference to FIGS. 46A and 46B. In addition, Long @ short-term @ memory (hereinafter, LSTM), which is a kind of RNN, will be described with reference to FIGS. 47A and 47B.
　図４６Ａは、機械学習モデルであるＲＮＮの構造を示す。ＲＮＮ４６２０は、ネットワークにループ構造を持ち、時刻ｔにおいてデータｘｔ４６１０が入力され、データｈｔ４６３０を出力する。ＲＮＮ４６２０はネットワークにループ機能を持つため、現時刻の状態を次の状態に引き継ぐことが可能であるため、時系列情報を扱うことができる。図４６Ｂには時刻ｔにおけるパラメータベクトルの入出力の一例を示す。データｘｔ４６１０にはＮ個（Ｐａｒａｍｓ１～ＰａｒａｍｓＮ）のデータが含まれる。また、ＲＮＮ４６２０より出力されるデータｈｔ４６３０には入力データに対応するＮ個（Ｐａｒａｍｓ１～ＰａｒａｍｓＮ）のデータが含まれる。 FIG. 46A shows the structure of RNN which is a machine learning model. RNN4620 has a loop structure to the network, the data x t 4610 at time t, and outputs the data h t 4630. Since the RNN 4620 has a loop function in the network, the state at the current time can be taken over to the next state, so that the time series information can be handled. FIG. 46B shows an example of input / output of the parameter vector at time t. The data x t 4610 includes N data (Params 1 to Params N). Further, the data h t 4630 output from RNN4620 includes data of N (Params1 ~ ParamsN) corresponding to the input data.
　しかしながら、ＲＮＮでは誤差逆伝搬時に長期時間の情報を扱うことができないため、ＬＳＴＭが用いられることがある。ＬＳＴＭは、忘却ゲート、入力ゲート、及び出力ゲートを備えることで長期時間の情報を学習することができる。ここで、図４７ＡにＬＳＴＭの構造を示す。ＬＳＴＭ４７４０において、ネットワークが次の時刻ｔに引き継ぐ情報は、セルと呼ばれるネットワークの内部状態ｃｔ－１と出力データｈｔ－１である。なお、図の小文字（ｃ、ｈ、ｘ）はベクトルを表している。 However, since the RNN cannot handle long-term information at the time of error back propagation, LSTM may be used. The LSTM can learn long-term information by providing a forgetting gate, an input gate, and an output gate. Here, FIG. 47A shows the structure of the LSTM. In the LSTM4740, the information that the network takes over at the next time t is the internal state ct -1 of the network called a cell and the output data ht -1 . Note that lowercase letters (c, h, x) in the figure represent vectors.
　次に、図４７ＢにＬＳＴＭ４７４０の詳細を示す。図４７Ｂにおいては、忘却ゲートネットワークＦＧ、入力ゲートネットワークＩＧ、及び出力ゲートネットワークＯＧが示され、それぞれはシグモイド層である。そのため、各要素が０から１の値となるベクトルを出力する。忘却ゲートネットワークＦＧは過去の情報をどれだけ保持するかを決め、入力ゲートネットワークＩＧはどの値を更新するかを判定するものである。また、図４７Ｂにおいては、セル更新候補ネットワークＣＵが示され、セル更新候補ネットワークＣＵは活性化関数ｔａｎｈ層である。これは、セルに加えられる新たな候補値のベクトルを作成する。出力ゲートネットワークＯＧは、セル候補の要素を選択し次の時刻にどの程度の情報を伝えるか選択する。 Next, FIG. 47B shows details of the LSTM4740. FIG. 47B shows a forgetting gate network FG, an input gate network IG, and an output gate network OG, each of which is a sigmoid layer. Therefore, a vector in which each element takes a value from 0 to 1 is output. The forgetting gate network FG determines how much past information is retained, and the input gate network IG determines which value to update. In FIG. 47B, a cell update candidate network CU is shown, and the cell update candidate network CU is an activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gate network OG selects the element of the cell candidate and selects how much information to transmit at the next time.
　なお、上述したＬＳＴＭのモデルは基本形であるため、ここで示したネットワークに限らない。ネットワーク間の結合を変更してもよい。ＬＳＴＭではなく、ＱＲＮＮ（Ｑｕａｓｉ　Ｒｅｃｕｒｒｅｎｔ　Ｎｅｕｒａｌ　Ｎｅｔｗｏｒｋ）を用いてもよい。さらに、機械学習モデルは、ニューラルネットワークに限定されるものではなく、ブースティングやサポートベクターマシン等が用いられてもよい。また、検者からの指示が文字又は音声等による入力の場合には、自然言語処理に関する技術（例えば、Ｓｅｑｕｅｎｃｅ　ｔｏ　Ｓｅｑｕｅｎｃｅ）が適用されてもよい。また、検者に対して文字又は音声等による出力で応答する対話エンジン（対話モデル、対話用の学習済モデル）が適用されてもよい。 The LSTM model described above is a basic model, and is not limited to the network shown here. The coupling between the networks may be changed. Instead of the LSTM, a QRNN (Quasi \ Current \ Neural \ Network) may be used. Further, the machine learning model is not limited to the neural network, and boosting, a support vector machine, or the like may be used. When the instruction from the examiner is an input using characters or voice, a technology related to natural language processing (for example, Sequence to Sequence) may be applied. Further, a dialogue engine (a dialogue model, a trained model for a dialogue) that responds to the examiner with an output using characters or voices may be applied.
（変形例１１）
　上述した様々な実施例及び変形例において、境界画像や領域ラベル画像、高画質画像等は、操作者からの指示に応じて記憶部に保存されてもよい。このとき、例えば、高画質画像を保存するための操作者からの指示の後、ファイル名の登録の際に、推奨のファイル名として、ファイル名のいずれかの箇所（例えば、最初の箇所、又は最後の箇所）に、高画質化用の学習済モデルを用いた処理（高画質化処理）により生成された画像であることを示す情報（例えば、文字）を含むファイル名が、操作者からの指示に応じて編集可能な状態で表示されてもよい。なお、同様に、境界画像や領域ラベル画像等についても、学習済モデルを用いた処理により生成された画像である情報を含むファイル名が表示されてもよい。 (Modification 11)
In the various embodiments and modifications described above, the boundary image, the region label image, the high-quality image, and the like may be stored in the storage unit according to an instruction from the operator. At this time, for example, after the instruction from the operator to save the high-quality image, when registering the file name, any part of the file name (for example, the first part or In the last part), a file name including information (for example, characters) indicating that the image is an image generated by the process using the learned model for image quality improvement (image quality improvement process) is received from the operator. It may be displayed in an editable state according to the instruction. Similarly, for a boundary image, an area label image, and the like, a file name including information that is an image generated by a process using the learned model may be displayed.
　また、レポート画面等の種々の表示画面において、表示部５０，２８２０に高画質画像を表示させる際に、表示されている画像が高画質化用の学習済モデルを用いた処理により生成された高画質画像であることを示す表示が、高画質画像とともに表示されてもよい。この場合には、操作者は、当該表示によって、表示された高画質画像が撮影によって取得した画像そのものではないことが容易に識別できるため、誤診断を低減させたり、診断効率を向上させたりすることができる。なお、高画質化用の学習済モデルを用いた処理により生成された高画質画像であることを示す表示は、入力画像と当該処理により生成された高画質画像とを識別可能な表示であればどのような態様のものでもよい。また、高画質化用の学習済モデルを用いた処理だけでなく、上述したような種々の学習済モデルを用いた処理についても、その種類の学習済モデルを用いた処理により生成された結果であることを示す表示が、その結果とともに表示されてもよい。また、画像セグメンテーション処理用の学習済モデルを用いたセグメンテーション結果の解析結果を表示する際にも、画像セグメンテーション用の学習済モデルを用いた結果に基づいた解析結果であることを示す表示が、解析結果とともに表示されてもよい。 In addition, when displaying high-quality images on the display units 50 and 2820 on various display screens such as a report screen, the displayed image is generated by processing using a trained model for improving image quality. A display indicating that the image is a high-quality image may be displayed together with the high-quality image. In this case, the operator can easily identify from the display that the displayed high-quality image is not the image itself obtained by shooting, thereby reducing erroneous diagnosis or improving diagnosis efficiency. be able to. Note that the display indicating that the image is a high-quality image generated by the process using the learned model for improving the image quality is a display that can identify the input image and the high-quality image generated by the process. Any mode may be used. Further, not only the processing using the learned model for high image quality but also the processing using the various learned models described above are the results generated by the processing using the type of the learned model. A display indicating the presence may be displayed together with the result. Also, when displaying the analysis result of the segmentation result using the trained model for the image segmentation process, the display indicating that the analysis result is based on the result using the trained model for the image segmentation is displayed. It may be displayed together with the result.
　このとき、レポート画面等の表示画面は、操作者からの指示に応じて記憶部に保存されてもよい。例えば、高画質画像等と、これらの画像が学習済モデルを用いた処理により生成された画像であることを示す表示とが並んだ１つの画像としてレポート画面が記憶部に保存されてもよい。 At this time, a display screen such as a report screen may be stored in the storage unit in accordance with an instruction from the operator. For example, the report screen may be stored in the storage unit as one image in which high-quality images and the like and a display indicating that these images are images generated by processing using the learned model are arranged.
　また、高画質化用の学習済モデルを用いた処理により生成された高画質画像であることを示す表示について、高画質化用の学習済モデルがどのような学習データによって学習を行ったものであるかを示す表示が表示部に表示されてもよい。当該表示としては、学習データの入力データと正解データの種類の説明や、入力データと正解データに含まれる撮影部位等の正解データに関する任意の表示を含んでよい。なお、例えば画像セグメンテーション処理等上述した種々の学習済モデルを用いた処理についても、その種類の学習済モデルがどのような学習データによって学習を行ったものであるかを示す表示が表示部に表示されてもよい。 In addition, the display indicating that the image is a high-quality image generated by processing using the learned model for high image quality is obtained by learning the learning model for high image quality with what learning data. A display indicating whether or not there is may be displayed on the display unit. The display may include an explanation of the types of the input data and the correct answer data of the learning data and an arbitrary display related to the correct answer data such as the imaging part included in the input data and the correct answer data. In addition, for the processing using the various learned models described above, such as the image segmentation processing, a display indicating what learning data is used by the type of the learned model is displayed on the display unit. May be done.
　また、学習済モデルを用いた処理により生成された画像であることを示す情報（例えば、文字）を、画像等に重畳した状態で表示又は保存されるように構成されてもよい。このとき、画像上に重畳する箇所は、撮影対象となる注目部位等が表示されている領域には重ならない領域（例えば、画像の端）であればどこでもよい。また、重ならない領域を判定し、判定された領域に重畳させてもよい。なお、高画質化用の学習済モデルを用いた処理だけでなく、例えば画像セグメンテーション処理等の上述した種々の学習済モデルを用いた処理により得た画像についても、同様に処理してよい。 The information (for example, characters) indicating that the image is generated by the process using the learned model may be displayed or saved in a state of being superimposed on the image or the like. At this time, the portion to be superimposed on the image may be any region (for example, the end of the image) that does not overlap with the region where the target region to be photographed is displayed. Alternatively, a non-overlapping area may be determined and superimposed on the determined area. In addition, not only the process using the learned model for improving the image quality, but also the image obtained by the process using the various learned models described above, such as the image segmentation process, may be similarly processed.
　また、レポート画面の初期表示画面として、図２２Ａ及び図２２Ｂに示すようなボタン２２２０がアクティブ状態（高画質化処理がオン）となるようにデフォルト設定されている場合には、検者からの指示に応じて、高画質画像等を含むレポート画面に対応するレポート画像がサーバに送信されるように構成されてもよい。また、ボタン２２２０がアクティブ状態となるようにデフォルト設定されている場合には、検査終了時（例えば、検者からの指示に応じて、撮影確認画面やプレビュー画面からレポート画面に変更された場合）に、高画質画像等を含むレポート画面に対応するレポート画像がサーバに（自動的に）送信されるように構成されてもよい。このとき、デフォルト設定における各種設定（例えば、レポート画面の初期表示画面におけるＥｎ－Ｆａｃｅ画像の生成のための深度範囲、解析マップの重畳の有無、高画質画像か否か、経過観察用の表示画面か否か等の少なくとも１つに関する設定）に基づいて生成されたレポート画像がサーバに送信されるように構成されもよい。なお、ボタン２２２０が画像セグメンテーション処理の切り替えを表す場合に関しても、同様に処理されてよい。 When the button 2220 as shown in FIGS. 22A and 22B is set to the active state (the high-quality processing is turned on) as an initial display screen of the report screen, an instruction from the examiner is given. May be configured to transmit a report image corresponding to a report screen including a high-quality image or the like to the server. In addition, when the button 2220 is set to the active state by default, at the end of the examination (for example, when the photographing confirmation screen or the preview screen is changed to the report screen in response to an instruction from the examiner). Alternatively, a report image corresponding to a report screen including a high-quality image or the like may be configured to be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, a depth range for generating an En-Face image on the initial display screen of the report screen, presence / absence of superposition of the analysis map, whether or not the image is a high-quality image, a display screen for follow-up observation The report image generated based on at least one of the settings, such as whether or not the report image may be transmitted to the server. Note that the same processing may be performed when the button 2220 indicates switching of the image segmentation processing.
（変形例１２）
　上述した様々な実施例及び変形例において、上述したような種々の学習済モデルのうち、第一の種類の学習済モデルで得た画像（例えば、高画質画像、解析マップ等の解析結果を示す画像、物体認識結果を示す画像、網膜層を示す画像、セグメンテーション結果を示す画像）を、第一の種類とは異なる第二の種類の学習済モデルに入力してもよい。このとき、第二の種類の学習済モデルの処理による結果（例えば、解析結果、診断結果、物体認識結果、網膜層の検出結果、セグメンテーション結果）が生成されるように構成されてもよい。 (Modification 12)
In the above-described various embodiments and modified examples, among the various learned models described above, an image (for example, a high-quality image, an analysis result such as an analysis map) obtained by the first type of learned model is shown. An image, an image indicating an object recognition result, an image indicating a retinal layer, and an image indicating a segmentation result) may be input to a second type of trained model different from the first type. At this time, a result (for example, an analysis result, a diagnosis result, an object recognition result, a detection result of a retinal layer, a segmentation result) by the processing of the second type of learned model may be generated.
　また、上述したような種々の学習済モデルのうち、第一の種類の学習済モデルの処理による結果（例えば、解析結果、診断結果、物体認識結果、網膜層の検出結果、セグメンテーション結果）を用いて、第一の種類の学習済モデルに入力した画像から、第一の種類とは異なる第一の種類の学習済モデルに入力する画像を生成してもよい。このとき、生成された画像は、第二の種類の学習済モデルを用いて処理する画像として適した画像である可能性が高い。このため、生成された画像を第二の種類の学習済モデルに入力して得た画像（例えば、高画質画像、解析マップ等の解析結果を示す画像、物体認識結果を示す画像、網膜層を示す画像、セグメンテーション結果を示す画像）の精度を向上することができる。 In addition, among the various learned models described above, a result of processing of the first type of learned model (for example, an analysis result, a diagnosis result, an object recognition result, a retinal layer detection result, and a segmentation result) is used. Then, an image to be input to a first type of learned model different from the first type may be generated from an image input to the first type of learned model. At this time, the generated image is likely to be an image suitable as an image to be processed using the second type of learned model. For this reason, an image obtained by inputting the generated image to the second type of trained model (for example, a high-quality image, an image indicating an analysis result such as an analysis map, an image indicating an object recognition result, The accuracy of the image shown and the image showing the segmentation result can be improved.
　また、上述したような学習済モデルの処理による解析結果や診断結果等を検索キーとして、サーバ等に格納された外部のデータベースを利用した類似画像検索を行ってもよい。なお、データベースにおいて保存されている複数の画像が、既に機械学習等によって該複数の画像それぞれの特徴量を付帯情報として付帯された状態で管理されている場合等には、画像自体を検索キーとする類似画像検索エンジン（類似画像検索モデル、類似画像検索用の学習済モデル）が用いられてもよい。 {Circle around (2)} Similar image search using an external database stored in a server or the like may be performed using, as a search key, an analysis result, a diagnosis result, or the like obtained by processing the learned model as described above. In the case where a plurality of images stored in the database are already managed in such a manner that feature amounts of the plurality of images are attached as additional information by machine learning or the like, the images themselves are used as search keys. A similar image search engine (similar image search model, trained model for similar image search) may be used.
（変形例１３）
　なお、上記実施例及び変形例におけるモーションコントラストデータの生成処理は、断層画像の輝度値に基づいて行われる構成に限られない。上記各種処理は、ＯＣＴ装置１０又は撮影装置２８１０で取得された干渉信号、干渉信号にフーリエ変換を施した信号、該信号に任意の処理を施した信号、及びこれらに基づく断層画像等を含む断層データに対して適用されてよい。これらの場合も、上記構成と同様の効果を奏することができる。 (Modification 13)
Note that the generation processing of the motion contrast data in the above embodiment and the modification is not limited to the configuration performed based on the luminance value of the tomographic image. The various processes include an interference signal acquired by the OCT device 10 or the imaging device 2810, a signal obtained by performing a Fourier transform on the interference signal, a signal obtained by performing an arbitrary process on the signal, and a tomographic image including a tomographic image based on the signals. May be applied to data. In these cases, the same effect as the above configuration can be obtained.
　分割手段としてカプラを使用したファイバ光学系を用いているが、コリメータとビームスプリッタを使用した空間光学系を用いてもよい。また、ＯＣＴ装置１０又は撮影装置２８１０の構成は、上記の構成に限られず、ＯＣＴ装置１０又は撮影装置２８１０に含まれる構成の一部をＯＣＴ装置１０又は撮影装置２８１０と別体の構成としてもよい。 Although a fiber optical system using a coupler is used as the dividing means, a spatial optical system using a collimator and a beam splitter may be used. The configuration of the OCT device 10 or the imaging device 2810 is not limited to the above configuration, and a part of the configuration included in the OCT device 10 or the imaging device 2810 may be configured separately from the OCT device 10 or the imaging device 2810. .
　また、上記実施例及び変形例では、ＯＣＴ装置１０又は撮影装置２８１０の干渉光学系としてマッハツェンダー型干渉計の構成を用いているが、干渉光学系の構成はこれに限られない。例えば、ＯＣＴ装置１０又は撮影装置２８１０の干渉光学系はマイケルソン干渉計の構成を有していてもよい。 Further, in the above-described embodiments and modified examples, the configuration of the Mach-Zehnder interferometer is used as the interference optical system of the OCT device 10 or the imaging device 2810, but the configuration of the interference optical system is not limited to this. For example, the interference optical system of the OCT apparatus 10 or the imaging apparatus 2810 may have a configuration of a Michelson interferometer.
　さらに、上記実施例及び変形例では、ＯＣＴ装置として、ＳＬＤを光源として用いたスペクトラルドメインＯＣＴ（ＳＤ－ＯＣＴ）装置について述べたが、本発明によるＯＣＴ装置の構成はこれに限られない。例えば、出射光の波長を掃引することができる波長掃引光源を用いた波長掃引型ＯＣＴ（ＳＳ－ＯＣＴ）装置等の他の任意の種類のＯＣＴ装置にも本発明を適用することができる。また、ライン光を用いたＬｉｎｅ－ＯＣＴ装置に対して本発明を適用することもできる。 Further, in the above-described embodiments and modified examples, a spectral domain OCT (SD-OCT) device using an SLD as a light source has been described as an OCT device, but the configuration of the OCT device according to the present invention is not limited to this. For example, the present invention can be applied to any other type of OCT device such as a wavelength-swept OCT (SS-OCT) device using a wavelength-swept light source capable of sweeping the wavelength of emitted light. Further, the present invention can also be applied to a Line-OCT apparatus using line light.
　また、上記実施例及び変形例では、取得部２１，２８０１は、ＯＣＴ装置１０又は撮影装置２８１０で取得された干渉信号や画像処理装置で生成された三次元断層画像等を取得した。しかしながら、取得部２１，２８０１がこれらの信号や画像を取得する構成はこれに限られない。例えば、取得部２１，２８０１は、制御部とＬＡＮ、ＷＡＮ、又はインターネット等を介して接続されるサーバや撮影装置からこれらの信号を取得してもよい。 In addition, in the above embodiments and modifications, the acquisition units 21 and 2801 acquired the interference signal acquired by the OCT device 10 or the imaging device 2810, the three-dimensional tomographic image generated by the image processing device, and the like. However, the configuration in which the acquisition units 21 and 801 acquire these signals and images is not limited to this. For example, the acquisition units 21 and 801 may acquire these signals from a server or an imaging device connected to the control unit via a LAN, a WAN, the Internet, or the like.
　なお、学習済モデルは、画像処理装置２０，８０，１５２，１７２，２８００に設けられることができる。学習済モデルは、例えば、ＣＰＵ等のプロセッサーによって実行されるソフトウェアモジュール等で構成されることができる。また、学習済モデルは、画像処理装置２０，８０，１５２，１７２，２８００と接続される別のサーバ等に設けられてもよい。この場合には、画像処理装置２０，８０，１５２，１７２，２８００は、インターネット等の任意のネットワークを介して学習済モデルを備えるサーバに接続することで、学習済モデルを用いて画質向上処理を行うことができる。 The learned model can be provided in the image processing apparatuses 20, 80, 152, 172, and 2800. The learned model can be constituted by, for example, a software module executed by a processor such as a CPU. In addition, the learned model may be provided in another server or the like connected to the image processing apparatuses 20, 80, 152, 172, and 2800. In this case, the image processing apparatuses 20, 80, 152, 172, and 2800 perform image quality improvement processing using the learned model by connecting to a server having the learned model via an arbitrary network such as the Internet. It can be carried out.
（変形例１４）
　また、上述した様々な実施例及び変形例による画像処理装置又は画像処理方法によって処理される画像は、任意のモダリティ（撮影装置、撮影方法）を用いて取得された医用画像を含む。処理される医用画像は、任意の撮影装置等で取得された医用画像や、上記実施例及び変形例による画像処理装置又は画像処理方法によって作成された画像を含むことができる。 (Modification 14)
Further, the image processed by the image processing device or the image processing method according to the various embodiments and the modified examples described above includes a medical image acquired using an arbitrary modality (imaging device, imaging method). The medical image to be processed can include a medical image acquired by an arbitrary imaging device or the like, and an image created by the image processing apparatus or the image processing method according to the above-described embodiment and the modification.
　さらに、処理される医用画像は、被検者（被検体）の所定部位の画像であり、所定部位の画像は被検者の所定部位の少なくとも一部を含む。また、当該医用画像は、被検者の他の部位を含んでもよい。また、医用画像は、静止画像又は動画像であってよく、白黒画像又はカラー画像であってもよい。さらに医用画像は、所定部位の構造（形態）を表す画像でもよいし、その機能を表す画像でもよい。機能を表す画像は、例えば、ＯＣＴＡ画像、ドップラーＯＣＴ画像、ｆＭＲＩ画像、及び超音波ドップラー画像等の血流動態（血流量、血流速度等）を表す画像を含む。なお、被検者の所定部位は、撮影対象に応じて決定されてよく、人眼（被検眼）、脳、肺、腸、心臓、すい臓、腎臓、及び肝臓等の臓器、頭部、胸部、脚部、並びに腕部等の任意の部位を含む。 {Furthermore, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. Further, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black and white image or a color image. Further, the medical image may be an image representing the structure (form) of the predetermined part or an image representing the function thereof. The images representing functions include, for example, images representing blood flow dynamics (blood flow, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasonic Doppler images. Note that the predetermined site of the subject may be determined according to the imaging target, and includes organs such as the human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, and liver, head, chest, Includes any parts such as legs and arms.
　また、医用画像は、被検者の断層画像であってもよいし、正面画像であってもよい。正面画像は、例えば、眼底正面画像や、前眼部の正面画像、蛍光撮影された眼底画像、ＯＣＴで取得したデータ（三次元のＯＣＴデータ）について撮影対象の深さ方向における少なくとも一部の範囲のデータを用いて生成したＥｎ－Ｆａｃｅ画像を含む。Ｅｎ－Ｆａｃｅ画像は、三次元のＯＣＴＡデータ（三次元のモーションコントラストデータ）について撮影対象の深さ方向における少なくとも一部の範囲のデータを用いて生成したＯＣＴＡのＥｎ－Ｆａｃｅ画像（モーションコントラスト正面画像）でもよい。また、三次元のＯＣＴデータや三次元のモーションコントラストデータは、三次元の医用画像データの一例である。 医 Also, the medical image may be a tomographic image of the subject or a front image. The front image is, for example, at least a part of the fundus front image, the front image of the anterior eye part, the fundus image obtained by fluorescence imaging, and data obtained by OCT (three-dimensional OCT data) in the depth direction of the imaging target. And an En-Face image generated using the data of FIG. The En-Face image is an OCTA En-Face image (motion contrast front image) generated using three-dimensional OCTA data (three-dimensional motion contrast data) using data in at least a part of the range in the depth direction of the imaging target. ). The three-dimensional OCT data and the three-dimensional motion contrast data are examples of three-dimensional medical image data.
　また、撮影装置とは、診断に用いられる画像を撮影するための装置である。撮影装置は、例えば、被検者の所定部位に光、Ｘ線等の放射線、電磁波、又は超音波等を照射することにより所定部位の画像を得る装置や、被写体から放出される放射線を検出することにより所定部位の画像を得る装置を含む。より具体的には、上述した様々な実施例及び変形例に係る撮影装置は、少なくとも、Ｘ線撮影装置、ＣＴ装置、ＭＲＩ装置、ＰＥＴ装置、ＳＰＥＣＴ装置、ＳＬＯ装置、ＯＣＴ装置、ＯＣＴＡ装置、眼底カメラ、及び内視鏡等を含む。 撮 影 The imaging device is a device for imaging an image used for diagnosis. The imaging apparatus detects, for example, a device that obtains an image of a predetermined portion by irradiating a predetermined portion of the subject with light, radiation such as X-rays, electromagnetic waves, or ultrasonic waves, or detects radiation emitted from a subject. And a device for obtaining an image of a predetermined part. More specifically, the imaging apparatuses according to the various embodiments and modifications described above include at least an X-ray imaging apparatus, a CT apparatus, an MRI apparatus, a PET apparatus, a SPECT apparatus, an SLO apparatus, an OCT apparatus, an OCTA apparatus, and a fundus. It includes a camera and an endoscope.
　なお、ＯＣＴ装置としては、タイムドメインＯＣＴ（ＴＤ－ＯＣＴ）装置やフーリエドメインＯＣＴ（ＦＤ－ＯＣＴ）装置を含んでよい。また、フーリエドメインＯＣＴ装置はスペクトラルドメインＯＣＴ（ＳＤ－ＯＣＴ）装置や波長掃引型ＯＣＴ（ＳＳ－ＯＣＴ）装置を含んでよい。また、ＳＬＯ装置やＯＣＴ装置として、波面補償光学系を用いた波面補償ＳＬＯ（ＡＯ－ＳＬＯ）装置や波面補償ＯＣＴ（ＡＯ－ＯＣＴ）装置等を含んでよい。また、ＳＬＯ装置やＯＣＴ装置として、偏光位相差や偏光解消に関する情報を可視化するための偏光ＳＬＯ（ＰＳ－ＳＬＯ）装置や偏光ＯＣＴ（ＰＳ－ＯＣＴ）装置等を含んでよい。 The OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. Further, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a wavelength sweep type OCT (SS-OCT) device. Further, the SLO device or OCT device may include a wavefront compensation SLO (AO-SLO) device using a wavefront compensation optical system, a wavefront compensation OCT (AO-OCT) device, or the like. Further, the SLO device and the OCT device may include a polarization SLO (PS-SLO) device, a polarization OCT (PS-OCT) device, and the like for visualizing information on a polarization phase difference and depolarization.
　また、上述の様々な実施例及び変形例に係る網膜層検出用や画像セグメンテーション処理用の学習済モデルでは、断層画像の輝度値の大小、明部と暗部の順番や傾き、位置、分布、連続性等を特徴量の一部として抽出して、推定処理に用いているものと考えらえる。同様に、領域ラベル画像の評価用や高画質化用、画像解析用、診断結果生成用の学習済モデルでも、断層画像の輝度値の大小、明部と暗部の順番や傾き、位置、分布、連続性等を特徴量の一部として抽出して、推定処理に用いているものと考えらえる。一方で、音声認識用や文字認識用、ジェスチャー認識用等の学習済モデルでは、時系列のデータを用いて学習を行っているため、入力される連続する時系列のデータ値間の傾きを特徴量の一部として抽出し、推定処理に用いているものと考えられる。そのため、このような学習済モデルは、具体的な数値の時間的な変化による影響を推定処理に用いることで、精度のよい推定を行うことができると期待される。 Further, in the learned model for retinal layer detection and image segmentation processing according to the various embodiments and modifications described above, the magnitude of the brightness value of the tomographic image, the order and inclination, position, distribution, and continuity of the bright and dark parts of the tomographic image. It is considered that gender and the like are extracted as a part of the feature amount and are used in the estimation processing. Similarly, in the trained model for evaluating the region label image, improving the image quality, analyzing the image, and generating the diagnosis result, the magnitude of the brightness value of the tomographic image, the order and inclination, position, distribution, It can be considered that continuity and the like are extracted as a part of the feature amount and are used in the estimation processing. On the other hand, in the trained models for voice recognition, character recognition, gesture recognition, etc., learning is performed using time-series data. It is considered that it was extracted as part of the quantity and used for the estimation processing. Therefore, such a learned model is expected to be able to perform accurate estimation by using the influence of a specific numerical change over time in the estimation process.
（様々な実施態様）
　本開示の実施態様１は医用画像処理装置に関する。該医用画像処理装置は、被検眼の断層画像を取得する取得部と、被検眼の断層画像において複数の網膜層のうち少なくとも一つの網膜層が示されたデータを学習して得た学習済モデルを用いて、前記取得された断層画像において複数の網膜層のうち少なくとも一つの網膜層を検出するための第一の検出処理を実行する第一の処理部とを備える。 (Various embodiments)
Embodiment 1 of the present disclosure relates to a medical image processing apparatus. The medical image processing apparatus includes: an acquisition unit configured to acquire a tomographic image of an eye to be inspected; and a learned model obtained by learning data indicating at least one retinal layer among a plurality of retinal layers in the tomographic image of the eye to be inspected. And a first processing unit that executes a first detection process for detecting at least one retinal layer among a plurality of retinal layers in the acquired tomographic image.
　実施態様２は、実施態様１に係る医用画像処理装置を含み、機械学習により得られる学習済モデルを用いずに、前記取得された断層画像において前記複数の網膜層のうち少なくとも一つの網膜層を検出するための第二の検出処理を実行する第二の処理部を更に備える。 Embodiment 2 includes the medical image processing apparatus according to Embodiment 1, and uses at least one retinal layer among the plurality of retinal layers in the acquired tomographic image without using a learned model obtained by machine learning. The image processing apparatus further includes a second processing unit that executes a second detection process for detecting.
　実施態様３は、実施態様２に係る医用画像処理装置を含み、前記第二の検出処理は、前記第一の検出処理を実行することにより検出された少なくとも一つの網膜層以外の少なくとも一つの網膜層を検出する処理である。 Embodiment 3 includes the medical image processing apparatus according to Embodiment 2, wherein the second detection processing is at least one retinal layer other than at least one retinal layer detected by executing the first detection processing. This is a process of detecting a layer.
　実施態様４は、実施態様２又は３に係る医用画像処理装置を含み、前記第一の検出処理は、前記少なくとも一つの網膜層として、前記取得された断層画像において網膜領域を検出する処理であり、前記第二の検出処理は、前記第一の検出処理を実行することにより検出された網膜領域における少なくとも一つの網膜層を検出する処理である。 Embodiment 4 includes the medical image processing apparatus according to Embodiment 2 or 3, wherein the first detection processing is processing for detecting a retinal region in the acquired tomographic image as the at least one retinal layer. The second detection process is a process of detecting at least one retinal layer in a retinal region detected by executing the first detection process.
　実施態様５は、実施態様２乃至４のいずれか一つに係る医用画像処理装置を含み、前記第一の検出処理は、被検眼の内境界膜と神経線維層との境界から視細胞内節外節接合部、網膜色素上皮層、及びブルッフ膜のいずれかまでの層を検出する処理であり、前記第二の検出処理は、前記第一の検出処理により検出した層の間の少なくとも一つの網膜層を検出する処理である。 A fifth embodiment includes the medical image processing apparatus according to any one of the second to fourth embodiments, wherein the first detection processing is performed based on a boundary between an inner limiting membrane and a nerve fiber layer of an eye to be examined, and a photoreceptor inner segment. Outer segment junction, retinal pigment epithelium layer, and a process to detect the layer up to any of Bruch's membrane, the second detection process, at least one of the layers detected by the first detection process This is a process for detecting a retinal layer.
　実施態様６は、実施態様２乃至５のいずれか一つに係る医用画像処理装置を含み、前記第二の処理部は、前記第一の処理部による前記第一の検出処理の後に、前記第二の検出処理を実行する。 A sixth embodiment includes the medical image processing device according to any one of the second to fifth embodiments, wherein the second processing unit performs the second detection after the first detection processing by the first processing unit. The second detection process is executed.
　実施態様７は、実施態様２に係る医用画像処理装置を含み、表示部を制御する表示制御部を更に備え、前記第一の検出処理及び前記第二の検出処理は、同一の網膜層を検出する処理であり、前記表示制御部は、前記第一の検出処理及び前記第二の検出処理の処理結果を前記表示部に表示させる。 A seventh embodiment includes the medical image processing device according to the second embodiment, and further includes a display control unit that controls a display unit, wherein the first detection process and the second detection process detect the same retinal layer The display control unit causes the display unit to display processing results of the first detection processing and the second detection processing.
　実施態様８は、実施態様７に係る医用画像処理装置を含み、前記表示制御部は、前記第一の検出処理及び前記第二の検出処理の処理結果の不一致部分を前記表示部に表示させる。 [Eighth Embodiment] An eighth embodiment includes the medical image processing apparatus according to the seventh embodiment, and the display control unit causes the display unit to display a mismatched portion between the processing results of the first detection process and the second detection process.
　実施態様９は、実施態様７又は８に係る医用画像処理装置を含み、前記第一の検出処理及び前記第二の検出処理は、被検眼の内境界膜と神経線維層との境界から視細胞内節外節接合部、網膜色素上皮層、及びブルッフ膜のいずれかまでの層を検出する処理であり、前記第二の処理部は、操作者の指示に応じて、前記第一の検出処理及び前記第二の検出処理のいずれか一方により検出した層の間の少なくとも一つの網膜層を検出する第三の検出処理を更に実行する。 A ninth embodiment includes the medical image processing apparatus according to the seventh or eighth embodiment, wherein the first detection processing and the second detection processing are performed using a photoreceptor cell from a boundary between an inner limiting membrane and a nerve fiber layer of an eye to be examined. Inner / outer segment junction, retinal pigment epithelium layer, and processing to detect the layer up to any of Bruch's membrane, the second processing unit, according to an instruction of the operator, the first detection processing And a third detection process of detecting at least one retinal layer between the layers detected by one of the second detection processes.
　実施態様１０は、実施態様２乃至９のいずれか一つに係る医用画像処理装置を含み、前記取得された断層画像に関する撮影条件に基づいて、前記第一の検出処理と前記第二の検出処理のうち少なくとも一つの選択を行う選択部を更に備える。 A tenth embodiment includes the medical image processing apparatus according to any one of the second to ninth embodiments, wherein the first detection process and the second detection process are performed based on imaging conditions for the acquired tomographic image. And a selecting unit for selecting at least one of the above.
　実施態様１１は、実施態様２乃至１０のいずれか一つに係る医用画像処理装置を含み、前記第一の処理部は、異なる学習データを用いて機械学習が行われた複数の学習済モデルのうち、前記取得された断層画像に関する撮影条件に対応する学習データを用いて機械学習が行われた学習済モデルを用いて、前記第一の検出処理を実行する。 An eleventh embodiment includes the medical image processing apparatus according to any one of the second to tenth embodiments, wherein the first processing unit is configured to execute a plurality of learned models on which machine learning has been performed using different learning data. The first detection process is performed using a learned model in which machine learning has been performed using learning data corresponding to imaging conditions for the acquired tomographic image.
　実施態様１２は、実施態様１０又は１１に係る医用画像処理装置を含み、前記撮影条件は、撮影部位、撮影方式、撮影領域、撮影画角、及び画像の解像度のうち少なくとも一つを含む。 {Embodiment 12] The medical image processing apparatus according to Embodiment 10 or 11, wherein the imaging conditions include at least one of an imaging region, an imaging method, an imaging region, an imaging angle of view, and an image resolution.
　実施態様１３は、実施態様２乃至１２のいずれか一つに係る医用画像処理装置を含み、前記第一の検出処理及び前記第二の検出処理の結果に基づいて、被検眼の形状特徴が計測される。 The thirteenth embodiment includes the medical image processing apparatus according to any one of the second to twelfth embodiments, and measures a shape characteristic of the eye to be inspected based on a result of the first detection processing and the second detection processing. Is done.
　実施態様１４は、実施態様１乃至１３のいずれか一つに係る医用画像処理装置を含み、網膜層における医学的特徴に基づいて、前記第一の処理部が検出した網膜層の構造を補正する補正部を更に備える。 Embodiment 14 includes the medical image processing apparatus according to any one of Embodiments 1 to 13, and corrects the structure of the retinal layer detected by the first processing unit based on medical characteristics in the retinal layer. A correction unit is further provided.
　実施態様１５は、実施態様１乃至１４のいずれか一つに係る医用画像処理装置を含み、前記第一の処理部は、前記学習済モデルを用いて、入力された画像について撮影部位毎に予め定められた境界を検出する。 A fifteenth embodiment includes the medical image processing apparatus according to any one of the first to fourteenth embodiments, wherein the first processing unit uses the learned model to determine in advance an input image for each imaging region using the learned model. Detects defined boundaries.
　実施態様１６は、実施態様１乃至１５のいずれか一つに係る医用画像処理装置を含み、被検眼の三次元の断層画像における少なくとも一部の深度範囲であって、前記検出された少なくとも一つの網膜層に基づいて決定された深度範囲に対応する正面画像を生成する生成部を更に備える。 Embodiment 16 includes the medical image processing apparatus according to any one of Embodiments 1 to 15, wherein at least a part of the depth range in the three-dimensional tomographic image of the subject's eye is included, and the at least one detected The image processing apparatus further includes a generation unit that generates a front image corresponding to the depth range determined based on the retinal layer.
　実施態様１７は、実施態様１６に係る医用画像処理装置を含み、前記生成部は、前記三次元の断層画像に対応する三次元のモーションコントラストデータを用いて、前記決定された深度範囲に対応するモーションコントラスト正面画像を生成する。 Embodiment 17 includes the medical image processing apparatus according to Embodiment 16, wherein the generating unit corresponds to the determined depth range using three-dimensional motion contrast data corresponding to the three-dimensional tomographic image. Generate a motion contrast front image.
　実施態様１８は、実施態様１乃至１５のいずれか一つに係る医用画像処理装置を含み、高画質化用の学習済モデルを用いて、前記取得された断層画像から、前記取得された断層画像と比べて高画質化された断層画像を生成する生成部を更に備え、前記第一の処理部は、前記生成された断層画像に前記第一の検出処理を実行する。 The eighteenth embodiment includes the medical image processing apparatus according to any one of the first to fifteenth embodiments, and uses the learned model for improving the image quality to convert the acquired tomographic image from the acquired tomographic image. The image processing apparatus further includes a generation unit that generates a tomographic image having a higher image quality than that of the first tomographic image. The first processing unit performs the first detection process on the generated tomographic image.
　実施態様１９は、実施態様１乃至１８のいずれか一つに係る医用画像処理装置を含み、操作者の指示に応じて、前記第一の処理部が検出した網膜層の情報を修正する修正部を更に備え、前記修正された網膜層の情報は、前記第一の処理部が用いる前記学習済モデルについての追加学習に用いられる。 A nineteenth embodiment includes the medical image processing device according to any one of the first to eighteenth embodiments, and corrects a retinal layer information detected by the first processing unit in accordance with an instruction of an operator. The modified retinal layer information is used for additional learning on the learned model used by the first processing unit.
　実施態様２０は、実施態様１乃至１８のいずれか一つに係る医用画像処理装置を含み、診断結果生成用の学習済モデルを用いて、前記第一の検出処理を実行して得た結果から、前記取得された断層画像の診断結果を生成する、診断結果生成部を更に備える。 The twentieth embodiment includes the medical image processing apparatus according to any one of the first to eighteenth embodiments, and uses a learned model for generating a diagnosis result to execute the first detection process and obtain a result. And a diagnostic result generation unit that generates a diagnostic result of the acquired tomographic image.
　実施態様２１は、医用画像処理方法に関する。該医用画像処理方法は、被検眼の断層画像を取得する工程と、学習済モデルを用いて、前記断層画像において被検眼の複数の網膜層のうち少なくとも一つの網膜層を検出するための第一の検出処理を実行する工程とを含む。 Embodiment 21 relates to a medical image processing method. The medical image processing method includes a step of acquiring a tomographic image of the eye to be inspected, and a first step of detecting at least one retinal layer among a plurality of retinal layers of the eye to be inspected in the tomographic image using the learned model. And performing a detection process.
　実施態様２２は、プログラムに関する。該プログラムは、プロセッサーによって実行されると、該プロセッサーに実施態様２１に係る医用画像処理方法の各工程を実行させる。 {Embodiment 22} relates to a program. The program, when executed by the processor, causes the processor to execute the steps of the medical image processing method according to the twenty-first embodiment.
　本開示の更なる実施態様１は、医用画像処理装置に関する。該医用画像処理装置は、学習済モデルを含むセグメンテーションエンジンを用いて、被検者の所定部位の断層画像である入力画像から解剖学的な領域を識別可能な領域情報を生成するセグメンテーション処理部と、学習済モデルを含む評価エンジン又は解剖学的な知識を用いた知識ベース処理を行う評価エンジンを用いて前記領域情報を評価する評価部とを備える。 更 Further embodiment 1 of the present disclosure relates to a medical image processing apparatus. The medical image processing apparatus includes: a segmentation processing unit configured to generate region information capable of identifying an anatomical region from an input image that is a tomographic image of a predetermined part of a subject using a segmentation engine including a learned model; and And an evaluation unit that evaluates the region information by using an evaluation engine that includes a learned model or an evaluation engine that performs a knowledge base process using anatomical knowledge.
　更なる実施態様２は、更なる実施態様１に係る医用画像処理装置を含み、前記入力画像の撮影条件を取得する撮影条件取得部を更に備え、前記セグメンテーション処理部は、前記撮影条件に基づいて、それぞれ異なる学習済モデルを含む複数のセグメンテーションエンジンを切り替えて用いる。 A further embodiment 2 includes the medical image processing apparatus according to the further embodiment 1, further comprising a photographing condition acquiring unit for acquiring photographing conditions of the input image, wherein the segmentation processing unit is configured to perform the operation based on the photographing conditions. In addition, a plurality of segmentation engines including different learned models are switched and used.
　更なる実施態様３は、更なる実施態様２に係る医用画像処理装置を含み、前記撮影条件取得部が、学習済モデルを含む撮影箇所推定エンジンを用いて、前記入力画像から撮影部位及び撮影領域の少なくとも一方を推定する。 A further embodiment 3 includes the medical image processing apparatus according to the further embodiment 2, wherein the imaging condition acquisition unit uses an imaging location estimation engine including a learned model to acquire an imaging region and an imaging region from the input image. Is estimated.
　更なる実施態様４は、更なる実施態様１乃至３のいずれか一つに係る医用画像処理装置を含み、前記セグメンテーション処理部は、前記入力画像の画像サイズを、前記セグメンテーションエンジンが対処可能な画像サイズに調整してセグメンテーションエンジンに入力する。 A further embodiment 4 includes the medical image processing apparatus according to any one of the further embodiments 1 to 3, wherein the segmentation processing unit converts the image size of the input image into an image which can be handled by the segmentation engine. Adjust the size and input to the segmentation engine.
　更なる実施態様５は、更なる実施態様１乃至３のいずれか一つに係る医用画像処理装置を含み、前記セグメンテーション処理部は、前記入力画像の画像サイズが、前記セグメンテーションエンジンによって対処可能な画像サイズとなるように、前記入力画像にパディングを行った画像をセグメンテーションエンジンに入力する。 A further embodiment 5 includes the medical image processing apparatus according to any one of the further embodiments 1 to 3, wherein the segmentation processing unit is configured to determine whether the image size of the input image can be handled by the segmentation engine. An image obtained by padding the input image so as to have a size is input to a segmentation engine.
　更なる実施態様６は、更なる実施態様１乃至３のいずれか一つに係る医用画像処理装置を含み、前記セグメンテーション処理部は、前記入力画像を複数の領域の画像に分割し、分割された領域の画像毎に前記セグメンテーションエンジンに入力する。 A further embodiment 6 includes the medical image processing apparatus according to any one of the further embodiments 1 to 3, wherein the segmentation processing unit divides the input image into images of a plurality of regions, and Each segment image is input to the segmentation engine.
　更なる実施態様７は、更なる実施態様１乃至６のいずれか一つに係る医用画像処理装置を含み、前記評価部は、前記評価の結果に応じて前記領域情報を出力するか否かを判断する。 A further embodiment 7 includes the medical image processing apparatus according to any one of the further embodiments 1 to 6, wherein the evaluation unit determines whether or not to output the area information according to a result of the evaluation. to decide.
　更なる実施態様８は、更なる実施態様１乃至７のいずれか一つに係る医用画像処理装置を含み、前記セグメンテーション処理部は、それぞれ異なる学習済モデルを含む複数のセグメンテーションエンジンを用いて、前記入力画像から前記複数の領域情報を生成し、前記評価部は、ユーザーの指示に応じて、前記複数の領域情報を評価して出力すべきと判断した複数の領域情報のうちの少なくとも１つを選択する。 A further embodiment 8 includes the medical image processing apparatus according to any one of the further embodiments 1 to 7, wherein the segmentation processing unit uses a plurality of segmentation engines each including a different learned model. The plurality of area information is generated from an input image, and the evaluation unit determines at least one of the plurality of area information determined to be output by evaluating the plurality of area information in accordance with a user's instruction. select.
　更なる実施態様９は、更なる実施態様１乃至７のいずれか一つに係る医用画像処理装置を含み、前記セグメンテーション処理部は、それぞれ異なる学習済モデルを含む複数のセグメンテーションエンジンを用いて、前記入力画像から前記複数の領域情報を生成し、前記評価部は、所定の選択基準に基づいて、前記複数の領域情報を評価して出力すべきと判断した複数の領域情報のうちの少なくとも１つを選択する。 A further embodiment 9 includes the medical image processing device according to any one of the further embodiments 1 to 7, wherein the segmentation processing unit uses a plurality of segmentation engines each including a different learned model, and The plurality of area information is generated from an input image, and the evaluation unit evaluates the plurality of area information based on a predetermined selection criterion, and determines at least one of the plurality of area information determined to be output. Select
　更なる実施態様１０は、更なる実施態様１乃至９のいずれか一つに係る医用画像処理装置を含み、前記セグメンテーションエンジンを用いて前記入力画像から前記領域情報を生成可能であるか否か判定する判定部を更に備える。 A further embodiment 10 includes the medical image processing device according to any one of the further embodiments 1 to 9, and determines whether or not the area information can be generated from the input image using the segmentation engine. And a determining unit for performing the determination.
　更なる実施態様１１は、更なる実施態様１乃至１０のいずれか一つに係る医用画像処理装置を含み、前記セグメンテーション処理部は、前記入力画像を該入力画像の次元よりも低い次元の複数の画像に分割し、分割した画像毎に前記セグメンテーションエンジンに入力する。 A further embodiment 11 includes the medical image processing apparatus according to any one of the further embodiments 1 to 10, wherein the segmentation processing unit converts the input image into a plurality of images having a dimension lower than the dimension of the input image. The image is divided into images, and the divided images are input to the segmentation engine.
　更なる実施態様１２は、更なる実施態様１１に係る医用画像処理装置を含み、前記セグメンテーション処理部は、複数のセグメンテーションエンジンを用いて前記複数の画像を並列的に処理する。 A further embodiment 12 includes the medical image processing apparatus according to the further embodiment 11, wherein the segmentation processing unit processes the plurality of images in parallel using a plurality of segmentation engines.
　更なる実施態様１３は、更なる実施態様１乃至１２のいずれか一つに係る医用画像処理装置を含み、前記領域情報は、画素毎に領域のラベルが付されたラベル画像である。 A further embodiment 13 includes the medical image processing apparatus according to any one of the further embodiments 1 to 12, wherein the area information is a label image in which an area label is assigned to each pixel.
　更なる実施態様１４は、更なる実施態様１３に係る医用画像処理装置を含み、前記セグメンテーションエンジンは、断層画像を入力とし、前記ラベル画像を出力とする。 A further embodiment 14 includes the medical image processing apparatus according to the further embodiment 13, wherein the segmentation engine inputs a tomographic image and outputs the label image.
　更なる実施態様１５は、更なる実施態様１４に係る医用画像処理装置を含み、前記セグメンテーションエンジンの学習済モデルは、２つ以上の層を含む断層画像を入力データとし、該断層画像に対応するラベル画像を出力データとして学習を行ったモデルである。 A further embodiment 15 includes the medical image processing apparatus according to the further embodiment 14, wherein the trained model of the segmentation engine receives a tomographic image including two or more layers as input data and corresponds to the tomographic image. This is a model in which learning is performed using a label image as output data.
　更なる実施態様１６は、更なる実施態様１乃至１５のいずれか一つに係る医用画像処理装置を含み、前記入力画像を撮影装置から取得する、又は該撮影装置から前記被検者の前記所定部位のデータを取得し、前記データに基づく前記入力画像を取得する。 A further embodiment 16 includes the medical image processing device according to any one of the further embodiments 1 to 15, wherein the input image is obtained from an imaging device, or the predetermined image of the subject is obtained from the imaging device. Obtaining part data and obtaining the input image based on the data.
　更なる実施態様１７は、更なる実施態様１乃至１５のいずれか一つに係る医用画像処理装置を含み、前記入力画像を画像管理システムから取得する、前記領域情報を該画像管理システムに出力する、又は、該入力画像を該画像管理システムから取得し、且つ該領域情報を該画像管理システムに出力する。 A further embodiment 17 includes the medical image processing apparatus according to any one of the further embodiments 1 to 15, wherein the input image is obtained from an image management system, and the region information is output to the image management system. Or, the input image is obtained from the image management system, and the area information is output to the image management system.
　更なる実施態様１８は、更なる実施態様１乃至１７のいずれか一つに係る医用画像処理装置を含み、前記領域情報を解剖学的な知識ベース処理によって修正する修正部を更に備える。 A further embodiment 18 includes the medical image processing apparatus according to any one of the further embodiments 1 to 17, and further includes a correction unit that corrects the region information by anatomical knowledge-based processing.
　更なる実施態様１９は、更なる実施態様１乃至１８のいずれか一つに係る医用画像処理装置を含み、前記評価部から出力された前記領域情報を用いて、前記入力画像の画像解析を行う解析部を更に備える。 A further embodiment 19 includes the medical image processing apparatus according to any one of the further embodiments 1 to 18, and performs image analysis of the input image using the area information output from the evaluation unit. An analysis unit is further provided.
　更なる実施態様２０は、更なる実施態様１乃至１９のいずれか一つに係る医用画像処理装置を含み、前記領域情報は学習済モデルを用いて生成された情報であることを出力する。 A further embodiment 20 includes the medical image processing apparatus according to any one of the further embodiments 1 to 19, and outputs that the area information is information generated using a learned model.
　更なる実施態様２１は、医用画像処理方法に係る。該医用画像処理方法は、学習済モデルを含むセグメンテーションエンジンを用いて、被検者の所定部位の断層画像である入力画像から解剖学的な領域を識別可能な領域情報を生成することと、学習済モデルを含む評価エンジン又は解剖学的な知識を用いた知識ベースの評価エンジンを用いて前記領域情報を評価することとを含む。 A further embodiment 21 relates to a medical image processing method. The medical image processing method includes using a segmentation engine including a learned model to generate region information capable of identifying an anatomical region from an input image that is a tomographic image of a predetermined part of a subject; Evaluating the region information using an evaluation engine that includes a completed model or a knowledge-based evaluation engine that uses anatomical knowledge.
　更なる実施態様２２は、プログラムに係る。該プログラムは、プロセッサーによって実行されると、該プロセッサーに更なる実施態様２１に係る医用画像処理方法の各工程を実行させる。 A further embodiment 22 relates to a program. The program, when executed by the processor, causes the processor to execute the steps of the medical image processing method according to the further embodiment 21.
（その他の実施例）
　本発明は、上述の実施例及び変形例の１以上の機能を実現するプログラムを、ネットワーク又は記憶媒体を介してシステム又は装置に供給し、そのシステム又は装置のコンピュータがプログラムを読出し実行する処理でも実現可能である。コンピュータは、１つ又は複数のプロセッサー若しくは回路を有し、コンピュータ実行可能命令を読み出し実行するために、分離した複数のコンピュータ又は分離した複数のプロセッサー若しくは回路のネットワークを含みうる。 (Other Examples)
The present invention provides a program for realizing one or more functions of the above-described embodiments and modifications to a system or an apparatus via a network or a storage medium, and a computer of the system or the apparatus reads and executes the program. It is feasible. A computer has one or more processors or circuits, and can include separate computers or a network of separate processors or circuits, to read and execute computer-executable instructions.
　プロセッサー又は回路は、中央演算処理装置（ＣＰＵ）、マイクロプロセッシングユニット（ＭＰＵ）、グラフィクスプロセッシングユニット（ＧＰＵ）、特定用途向け集積回路（ＡＳＩＣ）、又はフィールドプログラマブルゲートウェイ（ＦＰＧＡ）を含みうる。また、プロセッサー又は回路は、デジタルシグナルプロセッサ（ＤＳＰ）、データフロープロセッサ（ＤＦＰ）、又はニューラルプロセッシングユニット（ＮＰＵ）を含みうる。 A processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).
　本発明は上記実施例に制限されるものではなく、本発明の精神及び範囲から離脱することなく、様々な変更及び変形が可能である。従って、本発明の範囲を公にするために以下の請求項を添付する。 The present invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to make the scope of the present invention public.
　本願は、２０１８年８月１４日提出の日本国特許出願特願２０１８－１５２６３２、２０１８年１２月１０日提出の日本国特許出願特願２０１８－２３０６１２、及び２０１９年８月９日提出の日本国特許出願特願２０１９－１４７７３９を基礎として優先権を主張するものであり、その記載内容の全てをここに援用する。 This application is based on Japanese Patent Application No. 2018-152632 filed on August 14, 2018, Japanese Patent Application No. 2018-230612 filed on December 10, 2018, and Japanese Patent Application No. 2018-230612 filed on August 9, 2019. The priority is claimed based on Japanese Patent Application No. 2019-147739, the entire contents of which are incorporated herein by reference.
２０：画像処理装置、２１：取得部、２２２：処理部（第一の処理部） 20: image processing device, 21: acquisition unit, 222: processing unit (first processing unit)

Claims (44)

  1. 　被検眼の断層画像を取得する取得部と、
    　被検眼の断層画像において複数の網膜層のうち少なくとも一つの網膜層が示されたデータを学習して得た学習済モデルを用いて、前記取得された断層画像において複数の網膜層のうち少なくとも一つの網膜層を検出するための第一の検出処理を実行する第一の処理部と、
    を備える、医用画像処理装置。     An acquisition unit that acquires a tomographic image of the eye to be inspected,
    At least one of the plurality of retinal layers in the acquired tomographic image is obtained using a learned model obtained by learning data indicating at least one of the plurality of retinal layers in the tomographic image of the subject's eye. A first processing unit that executes a first detection process for detecting one retinal layer,
    A medical image processing apparatus comprising:
  2. 　機械学習により得られる学習済モデルを用いずに、前記取得された断層画像において前記複数の網膜層のうち少なくとも一つの網膜層を検出するための第二の検出処理を実行する第二の処理部を更に備える、請求項１に記載の医用画像処理装置。 A second processing unit that executes a second detection process for detecting at least one retinal layer of the plurality of retinal layers in the acquired tomographic image without using a learned model obtained by machine learning. The medical image processing apparatus according to claim 1, further comprising:
  3. 　前記第二の検出処理は、前記第一の検出処理を実行することにより検出された少なくとも一つの網膜層以外の少なくとも一つの網膜層を検出する処理である、請求項２に記載の医用画像処理装置。 The medical image processing according to claim 2, wherein the second detection process is a process of detecting at least one retinal layer other than the at least one retinal layer detected by performing the first detection process. apparatus.
  4. 　前記第一の検出処理は、前記少なくとも一つの網膜層として、前記取得された断層画像において網膜領域を検出する処理であり、
    　前記第二の検出処理は、前記第一の検出処理を実行することにより検出された網膜領域における少なくとも一つの網膜層を検出する処理である、請求項２又は３に記載の医用画像処理装置。     The first detection process, as the at least one retinal layer, is a process of detecting a retinal region in the obtained tomographic image,
    The medical image processing apparatus according to claim 2, wherein the second detection processing is processing for detecting at least one retinal layer in a retinal region detected by executing the first detection processing.
  5. 　前記第一の検出処理は、被検眼の内境界膜と神経線維層との境界から視細胞内節外節接合部、網膜色素上皮層、及びブルッフ膜のいずれかまでの層を検出する処理であり、
    　前記第二の検出処理は、前記第一の検出処理により検出した層の間の少なくとも一つの網膜層を検出する処理である、請求項２乃至４のいずれか一項に記載の医用画像処理装置。     The first detection process is a process of detecting a layer from the boundary between the inner limiting membrane of the eye to be examined and the nerve fiber layer to the junction between the photoreceptor inner and outer segments, the retinal pigment epithelium layer, and the Bruch's membrane. Yes,
    The medical image processing apparatus according to any one of claims 2 to 4, wherein the second detection processing is processing for detecting at least one retinal layer between layers detected by the first detection processing. .
  6. 　前記第二の処理部は、前記第一の処理部による前記第一の検出処理の後に、前記第二の検出処理を実行する、請求項２乃至５のいずれか一項に記載の医用画像処理装置。 The medical image processing according to any one of claims 2 to 5, wherein the second processing unit executes the second detection process after the first detection process by the first processing unit. apparatus.
  7. 　表示部を制御する表示制御部を更に備え、
    　前記第一の検出処理及び前記第二の検出処理は、同一の網膜層を検出する処理であり、
    　前記表示制御部は、前記第一の検出処理及び前記第二の検出処理の処理結果を前記表示部に表示させる、請求項２に記載の医用画像処理装置。     A display control unit that controls the display unit,
    The first detection process and the second detection process are processes for detecting the same retinal layer,
    The medical image processing device according to claim 2, wherein the display control unit causes the display unit to display processing results of the first detection processing and the second detection processing.
  8. 　前記表示制御部は、前記第一の検出処理及び前記第二の検出処理の処理結果の不一致部分を前記表示部に表示させる、請求項７に記載の医用画像処理装置。 8. The medical image processing apparatus according to claim 7, wherein the display control unit causes the display unit to display a portion where the processing results of the first detection process and the second detection process do not match each other.
  9. 　前記第一の検出処理及び前記第二の検出処理は、被検眼の内境界膜と神経線維層との境界から視細胞内節外節接合部、網膜色素上皮層、及びブルッフ膜のいずれかまでの層を検出する処理であり、
    　前記第二の処理部は、操作者の指示に応じて、前記第一の検出処理及び前記第二の検出処理のいずれか一方により検出した層の間の少なくとも一つの網膜層を検出する第三の検出処理を更に実行する、請求項７又は８に記載の医用画像処理装置。     The first detection process and the second detection process, from the boundary between the inner limiting membrane and the nerve fiber layer of the eye to be examined from the junction of the photoreceptor inner and outer node, the retinal pigment epithelium layer, and any of Bruch's membrane Is a process of detecting the layer of
    The second processing unit is configured to detect at least one retinal layer between layers detected by one of the first detection process and the second detection process in accordance with an instruction of an operator. The medical image processing apparatus according to claim 7, further comprising performing a detection process.
  10. 　前記取得された断層画像に関する撮影条件に基づいて、前記第一の検出処理と前記第二の検出処理のうち少なくとも一つの選択を行う選択部を更に備える、請求項２乃至９のいずれか一項に記載の医用画像処理装置。 10. The apparatus according to claim 2, further comprising: a selection unit configured to select at least one of the first detection process and the second detection process based on an imaging condition regarding the acquired tomographic image. 11. 3. The medical image processing apparatus according to claim 1.
  11. 　前記第一の処理部は、異なる学習データを用いて機械学習が行われた複数の学習済モデルのうち、前記取得された断層画像に関する撮影条件に対応する学習データを用いて機械学習が行われた学習済モデルを用いて、前記第一の検出処理を実行する、請求項２乃至１０のいずれか一項に記載の医用画像処理装置。 The first processing unit is configured to perform machine learning using learning data corresponding to an imaging condition related to the acquired tomographic image, among a plurality of learned models that have been subjected to machine learning using different learning data. The medical image processing device according to claim 2, wherein the first detection process is performed using the learned model.
  12. 　前記撮影条件は、撮影部位、撮影方式、撮影領域、撮影画角、及び画像の解像度のうち少なくとも一つを含む、請求項１０又は１１に記載の医用画像処理装置。 The medical image processing apparatus according to claim 10, wherein the imaging condition includes at least one of an imaging region, an imaging method, an imaging region, an imaging angle of view, and an image resolution.
  13. 　前記第一の検出処理及び前記第二の検出処理の結果に基づいて、被検眼の形状特徴が計測される、請求項２乃至１２のいずれか一項に記載の医用画像処理装置。 The medical image processing apparatus according to any one of claims 2 to 12, wherein a shape characteristic of the eye to be inspected is measured based on a result of the first detection processing and the result of the second detection processing.
  14. 　網膜層における医学的特徴に基づいて、前記第一の処理部が検出した網膜層の構造を補正する補正部を更に備える、請求項１乃至１３のいずれか一項に記載の医用画像処理装置。 14. The medical image processing apparatus according to claim 1, further comprising a correction unit configured to correct a structure of the retinal layer detected by the first processing unit based on a medical feature in the retinal layer.
  15. 　前記第一の処理部は、前記学習済モデルを用いて、入力された画像について撮影部位毎に予め定められた境界を検出する、請求項１乃至１４のいずれか一項に記載の医用画像処理装置。 The medical image processing according to any one of claims 1 to 14, wherein the first processing unit uses the learned model to detect a boundary predetermined for each imaging region in the input image. apparatus.
  16. 　被検眼の三次元の断層画像における少なくとも一部の深度範囲であって、前記検出された少なくとも一つの網膜層に基づいて決定された深度範囲に対応する正面画像を生成する生成部を更に備える、請求項１乃至１５のいずれか一項に記載の医用画像処理装置。 The apparatus further includes a generation unit that generates a front image corresponding to a depth range determined based on the at least one detected retinal layer, which is at least a part of a depth range in a three-dimensional tomographic image of the subject's eye, The medical image processing apparatus according to claim 1.
  17. 　前記生成部は、前記三次元の断層画像に対応する三次元のモーションコントラストデータを用いて、前記決定された深度範囲に対応するモーションコントラスト正面画像を生成する、請求項１６に記載の医用画像処理装置。 The medical image processing according to claim 16, wherein the generation unit generates a motion contrast front image corresponding to the determined depth range using three-dimensional motion contrast data corresponding to the three-dimensional tomographic image. apparatus.
  18. 　高画質化用の学習済モデルを用いて、前記取得された断層画像から、前記取得された断層画像と比べて高画質化された断層画像を生成する生成部を更に備え、
    　前記第一の処理部は、前記生成された断層画像に前記第一の検出処理を実行する、請求項１乃至１５のいずれか一項に記載の医用画像処理装置。     Using a trained model for high image quality, from the acquired tomographic image, further comprising a generating unit that generates a high-quality tomographic image compared to the acquired tomographic image,
    The medical image processing apparatus according to claim 1, wherein the first processing unit performs the first detection process on the generated tomographic image.
  19. 　操作者の指示に応じて、前記第一の処理部が検出した網膜層の情報を修正する修正部を更に備え、
    　前記修正された網膜層の情報は、前記第一の処理部が用いる前記学習済モデルについての追加学習に用いられる、請求項１乃至１８のいずれか一項に記載の医用画像処理装置。     According to the instruction of the operator, further comprising a correction unit for correcting the information of the retinal layer detected by the first processing unit,
    19. The medical image processing apparatus according to claim 1, wherein the corrected information on the retinal layer is used for additional learning on the learned model used by the first processing unit.
  20. 　診断結果生成用の学習済モデルを用いて、前記第一の検出処理を実行して得た結果から、前記取得された断層画像の診断結果を生成する、診断結果生成部を更に備える、請求項１乃至１９のいずれか一項に記載の医用画像処理装置。 The diagnosis result generation unit that generates a diagnosis result of the acquired tomographic image from a result obtained by executing the first detection process using a learned model for generating a diagnosis result, further comprising: 20. The medical image processing device according to any one of 1 to 19.
  21. 　被検眼の断層画像を取得する工程と、
    　被検眼の断層画像において複数の網膜層のうち少なくとも一つの網膜層が示されたデータを学習して得た学習済モデルを用いて、前記取得された断層画像において複数の網膜層のうち少なくとも一つの網膜層を検出するための第一の検出処理を実行する工程と、
    を含む、医用画像処理方法。     A step of acquiring a tomographic image of the subject's eye;
    At least one of the plurality of retinal layers in the acquired tomographic image is obtained using a learned model obtained by learning data indicating at least one of the plurality of retinal layers in the tomographic image of the subject's eye. Performing a first detection process for detecting one retinal layer;
    A medical image processing method comprising:
  22. 　プロセッサーによって実行されると、該プロセッサーに請求項２１に記載の医用画像処理方法の各工程を実行させる、プログラム。 A program that, when executed by a processor, causes the processor to execute the steps of the medical image processing method according to claim 21.
  23. 　学習済モデルを含むセグメンテーションエンジンを用いて、被検者の所定部位の断層画像である入力画像から解剖学的な領域を識別可能な領域情報を生成するセグメンテーション処理部と、
    　学習済モデルを含む評価エンジン又は解剖学的な知識を用いた知識ベース処理を行う評価エンジンを用いて前記領域情報を評価する評価部と、
    を備える、医用画像処理装置。     Using a segmentation engine including a trained model, a segmentation processing unit that generates region information capable of identifying an anatomical region from an input image that is a tomographic image of a predetermined part of the subject,
    An evaluation unit that evaluates the region information using an evaluation engine that includes a learned model or an evaluation engine that performs a knowledge base process using anatomical knowledge,
    A medical image processing apparatus comprising:
  24. 　前記入力画像の撮影条件を取得する撮影条件取得部を更に備え、
    　前記セグメンテーション処理部は、前記撮影条件に基づいて、それぞれ異なる学習済モデルを含む複数のセグメンテーションエンジンを切り替えて用いる、請求項２３に記載の医用画像処理装置。     The image processing apparatus further includes a shooting condition acquisition unit that acquires a shooting condition of the input image,
    24. The medical image processing apparatus according to claim 23, wherein the segmentation processing unit switches between a plurality of segmentation engines including different learned models based on the imaging conditions.
  25. 　前記撮影条件取得部が、学習済モデルを含む撮影箇所推定エンジンを用いて、前記入力画像から撮影部位及び撮影領域の少なくとも一方を推定する、請求項２４に記載の医用画像処理装置。 25. The medical image processing apparatus according to claim 24, wherein the imaging condition acquisition unit estimates at least one of an imaging region and an imaging region from the input image using an imaging location estimation engine including a learned model.
  26. 　前記セグメンテーション処理部は、前記入力画像の画像サイズを、前記セグメンテーションエンジンが対処可能な画像サイズに調整してセグメンテーションエンジンに入力する、請求項２３乃至２５のいずれか一項に記載の医用画像処理装置。 26. The medical image processing apparatus according to claim 23, wherein the segmentation processing unit adjusts an image size of the input image to an image size that can be handled by the segmentation engine and inputs the image size to the segmentation engine. .
  27. 　前記セグメンテーション処理部は、前記入力画像の画像サイズが、前記セグメンテーションエンジンによって対処可能な画像サイズとなるように、前記入力画像にパディングを行った画像をセグメンテーションエンジンに入力する、請求項２３乃至２５のいずれか一項に記載の医用画像処理装置。 26. The segmentation processing unit according to claim 23, wherein the segmentation processing unit inputs an image obtained by padding the input image to the segmentation engine such that an image size of the input image is an image size that can be handled by the segmentation engine. The medical image processing device according to claim 1.
  28. 　前記セグメンテーション処理部は、前記入力画像を複数の領域の画像に分割し、分割された領域の画像毎に前記セグメンテーションエンジンに入力する、請求項２３乃至２５のいずれか一項に記載の医用画像処理装置。 The medical image processing according to any one of claims 23 to 25, wherein the segmentation processing unit divides the input image into images of a plurality of regions and inputs the divided images to the segmentation engine for each of the images of the divided regions. apparatus.
  29. 　前記評価部は、前記評価の結果に応じて前記領域情報を出力するか否かを判断する、請求項２３乃至２８のいずれか一項に記載の医用画像処理装置。 29. The medical image processing apparatus according to claim 23, wherein the evaluation unit determines whether to output the region information according to a result of the evaluation.
  30. 　前記セグメンテーション処理部は、それぞれ異なる学習済モデルを含む複数のセグメンテーションエンジンを用いて、前記入力画像から前記複数の領域情報を生成し、
    　前記評価部は、ユーザーの指示に応じて、前記複数の領域情報を評価して出力すべきと判断した複数の領域情報のうちの少なくとも１つを選択する、請求項２３乃至２９のいずれか一項に記載の医用画像処理装置。     The segmentation processing unit uses a plurality of segmentation engines each including a different learned model, to generate the plurality of region information from the input image,
    30. The evaluation unit according to claim 23, wherein the evaluation unit selects at least one of the plurality of pieces of area information determined to be evaluated and output according to a user's instruction. 13. The medical image processing apparatus according to item 9.
  31. 　前記セグメンテーション処理部は、それぞれ異なる学習済モデルを含む複数のセグメンテーションエンジンを用いて、前記入力画像から前記複数の領域情報を生成し、
    　前記評価部は、所定の選択基準に基づいて、前記複数の領域情報を評価して出力すべきと判断した複数の領域情報のうちの少なくとも１つを選択する、請求項２３乃至２９のいずれか一項に記載の医用画像処理装置。     The segmentation processing unit uses a plurality of segmentation engines each including a different learned model, to generate the plurality of region information from the input image,
    30. The evaluation unit according to claim 23, wherein the evaluation unit selects at least one of the plurality of pieces of area information determined to be evaluated and output based on a predetermined selection criterion. The medical image processing device according to claim 1.
  32. 　前記セグメンテーションエンジンを用いて前記入力画像から前記領域情報を生成可能であるか否か判定する判定部を更に備える、請求項２３乃至３１いずれか一項に記載の医用画像処理装置。 32. The medical image processing apparatus according to claim 23, further comprising: a determination unit configured to determine whether the area information can be generated from the input image using the segmentation engine.
  33. 　前記セグメンテーション処理部は、前記入力画像を該入力画像の次元よりも低い次元の複数の画像に分割し、分割した画像毎に前記セグメンテーションエンジンに入力する、請求項２３乃至３２のいずれか一項に記載の医用画像処理装置。 The segmentation processing unit according to any one of claims 23 to 32, wherein the input image is divided into a plurality of images having dimensions lower than the dimension of the input image, and the divided images are input to the segmentation engine. The medical image processing apparatus according to the above.
  34. 　前記セグメンテーション処理部は、複数のセグメンテーションエンジンを用いて前記複数の画像を並列的に処理する、請求項３３に記載の医用画像処理装置。 34. The medical image processing apparatus according to claim 33, wherein the segmentation processing unit processes the plurality of images in parallel using a plurality of segmentation engines.
  35. 　前記領域情報は、画素毎に領域のラベルが付されたラベル画像である、請求項２３乃至３４のいずれか一項に記載の医用画像処理装置。 35. The medical image processing apparatus according to claim 23, wherein the area information is a label image in which an area label is assigned to each pixel.
  36. 　前記セグメンテーションエンジンは、断層画像を入力とし、前記ラベル画像を出力とする請求項３５に記載の医用画像処理装置。 36. The medical image processing apparatus according to claim 35, wherein the segmentation engine receives a tomographic image as input and outputs the label image.
  37. 　前記セグメンテーションエンジンの学習済モデルは、２つ以上の層を含む断層画像を入力データとし、該断層画像に対応するラベル画像を出力データとして学習を行ったモデルである、請求項３６に記載の医用画像処理装置。 37. The medical model according to claim 36, wherein the trained model of the segmentation engine is a model obtained by learning a tomographic image including two or more layers as input data, and using a label image corresponding to the tomographic image as output data. Image processing device.
  38. 　前記入力画像を撮影装置から取得する、又は該撮影装置から前記被検者の前記所定部位のデータを取得し、前記データに基づく前記入力画像を取得する、請求項２３乃至３７のいずれか一項に記載の医用画像処理装置。 38. The input device according to claim 23, wherein the input image is obtained from an imaging device, or data of the predetermined part of the subject is obtained from the imaging device, and the input image is obtained based on the data. 4. The medical image processing apparatus according to claim 1.
  39. 　前記入力画像を画像管理システムから取得する、前記領域情報を該画像管理システムに出力する、又は、該入力画像を該画像管理システムから取得し、且つ該領域情報を該画像管理システムに出力する、請求項２３乃至３７のいずれか一項に記載の医用画像処理装置。 Acquiring the input image from an image management system, outputting the area information to the image management system, or acquiring the input image from the image management system, and outputting the area information to the image management system, The medical image processing device according to any one of claims 23 to 37.
  40. 　前記領域情報を解剖学的な知識ベース処理によって修正する修正部を更に備える、請求項２３乃至３９のいずれか一項に記載の医用画像処理装置。 The medical image processing apparatus according to any one of claims 23 to 39, further comprising a correction unit configured to correct the region information by anatomical knowledge-based processing.
  41. 　前記評価部から出力された前記領域情報を用いて、前記入力画像の画像解析を行う解析部を更に備える、請求項２３乃至４０のいずれか一項に記載の医用画像処理装置。 41. The medical image processing apparatus according to any one of claims 23 to 40, further comprising: an analysis unit configured to perform an image analysis of the input image using the area information output from the evaluation unit.
  42. 　前記領域情報は学習済モデルを用いて生成された情報であることを出力する、請求項２３乃至４１のいずれか一項に記載の医用画像処理装置。 42. The medical image processing apparatus according to claim 23, wherein the region information is output as information generated using a learned model.
  43. 　学習済モデルを含むセグメンテーションエンジンを用いて、被検者の所定部位の断層画像である入力画像から解剖学的な領域を識別可能な領域情報を生成することと、
    　学習済モデルを含む評価エンジン又は解剖学的な知識を用いた知識ベースの評価エンジンを用いて前記領域情報を評価することと、
    を含む、医用画像処理方法。     Using a segmentation engine including the learned model, to generate region information capable of identifying an anatomical region from an input image that is a tomographic image of a predetermined part of the subject,
    Evaluating the area information using an evaluation engine including a learned model or a knowledge-based evaluation engine using anatomical knowledge,
    A medical image processing method comprising:
  44. 　プロセッサーによって実行されると、該プロセッサーに請求項４３に記載の医用画像処理方法の各工程を実行させる、プログラム。 A program that, when executed by a processor, causes the processor to execute the steps of the medical image processing method according to claim 43.
PCT/JP2019/031883 2018-08-14 2019-08-13 Medical image processing device, medical image processing method, and program WO2020036182A1 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
KR1020217006768A KR102543875B1 (en) 2018-08-14 2019-08-13 Medical image processing apparatus, medical image processing method, computer readable medium, and trained model
DE112019004104.5T DE112019004104T5 (en) 2018-08-14 2019-08-13 MEDICAL IMAGE PROCESSING DEVICE, MEDICAL IMAGE PROCESSING METHOD AND PROGRAM
GB2103260.2A GB2591890B (en) 2018-08-14 2019-08-13 Medical image processing apparatus, medical image processing method, program, and learned model
CN201980053915.XA CN112601487A (en) 2018-08-14 2019-08-13 Medical image processing apparatus, medical image processing method, and program
US17/168,776 US20210158525A1 (en) 2018-08-14 2021-02-05 Medical image processing apparatus, medical image processing method, computer-readable medium, and learned model

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
JP2018152632 2018-08-14
JP2018-152632 2018-08-14
JP2018-230612 2018-12-10
JP2018230612 2018-12-10
JP2019-147739 2019-08-09
JP2019147739A JP7229881B2 (en) 2018-08-14 2019-08-09 MEDICAL IMAGE PROCESSING APPARATUS, TRAINED MODEL, MEDICAL IMAGE PROCESSING METHOD AND PROGRAM

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/168,776 Continuation US20210158525A1 (en) 2018-08-14 2021-02-05 Medical image processing apparatus, medical image processing method, computer-readable medium, and learned model

Publications (1)

Publication Number Publication Date
WO2020036182A1 true WO2020036182A1 (en) 2020-02-20

Family

ID=69525427

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/031883 WO2020036182A1 (en) 2018-08-14 2019-08-13 Medical image processing device, medical image processing method, and program

Country Status (3)

Country Link
JP (1) JP7269413B2 (en)
GB (1) GB2614130B (en)
WO (1) WO2020036182A1 (en)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112419244A (en) * 2020-11-11 2021-02-26 浙江大学 Concrete crack segmentation method and device based on YOLOv4 target detection model and tubular flow field algorithm
CN112614568A (en) * 2020-12-28 2021-04-06 东软集团股份有限公司 Inspection image processing method and device, storage medium and electronic equipment
CN113554593A (en) * 2021-06-15 2021-10-26 汕头大学 Choroid thickness estimation method, apparatus, device and medium
WO2022056515A3 (en) * 2020-09-11 2022-04-21 Acucela Inc. Artificial intelligence for evaluation of optical coherence tomography images
US11357401B2 (en) 2018-06-20 2022-06-14 Acucela Inc. Miniaturized mobile, low cost optical coherence tomography system for home based ophthalmic applications
US11369266B2 (en) 2020-08-04 2022-06-28 Acucela Inc. Scan pattern and signal processing for optical coherence tomography
WO2022209574A1 (en) * 2021-03-31 2022-10-06 株式会社ニデック Medical image processing device, medical image processing program, and medical image processing method
US11497396B2 (en) 2021-03-24 2022-11-15 Acucela Inc. Axial length measurement monitor
US11627874B2 (en) 2016-12-21 2023-04-18 Acucela Inc. Miniaturized mobile, low cost optical coherence tomography system for home based ophthalmic applications
US11730363B2 (en) 2019-12-26 2023-08-22 Acucela Inc. Optical coherence tomography patient alignment system for home based ophthalmic applications
US11911105B2 (en) 2020-09-30 2024-02-27 Acucela Inc. Myopia prediction, diagnosis, planning, and monitoring device

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012019958A (en) * 2010-07-14 2012-02-02 Canon Inc Image processor, program, and storage medium
JP2017047111A (en) * 2015-09-04 2017-03-09 キヤノン株式会社 Ophthalmologic apparatus, display control method, and program
JP2018114068A (en) * 2017-01-17 2018-07-26 キヤノン株式会社 Image processing apparatus, optical coherence tomographic imaging apparatus, image processing method, and program

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2009139722A1 (en) 2008-05-14 2009-11-19 Agency For Science, Technology And Research Automatic cup-to-disc ratio measurement system
JP6240064B2 (en) * 2011-04-29 2017-11-29 ザ ジェネラル ホスピタル コーポレイション Method for determining depth-resolved physical and / or optical properties of a scattering medium
JP6489193B2 (en) 2017-11-02 2019-03-27 株式会社ニデック Ophthalmic analysis apparatus and ophthalmic analysis program
US10878574B2 (en) * 2018-02-21 2020-12-29 Topcon Corporation 3D quantitative analysis of retinal layers with deep learning

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012019958A (en) * 2010-07-14 2012-02-02 Canon Inc Image processor, program, and storage medium
JP2017047111A (en) * 2015-09-04 2017-03-09 キヤノン株式会社 Ophthalmologic apparatus, display control method, and program
JP2018114068A (en) * 2017-01-17 2018-07-26 キヤノン株式会社 Image processing apparatus, optical coherence tomographic imaging apparatus, image processing method, and program

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11890053B2 (en) 2016-12-21 2024-02-06 Acucela Inc. Miniaturized mobile, low cost optical coherence tomography system for home based ophthalmic applications
US11627874B2 (en) 2016-12-21 2023-04-18 Acucela Inc. Miniaturized mobile, low cost optical coherence tomography system for home based ophthalmic applications
US11576572B2 (en) 2018-06-20 2023-02-14 Acucela Inc. Miniaturized mobile, low cost optical coherence tomography system for home based ophthalmic applications
US11896308B2 (en) 2018-06-20 2024-02-13 Acucela Inc. Miniaturized mobile, low cost optical coherence tomography system for home based ophthalmic applications
US11357401B2 (en) 2018-06-20 2022-06-14 Acucela Inc. Miniaturized mobile, low cost optical coherence tomography system for home based ophthalmic applications
US11730363B2 (en) 2019-12-26 2023-08-22 Acucela Inc. Optical coherence tomography patient alignment system for home based ophthalmic applications
US11684254B2 (en) 2020-08-04 2023-06-27 Acucela Inc. Scan pattern and signal processing for optical coherence tomography
US11369266B2 (en) 2020-08-04 2022-06-28 Acucela Inc. Scan pattern and signal processing for optical coherence tomography
US11798164B2 (en) 2020-09-11 2023-10-24 Acucela Inc. Artificial intelligence for evaluation of optical coherence tomography images
US11393094B2 (en) 2020-09-11 2022-07-19 Acucela Inc. Artificial intelligence for evaluation of optical coherence tomography images
US11620749B2 (en) 2020-09-11 2023-04-04 Acucela Inc. Artificial intelligence for evaluation of optical coherence tomography images
WO2022056515A3 (en) * 2020-09-11 2022-04-21 Acucela Inc. Artificial intelligence for evaluation of optical coherence tomography images
US11911105B2 (en) 2020-09-30 2024-02-27 Acucela Inc. Myopia prediction, diagnosis, planning, and monitoring device
CN112419244A (en) * 2020-11-11 2021-02-26 浙江大学 Concrete crack segmentation method and device based on YOLOv4 target detection model and tubular flow field algorithm
CN112419244B (en) * 2020-11-11 2022-11-01 浙江大学 Concrete crack segmentation method and device
CN112614568A (en) * 2020-12-28 2021-04-06 东软集团股份有限公司 Inspection image processing method and device, storage medium and electronic equipment
US11497396B2 (en) 2021-03-24 2022-11-15 Acucela Inc. Axial length measurement monitor
US11779206B2 (en) 2021-03-24 2023-10-10 Acucela Inc. Axial length measurement monitor
WO2022209574A1 (en) * 2021-03-31 2022-10-06 株式会社ニデック Medical image processing device, medical image processing program, and medical image processing method
JP7439990B2 (en) 2021-03-31 2024-02-28 株式会社ニデック Medical image processing device, medical image processing program, and medical image processing method
CN113554593B (en) * 2021-06-15 2023-07-25 汕头大学 Choroid thickness estimation method, apparatus, equipment and medium
CN113554593A (en) * 2021-06-15 2021-10-26 汕头大学 Choroid thickness estimation method, apparatus, device and medium

Also Published As

Publication number Publication date
JP7269413B2 (en) 2023-05-08
JP2022103221A (en) 2022-07-07
GB2614130B (en) 2023-09-27
GB202217205D0 (en) 2023-01-04
GB2614130A (en) 2023-06-28

Similar Documents

Publication Publication Date Title
JP7229881B2 (en) MEDICAL IMAGE PROCESSING APPARATUS, TRAINED MODEL, MEDICAL IMAGE PROCESSING METHOD AND PROGRAM
JP7250653B2 (en) Image processing device, image processing method and program
US20210104313A1 (en) Medical image processing apparatus, medical image processing method and computer-readable medium
JP7269413B2 (en) MEDICAL IMAGE PROCESSING APPARATUS, MEDICAL IMAGE PROCESSING SYSTEM, MEDICAL IMAGE PROCESSING METHOD AND PROGRAM
JP7341874B2 (en) Image processing device, image processing method, and program
US20210390696A1 (en) Medical image processing apparatus, medical image processing method and computer-readable storage medium
US11922601B2 (en) Medical image processing apparatus, medical image processing method and computer-readable medium
KR20220051369A (en) Information processing apparatus, information processing method, information processing system and program
JP7413147B2 (en) Image processing device, image processing method, and program
US11887288B2 (en) Image processing apparatus, image processing method, and storage medium
JP7305401B2 (en) Image processing device, method of operating image processing device, and program
JP7374615B2 (en) Information processing device, information processing method and program
JP7362403B2 (en) Image processing device and image processing method
JP7332463B2 (en) Control device, optical coherence tomography device, control method for optical coherence tomography device, and program
WO2020138128A1 (en) Image processing device, image processing method, and program
WO2020075719A1 (en) Image processing device, image processing method, and program
JP2021164535A (en) Image processing device, image processing method and program
JP7344847B2 (en) Image processing device, image processing method, and program
WO2020049828A1 (en) Image processing apparatus, image processing method, and program
JP2022121202A (en) Image processing device and image processing method
JP2023010308A (en) Image processing device and image processing method
JP2021069667A (en) Image processing device, image processing method and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19849989

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 202103260

Country of ref document: GB

Kind code of ref document: A

Free format text: PCT FILING DATE = 20190813

122 Ep: pct application non-entry in european phase

Ref document number: 19849989

Country of ref document: EP

Kind code of ref document: A1