JP7204345B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

An eye tomography apparatus such as an apparatus using optical coherence tomography (OCT) (OCT apparatus) is capable of three-dimensionally observing the state inside the retinal layers. In recent years, this tomographic imaging apparatus has been attracting attention because it is useful for more accurately diagnosing diseases.

As a form of OCT, for example, SD-OCT (Spectral domain OCT), which uses a broadband light source and acquires an interferogram with a spectroscope, is known as a method of acquiring images at high speed. Also known is SS-OCT (Swept Source OCT), which uses a fast wavelength swept light source as a light source and measures spectral interference with a single-channel photodetector.

In recent years, an angiography method using OCT (OCTA: OCT angiography) has been proposed as an angiography method that does not use a contrast medium. In OCTA, a blood vessel image (hereinafter referred to as an OCTA image) is generated by projecting or integrating three-dimensional motion contrast data acquired by OCT onto a two-dimensional plane. Here, the motion contrast data is data obtained by repeatedly photographing the same section of the object to be measured by OCT, and detecting a temporal change in the object to be measured between the photographing. Motion contrast data can be obtained, for example, by calculating phases, vectors, and temporal changes in intensity of complex OCT signals from differences, ratios, or correlations.

JP 2016-209198 A

In the technique disclosed in Patent Document 1, a motion contrast front image (OCTA image) is generated by projecting or accumulating three-dimensional motion contrast data obtained by OCT. Here, the front image is a two-dimensional plane image in a direction intersecting the depth direction of the object. However, in such a front image generation method, since each pixel value is obtained by projecting or accumulating the motion contrast data values within the selected range, the positional relationship of the motion contrast data in the depth direction is not considered. Absent. Therefore, when observing those images, it may not be possible to grasp the state of the vertical relationship in the depth direction of the blood vessel intersection at the blood vessel intersection.

Accordingly, the present invention provides an image processing apparatus, an image processing method, and a program capable of providing a front image in which the positional relationship of blood vessels and the like in the depth direction can be observed.

An image processing apparatus according to an embodiment of the present invention generates a first front image, which is a motion contrast front image, using at least part of three-dimensional motion contrast data of a subject, and A second front image generated by a method different from that of the first front image using at least part of the second front image, wherein the relative positional relationship in the depth direction of the plurality of intersecting blood vessels of the subject can be identified. a generation unit that generates the second front image; and a display control unit that causes a display unit to display at least one of the first front image and the second front image , wherein the second front image is generated. is an image obtained by volume-rendering at least part of the three-dimensional motion contrast data from a front view viewpoint in a direction intersecting the depth direction of the subject .

An image processing method according to another embodiment of the present invention comprises the steps of: generating a first front image that is a motion contrast front image using at least part of three-dimensional motion contrast data of a subject; A second front image generated by a method different from that of the first front image using at least part of the motion contrast data, wherein the relative positional relationship in the depth direction of the plurality of intersecting blood vessels of the subject is generating the identifiable second front image; and causing a display unit to display at least one of the first front image and the second front image, wherein the second front image The image is an image obtained by volume-rendering at least part of the three-dimensional motion contrast data from a front view viewpoint in a direction intersecting the depth direction of the subject .

ADVANTAGE OF THE INVENTION According to this invention, the front image which can observe the positional relationship etc. in a depth direction, such as a blood vessel, can be provided.

1 shows a schematic configuration of an image processing system according to a first embodiment; FIG. 2 is a diagram for explaining the structure of an eye, a tomographic image, and a fundus image; 4 is a flowchart of a series of image processing according to Example 1; FIG. 4 is a diagram for explaining motion contrast data; FIG. It is a figure for demonstrating the effect of an image quality improvement process. It is a figure for demonstrating the effect of an image quality improvement process. FIG. 4 is a diagram for explaining a rendered image; FIG. FIG. 4 is a diagram for explaining a rendered image; FIG. 4 shows an example of a display screen according to Example 1. FIG. FIG. 11 is a diagram for explaining a rendering image according to Example 2; FIG. 11 shows an example of a display screen according to Example 2. FIG. 11 shows an example of a display screen according to Example 2. FIG. 11 shows an example of a display screen according to Example 3. FIG. 11 shows an example of a display screen according to Example 3. FIG. 11 shows an example of a display screen according to Example 3. FIG. 11 shows an example of a display screen according to Example 4. FIG. FIG. 11 shows a schematic configuration of an image processing system according to Example 5. FIG. FIG. 11 is a diagram for explaining hue image generation processing according to the fifth embodiment; 14 is a flowchart of high-quality data generation processing according to Modification 4;

Exemplary embodiments for carrying out the invention will now be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of components, etc. described in the following examples are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, the same reference numbers are used in the drawings to indicate identical or functionally similar elements.

In addition, although the eye to be examined, which is a human eye, will be described as an example of the subject, the subject may be skin, an organ, or the like. In this case, the image processing method according to the present disclosure can be applied to an image acquired from a medical device such as an endoscope. In addition, tomographic data is data containing information about the tomogram of the subject, and data based on interference signals by OCT, and data obtained by subjecting this to Fast Fourier Transform (FFT) and arbitrary signal processing. What it contains.

(Example 1)
An image processing system including an image processing apparatus according to a first embodiment of the present invention, in particular, an image processing system using a tomographic image of an eye will be described below with reference to FIGS. 1 to 8. FIG.

FIG. 1 shows a schematic configuration of an image processing system 1 including an image processing device 20 according to this embodiment. As shown in FIG. 1, the image processing system 1 includes an OCT apparatus 10 as an example of a tomography apparatus, an image processing apparatus 20, a fundus imaging apparatus 30, an external storage device 40, a display unit 50, and an input unit 60. is provided.

The OCT apparatus 10 is an example of a tomographic image capturing apparatus for capturing a tomographic image of an eye to be examined. Any type of OCT apparatus can be used as the OCT apparatus, such as SD-OCT and SS-OCT.

The image processing device 20 is connected to the OCT device 10, the fundus imaging device 30, the external storage device 40, the display section 50, and the input section 60 via interfaces, and can control these. Based on various signals acquired from the OCT device 10, the fundus imaging device 30, and the external storage device 40, the image processing device 20 generates a tomographic image of the subject's eye, an OCTA image, a rendering image, and an En-Face image based on luminance information. , and fundus images can be generated. Further, the image processing device 20 can perform image processing on these images. Note that the image processing apparatus 20 may be configured by a general-purpose computer, or may be configured by a computer dedicated to the image processing system 1 .

The fundus image capturing device 30 is a device for capturing a fundus image of the subject's eye, and for example, a fundus camera, an SLO (Scanning Laser Ophthalmoscope), or the like can be used as the device. The device configuration of the OCT device 10 and the fundus imaging device 30 may be an integrated type or a separate type.

The external storage device 40 holds information about the eye to be examined (patient's name, age, sex, etc.), various captured image data, imaging parameters, image analysis parameters, and parameters set by the operator in association with each other. there is The external storage device 40 may be configured by any storage device, and may be configured by, for example, a storage medium such as an optical disk or memory.

The display unit 50 is configured by an arbitrary display, and can display information and various images related to the subject's eye under the control of the image processing device 20 .

The input unit 60 is, for example, a mouse, a keyboard, or a touch operation screen. 20 can be entered. When the input unit 60 is a touch operation screen, the input unit 60 can be integrated with the display unit 50 .

In addition, although these components are shown as separate bodies in FIG. 1, some or all of these components may be integrated.

Next, the OCT apparatus 10 will be explained. The OCT apparatus 10 is provided with a light source 11 , a galvanomirror 12 , a focus lens stage 13 , a coherence gate stage 14 , a detector 15 and an internal fixation lamp 16 . Since the OCT apparatus 10 is a well-known apparatus, a detailed description thereof is omitted. Here, the imaging of a tomographic image performed according to an instruction from the image processing apparatus 20 will be described.

When the image processing device 20 gives an instruction for photographing, the light source 11 emits light. Light from the light source 11 is split into measurement light and reference light using a splitter (not shown). The OCT apparatus 10 irradiates a subject (eye to be examined) with measurement light and detects interference light between the return light from the subject and the reference light to generate an interference signal including tomographic information of the subject. be able to.

The galvanometer mirror 12 is used to scan the fundus of the subject's eye with measurement light, and the scanning range of the measurement light by the galvanometer mirror 12 can define the imaging range of the fundus in OCT imaging. By controlling the drive range and speed of the galvanomirror 12, the image processing apparatus 20 can define the planar imaging range and the number of scanning lines (scanning speed in the planar direction) on the fundus. In FIG. 1, the galvanomirror 12 is shown as one unit for the sake of simplification of explanation, but the galvanomirror 12 is actually composed of two mirrors for X scanning and Y scanning. A desired range on the above can be scanned with the measuring light. Note that the configuration of the scanning unit for scanning the measurement light is not limited to the galvanomirror, and any other deflection mirror can be used. Further, as the scanning unit, for example, a single deflecting mirror such as a MEMS mirror that can scan the measurement light in two-dimensional directions may be used.

A focus lens (not shown) is provided on the focus lens stage 13 . By moving the focus lens stage 13, the focus lens can be moved along the optical axis of the measurement light. Therefore, the focus lens can focus the measurement light on the retinal layer of the fundus via the anterior segment of the eye to be examined. The measurement light that irradiates the fundus is reflected and scattered by each retinal layer and returns along the optical path as return light.

The coherence gate stage 14 is used to adjust the length of the optical path of the reference light or measurement light in order to cope with the difference in axial length of the eye to be examined. In this embodiment, the coherence gate stage 14 is composed of a stage provided with a mirror, and by moving along the optical path of the reference light in the optical axis direction, the optical path length of the reference light can correspond to the optical path length of the measurement light. can. Here, the coherence gate represents a position in OCT where the optical distances of the measurement light and the reference light are equal. Coherence gate stage 14 may be controlled by image processor 20 . The image processing apparatus 20 can control the imaging range in the depth direction of the subject's eye by controlling the position of the coherence gate by the coherence gate stage 14, and can perform imaging on the retinal layer side or on the deeper side than the retinal layer. Shooting and the like can be controlled.

The detector 15 detects interference light between the return light of the measurement light from the eye to be inspected and the reference light, which is generated in an interference portion (not shown), and generates an interference signal. The image processing device 20 can generate a tomographic image of the subject's eye by acquiring an interference signal from the detector 15 and performing Fourier transform or the like on the interference signal.

The internal fixation lamp 16 is provided with a display section 161 and a lens 162 . In this embodiment, as an example of the display unit 161, a display unit in which a plurality of light emitting diodes (LD) are arranged in a matrix is used. The lighting position of the light-emitting diode is changed according to the part to be photographed under the control of the image processing device 20 . Light from the display unit 161 is guided to the subject's eye via the lens 162 . The light emitted from the display unit 161 has a wavelength of 520 nm, for example, and is displayed in a desired pattern under the control of the image processing device 20 .

Note that the OCT apparatus 10 may be provided with a drive control unit for the OCT apparatus 10 that controls driving of each component based on control by the image processing apparatus 20 .

Next, the structure and images of the eye acquired by the image processing system 1 will be described with reference to FIGS. 2(a) to 2(c). FIG. 2(a) is a schematic diagram of an eyeball. FIG. 2(a) shows the cornea C, lens CL, vitreous body V, macula M (the center of the macula represents the fovea), and optic papilla D. FIG. In this embodiment, a case of photographing the posterior pole of the retina including the vitreous body V, the macula M, and the optic papilla D will be mainly described. Although not described below, the OCT apparatus 10 can also photograph the anterior segment of the eye such as the cornea and the lens.

FIG. 2B shows an example of a tomographic image obtained by imaging the retina using the OCT apparatus 10. As shown in FIG. In FIG. 2B, AS indicates an image unit obtained by one A-scan. Here, the A-scan means acquisition of tomographic information in the depth direction at one point of the subject's eye through the series of operations of the OCT apparatus 10 . Acquiring two-dimensional tomographic information of the subject's eye in the transverse direction (main scanning direction) by performing A-scans multiple times in the transverse direction (main scanning direction) is called B-scanning. A single B-scan image can be configured by collecting a plurality of A-scan images acquired by A-scan. This B-scan image is hereinafter referred to as a tomographic image.

In FIG. 2(b), a blood vessel Ve, a vitreous body V, a macula M, and an optic papilla D are shown. In addition, the boundary line L1 is the boundary between the inner limiting membrane (ILM) and the nerve fiber layer (NFL), the boundary line L2 is the boundary between the nerve fiber layer and the ganglion cell layer (GCL), and the boundary line L3 is the photoreceptor inner segment. Represents outer segment junction (ISOS). Furthermore, the boundary line L4 represents the retinal pigment epithelium layer (RPE), the boundary line L5 represents the Bruch's membrane (BM), and the boundary line L6 represents the choroid. In a tomographic image, the horizontal axis (main scanning direction of OCT) is the x-axis, and the vertical axis (depth direction) is the z-axis.

FIG. 2C shows an example of a fundus image obtained by photographing the fundus of the subject's eye using the fundus imaging device 30 . FIG. 2(c) shows the macula M and the optic papilla D, and the blood vessels of the retina are indicated by thick curves. In the fundus image, the horizontal axis (main scanning direction of OCT) is the x-axis, and the vertical axis (sub-scanning direction of OCT) is the y-axis.

Next, the image processing device 20 will be described. The image processing device 20 is provided with an acquisition section 21 , an image processing section 22 , a drive control section 23 , a storage section 24 and a display control section 25 .

The acquisition unit 21 can acquire the data of the interference signal of the eye to be examined from the OCT apparatus 10 . Note that the interference signal data acquired by the acquisition unit 21 may be an analog signal or a digital signal. When the acquisition unit 21 acquires an analog signal, the image processing device 20 can convert the analog signal into a digital signal. The acquiring unit 21 can also acquire tomographic data, motion contrast data, tomographic images, fundus images, OCTA images, rendering images, and En-Face images generated by the image processing unit 22 . Furthermore, the acquisition unit 21 can acquire data including the fundus information acquired by the fundus imaging device 30 . Further, the acquisition unit 21 can acquire information for identifying the eye to be examined, such as the subject's identification number, from the input unit 60 or the like. The acquisition unit 21 can store the acquired various data and images in the storage unit 24 .

The image processing unit 22 can generate various images from the data acquired by the acquisition unit 21 and the data stored in the storage unit 24, and apply image processing to the generated images. The image processing unit 22 is provided with a tomographic image generating unit 221 , a motion contrast data generating unit 222 , a front image generating unit 223 , a detecting unit 224 , a high image quality data generating unit 225 and a rendering unit 226 .

The tomographic image generation unit 221 can generate tomographic data by performing processing such as Fourier transform on the interference signal acquired by the acquisition unit 21, and can generate a tomographic image based on the tomographic data. Note that any known method may be employed as a method for generating a tomographic image, and detailed description thereof will be omitted.

The motion contrast data generator 222 generates motion contrast data based on the tomographic data generated by the tomographic image generator 221 . Any known method may be used as a method for generating motion contrast data. For example, a decorrelation value or a variance value of pixel values of mutually corresponding pixels in two tomographic images based on interference signals acquired during a predetermined time interval, or a value obtained by dividing the maximum value by the minimum value (maximum value/ minimum value) can be obtained as motion contrast data. In this embodiment, the motion contrast data generator 222 generates motion contrast data based on the decorrelation value between tomographic data corresponding to two tomographic images.

The front image generator 223 can generate a two-dimensional motion contrast front image (OCTA image) from the three-dimensional motion contrast data generated by the motion contrast data generator 222 . The front image generation unit 223 can also generate a two-dimensional motion contrast front image from three-dimensional motion contrast data with high image quality. Furthermore, the front image generation unit 223 generates a two-dimensional front image (luminance En-Face image ) can be generated.

The detection unit 224 can detect the boundary line of each layer of the retina of the subject's eye based on the tomographic data acquired by the acquisition unit 21 . The high-quality data generation unit 225 can generate high-quality three-dimensional motion contrast data by aligning and averaging (combining) the three-dimensional motion contrast data. The high-quality data generation unit 225 can also generate high-quality three-dimensional tomographic data by aligning three-dimensional tomographic data and averaging them. Here, high-quality data refers to data with an improved S/N ratio compared to one-time imaging, or data with an increased amount of information necessary for examination/diagnosis.

The rendering unit 226 can perform volume rendering on high-quality three-dimensional tomographic data and high-quality three-dimensional motion contrast data to generate rendered images. A viewpoint determination unit 261 and a range selection unit 262 are provided in the rendering unit 226 . The rendering unit 226 volume-renders three-dimensional data based on the viewpoint position determined by the viewpoint determination unit 261 and the data range selected by the range selection unit 262 . Note that the rendering unit 226 may perform volume rendering on the three-dimensional tomographic data and the three-dimensional motion contrast data acquired by the acquisition unit 21 .

The viewpoint determination unit 261 determines a viewpoint position for volume rendering based on a predetermined viewpoint position stored in advance or a viewpoint position specified by the user. Note that the viewpoint position is also simply referred to as a viewpoint hereinafter.

The range selection unit 262 selects a range of data to be used for volume rendering based on a predetermined data range or a data range specified by the user. The range selection unit 262 can also select the data range of the 3D motion contrast data for volume rendering based on the upper and lower ends of the depth range (generation range) used when generating the OCTA image. For example, when the ILM/NFL boundary line is set as the upper end of the generation range of the OCTA image, and the GCL/IPL boundary line is set as the lower end, the range selection unit 262 selects the three-dimensional data range included between these boundary lines. can be selected.

The drive control unit 23 can control driving of each component of the OCT device 10 and the fundus imaging device 30 connected to the image processing device 20 . The storage unit 24 can store tomographic data acquired by the acquisition unit 21 and various images and data generated and processed by the image processing unit 22 . The storage unit 24 can also store a program or the like for performing the function of each component of the image processing apparatus 20 by being executed by the processor.

The display control unit 25 can control the display on the display unit 50 of various information acquired by the acquisition unit 21, various images generated and processed by the image processing unit 22, and information input by the user.

Each component of the image processing apparatus 20 other than the storage unit 24 may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit). Also, each component may be configured by a circuit or the like that performs a specific function, such as an ASIC. The storage unit 24 may be composed of, for example, an arbitrary storage medium such as an optical disk or memory.

Next, an image processing method according to this embodiment will be described. In the image processing method according to the present embodiment, a rendering image obtained by volume-rendering three-dimensional motion contrast data based on a front view viewpoint in the XY plane direction is displayed in a state that can be easily compared with an OCTA image. Here, the front view point of view is not strictly limited to a point of view seen from the front, but includes a point of view slightly angled from the front. By displaying such a rendered image in a state that can be easily compared with the OCTA image, it is possible to observe the positional relationship in the depth direction, such as the intersection of blood vessels, which cannot be grasped from the OCTA image. Note that the XY plane direction corresponds to a direction intersecting the depth direction of the subject.

A series of image processing procedures according to this embodiment will be described below with reference to FIG. FIG. 3 is a flowchart of a series of image processing according to this embodiment. When the image processing according to this embodiment is started, the process proceeds to step S301.

In step S<b>301 , the acquisition unit 21 acquires a subject identification number, which is an example of information for identifying an eye to be examined, from outside the image processing apparatus 20 such as the input unit 60 . The acquisition unit 21 acquires information about the subject's eye held in the external storage device 40 based on the subject identification number, and stores the information in the storage unit 24 .

In step S<b>302 , the drive control unit 23 controls the OCT apparatus 10 to scan the subject's eye for imaging, and the acquisition unit 21 acquires an interference signal including tomographic information of the subject's eye from the OCT apparatus 10 . The eye to be inspected is scanned by the drive control unit 23 controlling the OCT apparatus 10 and operating the light source 11, the galvanomirror 12, etc. in response to the user's instruction to start scanning.

The galvanomirror 12 includes a horizontal X scanner and a vertical Y scanner. Therefore, the drive control unit 23 can scan the measurement light in each of the horizontal (X) and vertical (Y) directions in the device coordinate system by changing the directions of these scanners. By simultaneously changing the directions of these scanners, the drive control unit 23 can also scan the measurement light in a direction that combines the horizontal direction and the vertical direction. Therefore, the drive control unit 23 can scan the measurement light in any direction on the fundus plane.

The drive control unit 23 adjusts various photographing parameters when photographing. Specifically, the drive control unit 23 sets at least the position of the pattern displayed by the internal fixation lamp 16, the scan range and scan pattern by the galvanomirror 12, the coherence gate position, and the focus.

The drive control unit 23 controls the light emitting diodes of the display unit 161 to control the position of the pattern displayed by the internal fixation lamp 16 so as to image the center of the macular region and the optic papilla of the subject's eye. Further, the drive control unit 23 sets, as the scan pattern of the galvanomirror 12, a scan pattern such as a raster scan, a radial scan, or a cross scan for imaging a three-dimensional volume. After the adjustment of these imaging parameters is completed, the driving control unit 23 controls the OCT apparatus 10 to perform imaging of the subject's eye in response to the user's instruction to start imaging.

In this embodiment, the scan pattern is a three-dimensional volume scan by raster scanning, and the OCT apparatus 10 captures the three-dimensional volume of the subject's eye N times (N is 2 or more) in order to generate high-quality data. do. The OCT apparatus 10 images the same imaging range with the same scan pattern for imaging repeatedly performed N times. For example, an area of 3 mm×3 mm is photographed repeatedly at intervals of 300×300 (main scanning×sub scanning).

Further, when performing a three-dimensional volume scan, the OCT apparatus 10 repeatedly images approximately the same line location m times (where m is 2 or more) in order to generate motion contrast data. That is, for example, when m is two times, the acquisition unit 21 actually acquires 2×300×300 data. In this case, the motion contrast data generation unit 222 generates 300×300 three-dimensional motion contrast data based on 2×300×300 data by the processing described later. Note that substantially the same line location includes not only the same line location, but also a line location that deviates to a degree that is allowed when motion contrast data is generated.

Although detailed description is omitted in the present disclosure, the OCT apparatus 10 can perform tracking of the subject's eye in order to photograph the same location for averaging. As a result, the OCT apparatus 10 can scan the subject's eye with less influence of involuntary eye movement. Furthermore, when the OCT apparatus 10 detects movement such as blinking that causes an artifact when generating an image, it is possible to automatically perform rescanning at the location where the artifact occurred.

In step S<b>303 , the tomographic image generation unit 221 generates a tomographic image based on the interference signal acquired by the acquisition unit 21 . The tomographic image generator 221 can generate a tomographic image by performing general reconstruction processing on each interference signal.

First, the tomographic image generator 221 removes fixed pattern noise from the interference signal. Fixed pattern noise removal is performed by averaging a plurality of acquired A-scan signals to extract fixed pattern noise and subtracting it from the input interference signal. After that, the tomographic image generator 221 performs desired window function processing in order to optimize the depth resolution and the dynamic range, which are in a trade-off relationship when the interference signal is Fourier transformed in a finite interval. The tomographic image generating unit 221 generates tomographic data by performing fast Fourier transform (FFT) processing on the window function-processed interference signal.

The tomographic image generation unit 221 obtains each pixel value of the tomographic image based on the generated tomographic data, and generates a tomographic image. Note that the method for generating the tomographic image is not limited to this, and any known method may be used.

In step S<b>304 , the motion contrast data generator 222 generates motion contrast data based on the tomographic data generated by the tomographic image generator 221 . Here, generation of motion contrast data will be described with reference to FIG. FIG. 4 shows three-dimensional motion contrast data MC and two-dimensional motion contrast data LMC constituting the three-dimensional motion contrast data MC. Here, a method for generating two-dimensional motion contrast data LMC will be described.

The motion contrast data generator 222 first corrects positional deviations between a plurality of tomographic images captured in substantially the same range of the subject's eye. Here, any method may be used to correct positional deviation between tomographic images. For example, the motion contrast data generation unit 222 uses the characteristics of the shape of the fundus oculi to determine the position of the tomographic data corresponding to substantially the same part of the three-dimensional tomographic data obtained by photographing the substantially same range m times. Align. In the following description, tomographic data corresponding to one tomographic image is referred to as one set of tomographic data, and tomographic data corresponding to m tomographic images is referred to as m sets of tomographic data.

Specifically, one of m sets of tomographic data corresponding to m times of imaging of substantially the same location is selected as a template, and the degree of similarity with other tomographic data is obtained while changing the position and angle of the template. , and the amount of positional deviation from the template. After that, the motion contrast data generator 222 corrects each tomographic data of the m sets of tomographic data based on the obtained positional deviation amount.

ここで、Ａ（ｘ,ｚ）は断層データＡの位置（ｘ，ｚ）における輝度、Ｂ（ｘ，ｚ）は断層データＢの同一位置（ｘ、ｚ）における輝度を示している。 Next, the motion contrast data generation unit 222 obtains the decorrelation value M(x, z) between two sets of tomographic data for which the imaging time for each tomographic data is within a predetermined time interval using Equation 1.

Here, A(x, z) indicates the brightness at the position (x, z) of the tomographic data A, and B(x, z) indicates the brightness at the same position (x, z) of the tomographic data B. FIG.

The decorrelation value M(x, z) is a value between 0 and 1, and the larger the difference between the two luminances, the larger the value of M(x, z). The motion contrast data generation unit 222 can obtain a plurality of decorrelation values M(x, z) at the same position (x, z) when the number m of repetitions of imaging performed at substantially the same position is 3 or more. can. The motion contrast data generation unit 222 can generate final motion contrast data by performing statistical processing such as maximum value calculation and average calculation of the plurality of decorrelation values M(x, z) obtained. can. Note that when the number of repetitions m is 2, statistical processing such as maximum value calculation and average calculation is not performed, and the decorrelation value M (x, z) of the two sets of tomographic data A and B is the position (x, z) is the value of the motion contrast data.

The motion contrast data calculation formula shown in Equation 1 tends to be susceptible to noise. For example, when there is noise in non-signal portions of multiple sets of tomographic data and the values differ from each other, the decorrelation value increases and noise is superimposed on the motion contrast image. In order to avoid this, the motion contrast data generator 222 can regard tomographic data below a predetermined threshold as noise and replace it with zero as preprocessing. As a result, in the processing described later, the front image generation unit 223 can generate a motion contrast image with reduced influence of noise based on the motion contrast data generated by performing the preprocessing.

The image processing apparatus 20 performs the processing of steps S302 to S304 on the interference signals acquired by multiple volume scans, thereby obtaining a plurality of three-dimensional tomographic data (three-dimensional tomographic images) and a plurality of three-dimensional motion contrasts. Data can be obtained. Note that the image processing apparatus 20 may perform the processing of steps S302 to S304 whenever an interference signal is acquired in one volume scan. In this case, since only one piece of three-dimensional tomographic data is obtained in the first volume scan, the motion contrast data generation process can be omitted, and motion contrast data can be generated in the second and subsequent processes.

In step S305, the detection unit 224 detects a boundary line (layer boundary) between retinal layers in the three-dimensional tomographic data. The detection unit 224 detects any one of the boundaries of L1 to L6 or the GCL/IPL, IPL/INL, INL/OPL, and OPL/ONL boundaries (not shown) in the tomographic image of FIG. 2(b).

Specifically, the detection unit 224 generates a median image and a Sobel image by applying a median filter and a Sobel filter to the tomographic image to be processed. Next, the detection unit 224 generates a profile for each tomographic data corresponding to the A-scan from the generated median image and Sobel image. Here, the generated profile is a luminance value profile for the median image, and a gradient profile for the Sobel image. After that, the detection unit 224 detects peaks in the profile generated from the Sobel image. The detection unit 224 detects the boundary of each area of the retinal layer by referring to the profile of the median image corresponding to before and after the detected peak and between the peaks.

Note that the method for detecting layer boundaries is not limited to this method. The detector 224 can detect layer boundaries by any known method.

In step S306, the high-quality data generation unit 225 generates high-quality three-dimensional data using multiple pieces of three-dimensional tomographic data and multiple pieces of three-dimensional motion contrast data. Three-dimensional data hereinafter refers to three-dimensional tomographic data consisting of luminance values and three-dimensional motion contrast data consisting of decorrelation values.

Specifically, the high image quality data generation unit 225 acquires the motion contrast front image generated by the front image generation unit 223 based on the layer boundary information and a plurality of three-dimensional motion contrast data as an image for alignment. . A method of generating a motion contrast front image by the front image generation unit 223 will be described later. The high-quality data generation unit 225 selects reference 3D motion contrast data from a plurality of 3D motion contrast data based on the acquired image for alignment. Thereafter, the high-quality data generation unit 225 aligns the reference 3D motion contrast data with other 3D motion contrast data in the X, Y, and Z directions, and adds and averages the aligned motion contrast data. do. Accordingly, the high image quality data generation unit 225 can generate high image quality 3D motion contrast data.

In addition, the high-quality data generation unit 225 aligns a plurality of 3D tomographic data with reference to the 3D tomographic data corresponding to the 3D motion contrast data used as a reference, and performs alignment of a plurality of 3D tomographic data after alignment. Add and average the tomographic data. Thereby, the high-quality data generation unit 225 can generate high-quality three-dimensional tomographic data (three-dimensional tomographic image). Note that the high-quality data generation unit 225 may align a plurality of pieces of three-dimensional tomographic data using alignment parameters of motion contrast data.

Note that the motion contrast data registration method and the three-dimensional tomographic data registration method are not limited to the above methods, and any known method may be used.

Here, examples of images based on three-dimensional data before and after the image quality enhancement process are shown in FIGS. 5(a) to 6(b). FIG. 5A shows an image 500 of one section generated from the 3D motion contrast data before the image quality enhancement process and an image 501 of one section generated from the 3D motion contrast data after the image quality enhancement process. show.

In the image quality enhancement process, as described above, the image quality enhancement process is performed on a plurality of pieces of three-dimensional tomographic data as well as the three-dimensional motion contrast data. Here, FIG. 5B shows a cross-sectional image 510 generated from three-dimensional tomographic data before image quality enhancement processing and a one-sectional image 511 generated from three-dimensional tomographic data after image quality enhancement processing. indicates

As shown in FIGS. 5(a) and 5(b), the image before the image quality enhancement process has a lot of noise, and the difference between blood vessels and noise and the boundary between layers are unclear. On the other hand, the image after high-quality image processing has little noise, and the structure of blood vessels and retinal layers can be recognized.

FIG. 6A shows an OCTA image 600 of the retinal surface layer generated from the 3D motion contrast data before the image quality enhancement process, and an OCTA image 600 of the retinal surface layer generated from the 3D motion contrast data after the image quality enhancement process. 601 is shown. FIG. 6B shows a front image 610 generated from three-dimensional tomographic data before image quality enhancement and a front image 611 generated from three-dimensional tomographic data after image quality enhancement processing. As shown in FIGS. 6(a) and 6(b), noise and artifacts were reduced in the OCTA image and the front image (En-Face image) based on the luminance value by performing the image quality enhancement process. A clear image can be produced.

After the image quality enhancement process is performed, the process proceeds to step S307. In step S307, the front image generating unit 223 generates an OCTA image, which is a front image of the motion contrast data, from the high-quality three-dimensional motion contrast data based on the front image generation range specified by the user. Note that the acquisition unit 21 can acquire an instruction from the user regarding the generation range of the front image. Further, the acquisition unit 21 may acquire a default value predetermined according to the imaging mode of the OCT apparatus 10 or the image processing apparatus 20, the imaging region, or the like, as the generation range information.

Specifically, the front image generation unit 223 projects the three-dimensional motion contrast data corresponding to the range between the specified generation range upper end and generation range lower end onto a two-dimensional plane to generate an OCTA image. More specifically, the front image generator 223 extracts motion contrast data corresponding to a range between the upper end of the generation range and the lower end of the generation range from the entire 3D motion contrast data. After that, the front image generation unit 223 determines the pixel value of the OCTA image by performing processing such as average intensity projection (AIP) or maximum intensity projection (MIP) on the extracted motion contrast data in the depth direction. do. This allows the front image generator 223 to generate an OCTA image.

Note that the method of generating the OCTA image is not limited to the average intensity projection process or the maximum intensity projection process. For example, for the extracted motion contrast data, each pixel value of the OCTA image is generated by calculating a value such as the minimum value, median value, variance, standard deviation, integrated value, or sum in the depth direction. good too.

Next, in step S308, the rendering unit 226 performs volume rendering on the high-quality three-dimensional motion contrast data in the XY plane direction from a front view viewpoint to generate a rendered image (second front image). The rendering unit 226 performs volume rendering on the 3D motion contrast data based on the viewpoint information determined by the viewpoint determination unit 261 and the data range selected by the range selection unit 262 .

In the present embodiment, the viewpoint determination unit 261 determines front view viewpoint information when the 3D motion contrast data is viewed from the XY front direction, similarly to the OCTA images 600 and 601 shown in FIG. 6A. Also, the range selection unit 262 selects all of the 3D motion contrast data as the data range.

In this embodiment, the rendering unit 226 volume-renders the 3D motion contrast data using a ray casting method. In the ray-casting method, a light ray passing through the viewpoint and pixels is assumed, and by integrating the luminance values on this ray, the luminance of the light finally reaching the viewpoint is determined. Specifically, the brightness of the light that has passed through the voxel can be obtained by synthesizing the color and opacity given in advance to the voxel of the three-dimensional data on which the light beam is incident. The transparency is a value between 0 and 1, and the value obtained by subtracting the transparency from 1 is the opacity.

Moreover, the rendering method is not limited to the ray casting method, and any rendering method may be used. For example, the ray tracing method may be used. In the ray tracing method, not only shadows but also reflection and refraction can be calculated with parallel light sources and point light sources. In ray tracing, light is traced back from the viewpoint through each pixel on the screen to determine the brightness value and color of each pixel.

Here, FIG. 7A shows an example of a rendered image 701 obtained by volume rendering the 3D motion contrast data after the image quality enhancement processing by the rendering unit 226 . Note that FIG. 7A also shows a rendered image 700 obtained by performing volume rendering on the 3D motion contrast data before the image quality enhancement process. I know there is.

FIG. 7B shows a rendered image 702 in which a part of the blood vessel crossing portion in the rendered image 701 generated by the rendering unit 226 shown in FIG. 7A is enlarged. FIG. 7B also shows an OCTA image 703 that is an enlarged display of the OCTA image of the same location spatially as the rendered image 702 . Further, FIG. 7B shows a schematic 704 of the rendered image 702 and a schematic 705 of the OCTA image 703 .

Here, the rendered image 701 will be described with reference to a rendered image 702 and a schematic diagram 704 in which a part of the blood vessel crossing portion is enlarged. In the rendered image 702, a blood vessel Ve01 running in the horizontal direction and blood vessels Ve02 and Ve03 running in the vertical direction appear. Here, in the rendering image 702, as shown in a schematic diagram 704, it can be confirmed that the blood vessel Ve02 runs deep and the blood vessel Ve03 runs shallow with respect to the blood vessel Ve01.

On the other hand, in an OCTA image 703 obtained by displaying a part of the OCTA image in an enlarged manner, blood vessel intersections are displayed brightly as shown in a schematic diagram 705 due to mean intensity projection, maximum intensity projection, and the like. Therefore, in the OCTA image 703, it is difficult to confirm the vertical relationship in the depth direction of the blood vessels Ve01', Ve02', and Ve03'.

In this way, it is possible to identify the positions of blood vessels and the like in the subject in the depth direction in the rendered image generated from the three-dimensional motion contrast data. Therefore, by observing the rendered image generated for the three-dimensional motion contrast data, the user can easily grasp the vertical relationship of the blood vessels in the depth direction, which is difficult to recognize in the two-dimensional OCTA image.

After generating the rendering image, the process proceeds to step S309. In step S309, the display control unit 25 causes the display unit 50 to display the OCTA image generated from the three-dimensional motion contrast data and the rendered image in a state that facilitates comparison. In the present embodiment, these images are displayed side by side or displayed in a switchable manner to facilitate comparison. Here, FIG. 8 shows an example of a screen displayed on the display unit 50. As shown in FIG.

FIG. 8 shows a display screen 800 on which view mode tabs 823 and 824 are provided. A tab 823 displays a rendered image generated from three-dimensional motion contrast data together with an OCTA image, a fundus image, etc., and a tab 824 displays a rendered image generated from the three-dimensional data.

In addition to these, the display screen 800 of the display unit 50 may be provided with a patient tab, an imaging tab, a report tab, a setting tab, and the like. In this case, view mode tabs 823 and 824 are displayed within the report tab. The display screen 800 can also display a patient information display section, examination sort tabs, an examination list, and the like. Thumbnails of fundus images, tomographic images, and OCTA images may be displayed in the examination list. Note that these thumbnails can be generated from high quality data.

A tab 823 is provided with a button 822 for setting ON/OFF of a mode for generating high image quality data (high image quality mode). In this embodiment, various images generated based on high-quality data will be described below.

In addition, a tab 823 includes a fundus image 801, a first OCTA image 802, a first tomographic image 803, a rendered image 804 generated from three-dimensional motion contrast data, a second OCTA image 805, and a second tomographic image. 806 is displayed. Further, the tab 823 displays an image 807 superimposed on the fundus image 801 and a tab 808 for switching the type of the image 807 .

The tab 823 also displays the type 809 of the OCTA image to be displayed as the first OCTA image 802 , and the generation range top end 810 and generation range bottom end 811 of the first OCTA image 802 . Types of OCTA include superficial, deep, choroidal, etc., and OCTA images created in any range. The generation range upper end 810 of the first OCTA image 802 shows the type of layer boundary corresponding to the upper end and the offset value from the layer boundary. Offset values are indicated. A boundary line 818 indicating the upper end 810 of the generation range of the first OCTA image 802 and a boundary line 819 indicating the lower end 811 of the generation range are superimposed on the first tomographic image 803 .

The user can move the boundary lines 818 and 819 via the input unit 60, or change the type of layer boundary and the offset value corresponding to the generation range upper end 810 and the generation range lower end 811, so that the first The generation range of the OCTA image 802 can be changed.

A tab 823 also shows a tab 812 for switching the type of front image such as the rendered image 804 . By using the tab 812, the rendered image 804 can be switched between the first OCTA image 802, the front image (En-Face) based on the brightness information, and the like. Note that the user interface (UI) for switching the type of front image is not limited to tabs, and any UI may be used.

Furthermore, the tab 823 shows a type 813 of the second OCTA image 805, a pulldown 814 for selecting the type 813, a selection display 815, and a generation range top 816 and a generation range bottom 817 of the second OCTA image 805. ing. Pull-down 814 is an example of options for selecting the type of second OCTA image 805 . Generation range top 816 and generation range bottom 817 are similar to generation range top 810 and generation range bottom 811 of first OCTA image 802 . A boundary line 820 indicating the upper end 816 of the generation range of the second OCTA image 805 and a boundary line 821 indicating the lower end 817 of the generation range of the second OCTA image 805 are superimposed on the second tomographic image 806 .

The user can move the boundary lines 820 and 821 via the input unit 60 or change the type and offset value of the layer boundary corresponding to the generation range upper end 816 and the generation range lower end 817, thereby making the second The generation range of the OCTA image 805 can be changed.

The tab 823 also displays an arrow 845 indicating the position of the first tomographic image 803 in the XY plane and an arrow 846 indicating the position of the second tomographic image 806 in the XY plane. The display/non-display of these arrows and the change of the position of the arrow can be switched based on the user's instruction via the input unit 60 .

A rendered image 804 is an image rendered by the rendering unit 226 from the same viewing direction (front view direction) as the first OCTA image 802 and the second OCTA image 805 for the three-dimensional motion contrast data. In the rendered image 804, as described above, it is easy to grasp the vertical relationship in the depth direction of blood vessels, etc., which is difficult to recognize in the two-dimensional OCTA image. In this embodiment, such a rendered image 804 can be displayed side by side with the first OCTA image 802 and the second OCTA image 805 . Also, the rendering image 804, the first OCTA image 802, and the like can be displayed in a switchable manner using a tab 823. FIG. In these cases, the front-view rendering image and the OCTA image are displayed in a state that facilitates comparison, so that the user can easily grasp the positional relationship in the depth direction of the blood vessel intersections and the like shown in the OCTA image. can do.

Next, in step S<b>310 , the acquisition unit 21 acquires from the user, via the input unit 60 , an instruction as to whether or not the image processing by the image processing device 20 should end. If the acquisition unit 21 acquires an instruction not to end the process, the process returns to step S302 to continue the process. On the other hand, when the acquiring unit 21 acquires an instruction to end the processing, the image processing device 20 ends the series of image processing.

As described above, the image processing apparatus 20 according to this embodiment includes the acquisition unit 21 , the image processing unit 22 , and the display control unit 25 . The acquisition unit 21 acquires three-dimensional motion contrast data of the subject. The front image generation unit 223 of the image processing unit 22 generates an OCTA image (first front image) using at least part of the three-dimensional motion contrast data.

The rendering unit 226 of the image processing unit 22 uses at least part of the three-dimensional motion contrast data to create a rendered image (second front image) that is different from the OCTA image and is capable of identifying the position of the subject in the depth direction. ). More specifically, the rendering unit 226 generates a rendered image by performing volume rendering on at least part of the three-dimensional motion contrast data from a front view viewpoint in a direction intersecting the depth direction of the subject. The display control unit 25 causes the display unit 50 to display at least one of the OCTA image and the rendering image. More specifically, the display control unit 25 causes the OCTA image and the rendered image to be displayed side by side on the display unit 50, or display them in a switchable manner.

As a result, the image processing apparatus 20 according to the present embodiment can display the OCTA image and the rendered image in a state that facilitates comparison, and provides a front image that allows observation of the positional relationship in the depth direction of blood vessel intersections and the like. can do. Therefore, the user can easily grasp the vertical relationship of the blood vessels in the depth direction of the subject from the displayed image.

In this embodiment, the second front image generation process (step S308) is performed after the front image generation process (step S307), but the timing of generating the rendering image is not limited to this. In this embodiment, since all of the three-dimensional motion contrast data is volume rendered, the second front image processing should be performed after the image quality improvement processing (step S307) and before the display processing (step S310). good. Therefore, the second front image generation process (step S308) may be performed before the front image generation process (step S307), or may be performed in parallel with the front image generation process (step S307).

(Example 2)
In Example 1, the rendered image created using all three-dimensional motion contrast data and the OCTA image were displayed in a state that facilitates comparison. On the other hand, in the second embodiment, a rendered image is displayed using data in a range corresponding to the range in the depth direction (generation range) of the motion contrast data used to generate the OCTA images to be compared.

The image processing system according to this embodiment will be described below with reference to FIGS. 9A to 10B. Image processing by the image processing system according to the present embodiment will be described below, focusing on differences from the image processing according to the first embodiment. The configuration and processing flow of the image processing system according to the present embodiment are the same as those of the image processing system 1 according to the first embodiment, so the same reference numerals are used to denote them, and the description thereof is omitted. Since the image processing according to the present embodiment differs from the image processing according to the first embodiment mainly in the second front image generation processing (step S308) and the display processing (step S309), these processing will be described below. .

In the second front image generation process (step S308) according to the present embodiment, the range selection unit 262 of the rendering unit 226 selects the same data range as the generation range of the OCTA image generated in step S307. Specifically, the range selection unit 262 selects the 3D motion contrast data based on the upper and lower ends of the generation range of the OCTA image, more specifically, each layer boundary and the offset value corresponding to the upper and lower ends of the generation range. Select a data range. For example, when the ILM/NFL boundary line is set as the upper end of the OCTA image generation range and the GCL/IPL boundary line is set as the lower end, the range selection unit 262 selects only the 3D motion contrast data between these boundaries. do.

The rendering unit 226 performs volume rendering on the 3D motion contrast data based on the front view viewpoint determined by the viewpoint determination unit 261 and the data range selected by the range selection unit 262 . Note that the volume rendering is the same as that of the first embodiment, so the explanation is omitted.

FIGS. 9(a) and 9(b) show an example of an OCTA image in this embodiment and an example of a rendering image obtained by volume rendering the three-dimensional motion contrast data of the same range as the OCTA image. FIG. 9A shows an OCTA image 900 of a portion corresponding to the surface layer of the retina, and a rendered image 901 obtained by rendering the data of the same range from the viewpoint in the XY front direction. FIG. 9B shows an OCTA image 910 of a portion corresponding to the deep layers of the retina, and a rendered image 911 obtained by rendering the data of the same range from the viewpoint in the XY front direction.

In the rendered images 901 and 911, similarly to the rendered image 701 according to the first embodiment, it can be seen that the positional relationship in the depth direction of the intersecting portions of blood vessels can be grasped.

In the display process (step S309) according to this embodiment, the display control unit 25 causes the display unit 50 to display the rendered image 804 generated in step S308, as shown in FIGS. 10A and 10B. 10A shows an example of displaying a rendering image 804 corresponding to the first OCTA image 802, and FIG. 10B shows an example of displaying a rendering image 804 corresponding to the second OCTA image 805. FIG.

In this embodiment, the tab 812 for switching the type of front image is used to display the rendering image 804 corresponding to the generation range of the first OCTA image 802 and the rendering image 804 corresponding to the second OCTA image 805. can be switched. Also, using a tab 812, the displayed image can be switched to a rendered image using all three-dimensional motion contrast data, various OCTA images, and En-Face images.

With such a display, it is possible to display the rendering image generated from the 3D motion contrast data in the data range corresponding to the generation range of the OCTA image to be compared in a state that facilitates comparison. In this case, since the OCTA image and the rendered image to be compared are generated from the data in the corresponding range, it is easy to compare the images, and the positional relationship such as the intersection of blood vessels appearing in the OCTA image can be more easily grasped. be able to.

As described above, the image processing apparatus 20 according to the present embodiment generates a rendered image (second front image) based on the motion contrast data in the range corresponding to the generation range of the OCTA image. This makes the OCTA image and the rendered image more comparable to each other.

In this embodiment, the range selection unit 262 selects the same depth range as the OCTA image generation range as the data range corresponding to the OCTA image generation range. However, the data range selected by the range selection unit 262 does not need to completely match the generation range of the OCTA image. The range selection unit 262 may select, for example, a depth range that is several tens of percent wider or narrower than the OCTA image generation range as the data range corresponding to the OCTA image generation range.

(Example 3)
In Examples 1 and 2, the rendering image generated from the 3D motion contrast data and the OCTA image were displayed in a state that facilitates comparison. In the third embodiment, in addition to this, a configuration for switchably displaying a rendered image based on 3D motion contrast data and a rendered image based on 3D tomographic data will be described. The image processing according to this embodiment corresponds to the image processing related to the display on the tab 824 of the display screen 800 shown in FIG. 8, FIG. 10A, and the like.

The image processing system according to this embodiment will be described below with reference to FIGS. 11A to 11C. Image processing by the image processing system according to the present embodiment will be described below, focusing on differences from the image processing according to the first embodiment. The configuration and processing flow of the image processing system according to the present embodiment are the same as those of the image processing system 1 according to the first embodiment, so the same reference numerals are used to denote them, and the description thereof is omitted. Since the image processing according to the present embodiment differs from the image processing according to the first embodiment mainly in the second front image generation processing (step S308) and the display processing (step S309), these processing will be described below. .

In the second front image generation processing (step S308) according to the present embodiment, the viewpoint determination unit 261 determines the viewpoint according to the type of data for volume rendering. Specifically, the viewpoint determination unit 261 determines the viewpoint in the XY plane direction rather than in the XZ plane direction when rendering three-dimensional motion contrast data, and determines the viewpoint in the XY plane direction when rendering three-dimensional tomographic data. A viewpoint is determined with respect to the XZ or YZ plane direction. As a result, the generated rendering image can display a surface with a large amount of information in each data. Here, the XZ plane direction or the XY plane direction corresponds to a direction parallel to the depth direction of the subject.

Note that the viewpoint determination unit 261 does not need to determine the viewpoint in front of the XY plane or the XZ plane, and may set a point diagonally from the front direction as the viewpoint. Therefore, the viewpoint determination unit 261 can also determine the viewpoint according to the user's specification or a preset viewpoint angle.

A range selection unit 262 selects a data range to be used for rendering, as in the first embodiment. The rendering unit 226 volume-renders the three-dimensional data based on the viewpoint information determined by the viewpoint determination unit 261 and the data range selected by the range selection unit 262 to generate a rendered image. Note that the volume rendering is the same as that of the first embodiment, so the explanation is omitted.

In the display process (step S309), the display control unit 25 causes the display unit 50 to display only the rendering image generated in step S308. Here, FIGS. 11A to 11C show an example of the display screen 800 in this embodiment. FIG. 11A shows an example of a rendered image 1101 based on 3D motion contrast data generated by front view volume rendering in the XY directions. FIG. 11B shows an example of a rendered image 1105 obtained by volume-rendering three-dimensional tomographic data from a viewpoint viewed from the XZ direction. FIG. 11C shows an example of a rendered image 1101' obtained by volume-rendering the 3D motion contrast data at a viewpoint obliquely angled from the front view viewpoint in the XY directions.

The display screen 800 is provided with view mode tabs 823 and 824 . A tab 823 displays the screens shown in the first and second embodiments. Tab 824 displays the screens shown in FIGS. 11A to 11C.

A tab 824 shows an area 1100 for displaying rendered images, a rendered image 1101, and an orientation cube 1102. FIG. Here, the orientation cube 1102 indicates the orientation of rendering data (rendering image) corresponding to the viewpoint of volume rendering. Each face of the orientation cube 1102 displays T, S, N, I, A, and P one by one. These mean Temporal, Superior, Nasal, Inferior, Anterior, and Posterior, respectively, and correspond to the respective postures. By checking the orientation cube 1102, the user can grasp the orientation of the rendering data.

Note that the UI can be configured so that the user can change the viewing direction of volume rendering and the image display magnification by dragging the mouse or rotating the mouse wheel in the area 1100 . Also, a button or the like for changing the viewpoint direction or the display magnification may be provided.

The tab 824 also shows radio buttons 1110 and 1118, sliders 1111 and 1112, a reset button 1113, a layer type 1114, checkboxes 1115 and 1116, and a button 1117 for selecting all checkboxes.

A radio button 1110 is used to select a volume rendering mode. As shown in FIG. 11A, when displaying a rendered image 1101 of motion contrast data, angiography is selected as a display mode using a radio button 1110 . In this embodiment, as an initial image when angiography is selected, the viewpoint determination unit 261 determines a front view viewpoint in the XY plane direction and displays a rendered image 1101 that has undergone volume rendering.

Sliders 1111 and 1112 are sliders for adjusting the brightness and contrast of the rendered image to be displayed. A reset button 1113 indicates a button for resetting the adjusted value to the default value.

A layer type 1114 indicates a layer type selected by check boxes 1115 and 1116 . Check boxes 1115 and 1116 represent layer visualization selection and Peeling selection checkboxes. A button 1117 is a button for selecting all of the check boxes 1115 and 1116 . In addition, although not shown, there may be a separate button for canceling the selection of all the check boxes 1115 and 1116 . Also, a function of canceling the selection of all the check boxes 1115 and 1116 may be added to the button 1117 .

The range selection unit 262 may acquire a designation from the user regarding the data range to be used for volume rendering according to the selection of the check box 1115 . In addition, the range selection unit 262 may obtain designation of the data range independently of the selection of the check box 1115 . In this case, the data corresponding to the selection of check box 1115 may be visualized or hidden by other techniques. In this embodiment, as an initial image when angiography is selected, the range selection unit 262 selects all of the 3D motion contrast data as the data range and displays a rendering image 1101 that has undergone volume rendering.

Radio button 1118 is used to select a projection method for the displayed image. By using the radio button 1118, projection by volume rendering such as the ray casting method or the maximum intensity projection (MIP) method can be selected. When the maximum intensity projection method is selected with the radio button 1118 , an image (not shown) obtained by performing maximum intensity projection on the three-dimensional data is displayed in the area 1100 .

In addition, in the present embodiment, two types of options for the projection method are shown, but the type of projection method is not limited to this. Here, as described above, in the OCTA image projected by the maximum intensity projection method, it is difficult to comprehend the positional relationship in the depth direction of the intersections of the blood vessels. Therefore, it is preferable to display by a method other than the maximum intensity projection method as the projection method in the case of the angiography display mode in the tab 824 . However, maximum intensity projection may be selected in the angiography display mode, such as when it is desired to see a relatively large OCTA image.

FIG. 11B shows an example of displaying a rendering image 1105 obtained by volume-rendering three-dimensional tomographic data obtained by raster scanning. In the example of FIG. 11B, when displaying the rendered image 1105, the viewpoint determination unit 261 determines the viewpoint from the XZ plane, and the rendering unit 226 generates the rendered image 1105 based on the viewpoint.

A peeling control 1103 is superimposed on a rendered image 1105 obtained by volume rendering three-dimensional tomographic data. By operating the peeling control 1103, the user can peel off (peeling) only the layer selected by the check box 1116 from the volume data.

Note that FIGS. 11A and 11B show an example in which the front view viewpoint is determined with respect to the XY plane and the XZ plane. On the other hand, as described above, the viewpoint need not be set in front of each surface, and the angle may be set in an oblique direction from the front direction. Here, the set oblique angle can be, for example, about ±20 degrees.

FIG. 11C shows an example of a rendered image 1101' when an oblique angle is set with respect to the viewpoint of the rendered image 1101 shown in FIG. 11A. FIG. 11C also shows an orientation cube 1102' when the angle is set in an oblique direction, and the user can check the viewpoint angle by checking the orientation cube 1102'.

As described above, in the image processing apparatus 20 according to the present embodiment, the acquisition unit 21 further acquires three-dimensional tomographic data including tomographic information of the subject. Also, the rendering unit 226 generates a rendered image (third image) based on the tomographic data by volume-rendering at least part of the three-dimensional tomographic data.

Here, the viewpoint determination unit 261 of the rendering unit 226 sets the viewpoint of the rendered image based on the motion contrast data to the direction intersecting the depth direction of the subject more frontally than the viewpoint of the rendered image based on the tomographic data. Determine the viewpoint close to your eyes. In particular, in this embodiment, the rendering unit 226 performs volume rendering on at least a part of the three-dimensional tomographic data from a front view viewpoint in a direction parallel to the depth direction of the subject, thereby rendering a rendered image based on the tomographic data. to generate

Then, the display control unit 25 causes the display unit 50 to switch between rendering images based on the motion contrast data and rendering images based on the tomographic data.

Thereby, the user can confirm the planar image from the viewpoint direction suitable for the observation purpose according to the type of the planar image to be observed.

(Example 4)
Examples 1 and 2 show examples in which a rendering image generated from 3D motion contrast data and an OCTA image are displayed side by side or can be switched. On the other hand, in the third embodiment, an example of displaying a part of the rendering image for the OCTA image will be described.

The image processing system according to this embodiment will be described below with reference to FIGS. 12(a) to 12(c). Image processing by the image processing system according to the present embodiment will be described below, focusing on differences from the image processing according to the second embodiment. Note that the configuration and processing flow of the image processing system according to the present embodiment are the same as those of the image processing system 1 according to the second embodiment, so the same reference numerals are used to denote them, and a description thereof will be omitted. Since the image processing according to the present embodiment differs from the image processing according to the second embodiment mainly in the display processing (step S309), the display processing will be described below.

In the display processing (step S309) according to the present embodiment, the display control unit 25 causes the display unit 50 to partially display the rendering image generated in step S308 with respect to the OCTA image. Specifically, the display control unit 25 causes the display unit 50 to display a part of the rendering image superimposed on the OCTA image, or to display it around the OCTA image.

FIG. 12(a) shows an example in which part of the rendered image is superimposed on the OCTA image and displayed, and FIG. 12(b) shows an example in which part of the rendered image is displayed around the OCTA image. 12(a) and 12(b) show a first OCTA image 802 and a first tomographic image 803, which are part of the display screen 800. FIG. UIs 1201 and 1202 are shown in FIGS. 12(a) and 12(b).

The UI 1201 is used to switch ON/OFF of partial display of the OCTA image according to this embodiment. In addition, the UI 1202 is used to select whether to enlarge a portion of the OCTA image or display a portion of the rendered image as partial display of the OCTA image. Note that these UIs 1201 and 1202 are examples, and may have other aspects. In this embodiment, an example in which display of a portion of the rendered image is selected will be described.

First, the example shown in FIG. 12(a) will be described. In FIG. 12A, a rendered image 1203, which is part of the rendered image generated in step S308, is superimposed on the first OCTA image 802. In FIG. The display control unit 25 causes the display unit 50 to superimpose and display a part of the rendered image corresponding to the portion of the first OCTA image 802 on which the rendered image 1203 is superimposed as the rendered image 1203 on the first OCTA image 802 . .

In the example shown in FIG. 12A, for example, when the user selects the UI 1201 and then moves the mouse cursor or the like on the first OCTA image 802, a rendered image 1203 is displayed in a partial area around the mouse cursor. be. As a result, the rendering image can be superimposed on only the part of the first OCTA image 802 that the user wants to focus on, and the OCTA image and the rendering image can be displayed in a state that facilitates comparison. Therefore, the user can easily comprehend the positional relationship in the depth direction of the blood vessel crossing portion of interest.

Next, the example shown in FIG.12(b) is demonstrated. In FIG. 12B, an area 1204 indicating a portion corresponding to the rendered image is superimposed on the first OCTA image 802, and the rendered image 1205 is placed in a different location from the first OCTA image 802, for example, the first OCTA image. It is displayed around the image 802 . The display control unit 25 displays a portion of the rendered image corresponding to the portion where the region 1204 of the first OCTA image 802 is superimposed as the rendered image 1205 around the first OCTA image 802 . Note that the location where the rendering image 1205 is displayed is not limited to the left side of the OCTA image as shown in FIG.

In the example shown in FIG. 12B, for example, after the user selects the UI 1201 and then moves the mouse cursor or the like on the first OCTA image 802, an area 1204 is displayed in a partial area around the mouse cursor. . A rendering image 1205 corresponding to the area 1204 is displayed around the first OCTA image 802 . In this case, the rendering image can be displayed in a state in which the OCTA image and the rendering image of the part that the user wants to focus on can be observed at the same time. Therefore, the OCTA image and the rendered image are displayed in a state that facilitates comparison. As a result, the user can easily grasp the positional relationship in the depth direction of blood vessel intersections and the like to be focused on.

As described above, in the image processing apparatus 20 according to the present embodiment, the display control unit 25 causes the display unit 50 to display the OCTA image and a part of the rendered image side by side or superimpose them. Therefore, the OCTA image and the rendered image are displayed in a state in which it is easy to compare them, so that the user can easily grasp the positional relationship in the depth direction of blood vessels and the like.

In particular, the display control unit 25 causes the display unit 50 to display the image of the portion corresponding to the user's instruction in the rendering image side by side with the OCTA image, or superimpose it on the OCTA image. Therefore, only the region that the user pays attention to is displayed in an easy-to-contrast state, so that the user can easily grasp the positional relationship in the depth direction of the blood vessel crossing portion that the user wants to pay attention to.

The display control unit 25 can also cause the display unit 50 to display a rendering image obtained by volume-rendering the three-dimensional tomographic data so as to be superimposed on the tomographic image or displayed around the tomographic image. FIG. 12(c) shows an example in which a part of a rendering image obtained by volume rendering three-dimensional tomographic data is displayed superimposed on the first tomographic image 803 . The display control unit 25 superimposes and displays a part of the rendered image corresponding to the portion of the first tomographic image 803 on which the rendered image 1206 is superimposed as the rendered image 1206 on the first tomographic image 803 .

In the example shown in FIG. 12C, when the user selects the UI 1201 and then moves the mouse cursor or the like on the first tomographic image 803, a rendering image 1206 corresponding to the region is displayed in a partial region around the mouse cursor. is displayed. As a result, the user can easily comprehend the three-dimensional tomographic structure of the desired portion of the tomographic image. Also, in the case of a tomographic image, as shown in FIG. 12B, a rendering image may be displayed in a region separate from the tomographic image.

Note that the viewpoint when generating the rendering image 1203 for the OCTA image is the front view viewpoint in the XY plane direction, as in the second embodiment. Here, the viewpoint for generating the rendered image 1203 is not limited to the front view viewpoint, and may have a certain degree of angle as described above. The angle in this case can be changed by operating the mouse wheel, keyboard, UI on the screen, or touch panel screen operation.

On the other hand, the viewpoint for generating the rendered image 1206 for the tomographic image is the viewpoint for the XZ plane or the YZ plane direction, as in the third embodiment. Also, the viewpoint in this case can be changed to have a certain degree of angle.

In such a case, it is possible to display a rendered image from an appropriate viewpoint according to the OCTA image or tomographic image to which the rendered image corresponds. Therefore, the user can confirm the front image from the viewpoint direction suitable for the purpose of observation according to the type of image to be observed.

12A to 12C, when enlarged display is selected on the UI 1202, each rendering image is replaced with each rendering image. An enlarged image is displayed. Any known method may be used to enlarge each image. Moreover, when displaying a rendered image, the rendered image may be enlarged and displayed.

Further, in this embodiment, the display control unit 25 causes the display unit 50 to partially display the rendering image generated in step S308 with respect to the OCTA image. On the other hand, the rendering unit 226 may generate a rendered image only for the area selected for the OCTA image. In this case, the display control unit 25 causes the display unit 50 to superimpose and display the entire generated rendering image on the selected region of the OCTA image or display it around the OCTA image. can be done. Note that, in such a case, the rendering unit 226 generates a rendering image as needed according to the region selected for the OCTA image, and the display control unit 25 renders the generated rendering image to be displayed. Images can be updated at any time.

(Example 5)
In the first to fourth embodiments, an example of providing an image obtained by volume-rendering three-dimensional data has been described in order to provide an image in which the positional relationship in the depth direction of blood vessel intersections, etc. can be observed. On the other hand, in a fifth embodiment, a method of generating an image in which the positional relationship in the depth direction of blood vessel intersections and the like can be observed using color information will be described.

The image processing system according to this embodiment will be described below with reference to FIGS. 13 to 14(b). Image processing by the image processing system according to the present embodiment will be described below, focusing on differences from the image processing according to the first embodiment. The configuration and processing of the image processing system according to the present embodiment, which are the same as the configuration and processing of the image processing system 1 according to the first embodiment, are indicated using the same reference numerals, and the description thereof is omitted.

FIG. 13 shows a schematic configuration of an image processing system 2 according to this embodiment. In the image processing system 2 according to this embodiment, the image processing unit 132 of the image processing device 130 is provided with a hue image generation unit 1326 instead of the rendering unit 226 .

The hue image generator 1326 generates an OCTA image (hue image) by coloring the three-dimensional motion contrast data according to the depth position. Specifically, the hue image generation unit 1326 uses a predetermined tone curve to determine pixel values having color information according to the depth position of the 3D motion contrast data within the generation range for generating the OCTA image. , to generate a hue image.

The display control unit 25 can cause the display unit 50 to display the hue image generated by the hue image generation unit 1326 in the same manner as the fundus image, the tomographic image, and the normal OCTA image.

Next, a series of image processing according to this embodiment will be described. Since the image processing according to the present embodiment differs from the image processing according to the first embodiment mainly in the second front image generation processing (step S308) and the display processing (step S309), these processing will be described below. do.

In the second front image generation processing (step S308) according to the present embodiment, the hue image generation unit 1326 has color information corresponding to the depth position of the 3D motion contrast data within the generation range for generating the OCTA image. Generate a hue image. Specifically, the hue image generation unit 1326 uses a predetermined tone curve to determine pixel values having color information according to the depth position of the 3D motion contrast data within the generation range for generating the OCTA image. , to generate a hue image.

More specifically, first, the hue image generation unit 1326 acquires the range (generation range) in the depth direction of the three-dimensional motion contrast data for generating the hue image. The hue image generator 1326 can acquire the generation range for generating the OCTA image by the front image generator 223 as the generation range for generating the hue image. Note that the generation range for generating the hue image does not need to strictly match the generation range for the front image generation unit 223 to generate the OCTA image. good.

The hue image generator 1326 determines a representative value at each pixel position for the three-dimensional motion contrast data within the generation range by the maximum intensity projection method. In the maximum intensity projection method, the maximum value within the generation range or the maximum value (slab MIP) of the range having a certain thickness in the data obtained by each A-scan is used as the representative value of each pixel. Here, FIG. 14A shows an example of a normal OCTA image 1402 generated based on the motion contrast data in the range of boundary lines 818 and 819 corresponding to the upper and lower ends of the generation range by the maximum intensity projection method. show. The hue image generator 1326 performs coloring based on the depth position of the motion contrast data having the maximum value within the generation range, which is the representative value at each pixel position. As a result, a hue image composed of pixel values having color information corresponding to the OCTA image 1402 is generated.

In this embodiment, the hue image generator 1326 performs coloring based on the depth position of the motion contrast data using a tone curve 1400 as shown in FIG. 14(b). In this embodiment, the motion contrast data near the upper end of the generation range can be colored bluish, and the farther away from the upper end, the more reddish the motion contrast data becomes. The horizontal axis of the tone curve 1400 is the position in the depth direction, and the vertical axis is the output of each element of RGB. The origin of the vertical axis is 0, and the maximum value is 255.

The reference position corresponding to 0 on the horizontal axis is the position of the boundary line 818 corresponding to the upper end of the generation range. Let it be an offset position. In this embodiment, the reference position is the position of the boundary line 818. However, when the generation range of the hue image is changed from the surface layer to the deep layer, the boundary line The reference position of the horizontal axis is also changed according to the position of . Note that the reference position is the position of the boundary line 819 corresponding to the lower end of the generation range, and the position offset by a predetermined depth from the position of the boundary line 819 in the shallow direction is the maximum value (or the deeper direction is positive) may be the minimum value).

Furthermore, the reference position of the horizontal axis does not necessarily have to match the shape of the boundary line, and the horizontal line where the boundary line contacts or a straight line with a predetermined inclination may be used as the reference position. Also, the range from the reference position to the maximum value on the horizontal axis of tone curve 1400 does not always need to be the same. For example, the maximum value of the horizontal axis may be changed to make the range from the reference position to the maximum value different between when the retinal surface layer is the generation range and when the entire retinal layer is the generation range. The thickness of the surface layer is several tens of micrometers, but the thickness of all layers is several hundred micrometers. Therefore, the maximum value in the case where all layers are the generation range may be made larger than in the case where only the surface layer is the generation range.

The hue image generation unit 1326 uses the tone curve 1400 in this way to determine each pixel value by adding the output of each element of RGB corresponding to the position in the depth direction. Thereby, the hue image generation unit 1326 can generate a hue image, which is an OCTA image having color information indicating the position of the blood vessel in the depth direction in hue. Note that the hue image generation unit 1326 can also set the pixel value to black when the maximum value of the motion contrast data at each pixel position is equal to or less than a predetermined threshold value. In this case, since pixel positions with small values of motion contrast data are not colored, it is possible to visually distinguish between moving parts and non-moving parts.

Note that the hue image generation unit 1326 may generate a hue image by a method using an integrated value instead of the maximum intensity projection method. In this case, the hue image generation unit 1326 integrates the values of the motion contrast data from the upper end to the lower end of the generation range, and colors can be applied according to the depth position where the integrated value exceeds a predetermined threshold. can.

As an integration method, values of motion contrast data may be simply integrated, but an integration method that takes connectivity into account may also be employed. In the integration method that considers connectivity, only values of motion contrast data that are equal to or greater than a predetermined threshold value are integrated when performing integration from the upper end to the lower end of the generation range.

For example, integration is performed only when motion contrast data values equal to or greater than a threshold continue, and when a motion contrast data value equal to or less than the threshold is detected, the integrated value I0 and depth position Z0 up to that point are temporarily stored. do. Then, the motion contrast data value and the threshold value are compared in the depth direction, and when the motion contrast data value exceeds the threshold value, a new accumulation is started, and when a value below the threshold value is detected, accumulation is performed. Save the value I1 and the depth position Z1. The processing may be continued to the lower end of the generation range, and the pixel value of the pixel position may be determined from the tone curve 1400 according to the depth position corresponding to the maximum integrated value among the stored integrated values I. . Alternatively, the pixel value may be determined by synthesizing the colors corresponding to the depth positions of all locations where the average value of the continuous integrated values is equal to or greater than the threshold value.

After the hue image generator 1326 generates the hue image in step S308, the process proceeds to step S309. In step S309, the display control unit 25 displays the hue image side by side with the first OCTA image 802 and the like, like the rendering image 804 of the display screen 800 shown in FIG. 8 and the like.

As described above, since the normal OCTA image does not have color information, it is difficult to grasp the positional relationship of the subject in the depth direction only by observing the normal OTA image. On the other hand, the hue image is an OCTA image to which color information corresponding to the position of the object in the depth direction is added. position can be identified. Therefore, the normal OCTA image and the hue image are displayed in a state in which they can be easily compared, so that the user can easily grasp the positional relationship in the depth direction of the blood vessel intersections and the like.

As described above, in the image processing apparatus 130 according to the present embodiment, the hue image generation unit 1326 uses color information corresponding to at least a part of the three-dimensional motion contrast data and the position of the subject in the depth direction. A hue image (second front image) is generated.

The display control unit 25 displays the OCTA image and the hue image side by side on the display unit 50 . Therefore, since the normal OCTA image and the hue image are displayed in a state in which it is easy to compare them, the user can easily grasp the positional relationship in the depth direction of the blood vessel intersections and the like.

Note that the arrangement of the hues of the tone curve 1400 may be arbitrary. Further, in this embodiment, the gradients of RGB in the tone curve 1400 are linear gradients, but may be curved gradients.

Also, the hue image generated in this embodiment is not limited to being displayed side by side with the first OCTA image 802 . As in the first embodiment, the hue image may be displayed switchably with the first OCTA image 802, or as in the fourth embodiment, part of the hue image may be superimposed on the first OCTA image 802. It may be displayed or displayed around the first OCTA image. Also, when displaying as in the fourth embodiment, an enlarged hue image may be displayed. Furthermore, a hue image may be generated only for a selected region on the OCTA image, and the entire generated hue image may be superimposed on the OCTA image or displayed around the OCTA image.

(Modification 1)
In Examples 1 to 5, an example in which an OCTA image and a rendered image or a hue image are mainly displayed side by side has been described, but as described above, these images may be switched and displayed in the same area. For example, in the area where the first and second OCTA images are displayed, the OCTA image and the volume rendering data may be switched and displayed. As a result, images can be switched and displayed in the same region, making it easier to grasp the spatial correspondence of parts.

(Modification 2)
In the first to fifth embodiments, the rendering unit 226 has explained the method of performing volume rendering using the ray casting method, which is a direct method, but the method of performing volume rendering is not limited to this. For example, the rendering unit 226 may use an indirect method to represent the volume data as an isosurface and display it.

(Modification 3)
The image processing apparatus 20 may have an OCT mode for observing a tomographic image and an OCTA mode for observing an OCTA image as observation modes for observing a subject. In this case, in the OCT mode, the display control unit 25 displays, as an initial image of the rendering image, the rendering image generated based on the viewpoint having an angle with respect to the XY plane and the entire range of the three-dimensional tomographic data. to display. On the other hand, in the OCTA mode, the display control unit 25 causes the display unit 50 to use a rendered image generated based on a viewpoint that is closer to the front view with respect to the XY plane than the viewpoint in the OCT mode as an initial image of the rendered image. to display. In the OCTA mode, a rendered image can also be generated based on the 3D motion contrast data of the data range corresponding to the generation range of the OCTA image.

In this case, in the OCT mode, the user can easily grasp the depth direction of the tomogram of the subject from the rendered image, and in the OCTA mode, it is easy to grasp the depth direction positional relationship of blood vessels and the like from the rendered image. Therefore, the user can easily observe the tissue or the like of the subject according to the observation purpose by observing the rendered image according to the observation mode.

(Modification 4)
In Modified Example 4, a more specific example of the high image quality data generation process in step S306 will be described with reference to FIG. FIG. 15 is a flowchart of high-quality data generation processing according to this modification. In the high-quality data generation process according to this modified example, it is possible to reduce artifacts and generate 3D motion contrast data with higher image quality. In particular, the high image quality data generation process according to this modification is characterized by aligning a plurality of pieces of motion contrast data and selecting motion contrast data that serves as a reference for aligning in order to perform addition averaging.

When the high image quality data generation process according to this modification is started, the process moves to step S1501. In step S1501, the front image generation unit 223 generates an OCTA image for alignment based on motion contrast data within a predetermined generation range. In this modification, the upper end of the generation range is the ILM/NFL boundary line, and the lower end of the generation range is the boundary line 50 μm lower in the depth direction from GCL/IPL. Note that the generation range for generating the OCTA image for alignment may be set in advance to another range, or may be specified by the user.

In step S1502, alignment with other OCTA images (first alignment) is performed using all the OCTA images for alignment generated by the high-quality data generation unit 225 as reference images. The high-quality data generation unit 225 obtains alignment parameters and rotation parameters in the x-axis and y-axis directions, image evaluation values, image similarities, and the like for each OCTA image, and stores them in the storage unit 24 . The high-quality data generation unit 225 can also detect artifacts for each OCTA image, generate a mask image for the detected artifacts, and store the mask image in the storage unit 24 .

In step S1503, the high-quality data generation unit 225 selects a reference image based on the alignment result performed in step S1502. Specifically, the high-quality data generation unit 225 calculates an evaluation value that serves as a reference image selection criterion based on each alignment parameter, the image evaluation value, and the artifact area evaluation value obtained from the mask image. Select reference images based on values.

In step S1504, the high-quality data generation unit 225 uses the selected reference image to align the OCTA images in the x-axis direction (second alignment). The high-quality data generation unit 225 aligns the reference image and other OCTA images with respect to each line in the x-axis direction. Also, the high-quality data generation unit 225 aligns the rotation direction of each OCTA image based on the rotation parameter obtained in step S1502.

In step S1505, the high-quality data generation unit 225 uses the three-dimensional data corresponding to the reference image as reference data to perform global registration in the depth direction ( third alignment). More specifically, the high-quality data generator 225 first aligns the depth direction position and inclination in the data for each three-dimensional motion contrast data and each three-dimensional tomographic data. After that, the high-quality data generation unit 225 uses the reference data to match the positions and inclinations in the depth direction between the plurality of 3D motion contrast data and between the plurality of 3D tomographic data. Then, the high-quality data generation unit 225 transforms each piece of three-dimensional data based on the alignment parameters obtained in the first alignment, the second alignment, and the third alignment.

In step S1506, the high-quality data generation unit 225 sets a plurality of regions for alignment in the feature part in each tomographic image between the reference data and the target data, and calculates the x-axis direction and the depth for each set region. A local alignment of directions (fourth alignment) is performed. The high-quality data generation unit 225 transforms each three-dimensional data based on the alignment result.

In step S1507, the high-quality data generation unit 225 adds and averages the deformed 3D motion contrast data. In addition, the high-quality data generation unit 225 adds and averages a plurality of deformed three-dimensional tomographic data.

In step S1508, the high-quality data generation unit 225 calculates the position of the three-dimensional data in the depth direction based on the depth-direction alignment parameter in the reference data obtained in step S1505 for the averaged three-dimensional data. to the state before alignment. After that, the process moves to step S307.

According to the high-quality data generation processing according to this modified example, high-quality three-dimensional motion contrast data can be obtained even when artifacts are present in the motion contrast data due to involuntary eye movement or the like.

In this modified example, the three-dimensional data after averaging is returned to the state before alignment in the depth direction, but this process may not be performed. In this case, there is a gap in the depth direction between the layer boundary information detected in step S305 and the high-quality three-dimensional data. Therefore, after the high-quality data generation process, the detection unit 224 may perform layer boundary detection processing again for OCTA image generation. It may be deformed according to matching parameters.

Note that the high-quality data generation unit 225 may be configured using separate components for performing the first to fourth alignment processing, artifact removal processing, averaging processing, and the like.

(Modification 5)
In the first to fifth embodiments, a detailed front image is generated based on three-dimensional data whose image quality is improved by averaging a plurality of three-dimensional data. However, the target to which the detailed front image generation processing is applied is not limited to this. The process of generating a detailed front image may be applied to one piece of three-dimensional data as long as it is three-dimensional data with a certain degree of accuracy. The present invention may be applied to detailed front image generation processing, for example, one three-dimensional data subjected to noise removal, enhancement processing, or the like by arbitrary processing that does not use a plurality of three-dimensional data.

(Modification 6)
In Examples 1 to 5, the acquisition unit 21 acquires the interference signal acquired by the OCT apparatus 10 and the three-dimensional tomographic data generated by the tomographic image generation unit 221 . However, the configuration in which the acquisition unit 21 acquires these signals is not limited to this. For example, the acquisition unit 21 may acquire these signals from a server or imaging device connected to the image processing device 20 via a LAN, WAN, Internet, or the like. In this case, imaging-related three-dimensional tomographic data can be obtained by omitting processing related to imaging. Then, in step S305, high-quality data generation processing is performed. Note that the acquisition unit 21 may acquire not only the three-dimensional tomographic data but also the three-dimensional motion contrast data generated from the data. In this case, the process of generating motion contrast data can also be omitted. Therefore, it is possible to shorten a series of processing time from acquisition of tomographic information to display of a front image.

The configuration of the OCT apparatus 10 is not limited to the configuration described above, and part of the configuration included in the OCT apparatus 10 may be configured separately from the OCT apparatus 10 . Also, the user interface such as buttons and the display layout are not limited to those shown above.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Although the present invention has been described with reference to the examples, the present invention is not limited to the above examples. Inventions modified within the scope of the present invention and inventions equivalent to the present invention are also included in the present invention. In addition, the embodiments and modifications described above can be appropriately combined within the scope of the present invention.

20: Image processing device, 21: Acquisition unit, 25: Display control unit, 50: Display unit

Claims (21)

generating a first front image that is a motion contrast front image using at least part of the three-dimensional motion contrast data of the subject; and using at least part of the three-dimensional motion contrast data to generate the first front image. a generation unit that generates a second front image that is generated by a method different from the front image and that enables identification of a relative positional relationship in the depth direction of a plurality of intersecting blood vessels of the subject; ,
a display control unit that causes a display unit to display at least one of the first front image and the second front image;
with
The second front image is an image obtained by volume rendering at least part of the three-dimensional motion contrast data from a front view viewpoint in a direction intersecting the depth direction of the subject . Image processing device. 2. The image according to claim 1, wherein the display control unit causes the display unit to display the first front image and the second front image side by side, switchably display them, or display them in a superimposed manner. processing equipment. 3. The second front image according to claim 1, wherein said generator generates said second front image using said three-dimensional motion contrast data in a range corresponding to a depth range for generating said motion contrast front image. Image processing device. 4. The display control unit according to any one of claims 1 to 3 , wherein the display control unit causes the display unit to display a third image obtained by volume rendering at least part of the three-dimensional tomographic data of the subject. The image processing device according to . 5. The image processing according to claim 4 , wherein the viewpoint of the second front image is closer to the front view in a direction intersecting the depth direction of the subject than the viewpoint of the third image. Device. 5. The third image is an image obtained by volume-rendering at least part of the three -dimensional tomographic data from a front view viewpoint in a direction parallel to the depth direction of the subject. 6. The image processing apparatus according to 5 . The image processing apparatus according to any one of claims 4 to 6 , wherein the display control section causes the display section to switchably display the second front image and the third image. The image processing device has a mode for observing a tomographic image obtained using at least part of the three-dimensional tomographic data, and a mode for observing the motion contrast front image,
The display control unit
in the mode of observing the tomographic image, displaying the third image as an initial image on the display unit;
8. The image processing apparatus according to any one of claims 4 to 7 , wherein in the motion contrast front image observation mode, the second front image is displayed as an initial image on the display unit. The display control unit is configured to display the first front image, the second front image, and at least part of the three-dimensional tomographic data of the subject in a front view in a direction parallel to the depth direction of the subject. 4. The third image obtained by volume rendering from the viewpoint of is displayed side by side on the display unit, displayed in a switchable manner, or displayed in an overlapping manner, according to any one of claims 1 to 3 . Image processing device. generating a first front image that is a motion contrast front image using at least part of the three-dimensional motion contrast data of the subject; and using at least part of the three-dimensional motion contrast data to generate the first front image. a generation unit that generates a second front image that is generated by a method different from the front image and that enables identification of a relative positional relationship in the depth direction of a plurality of intersecting blood vessels of the subject; ,
a display control unit that causes a display unit to display at least one of the first front image and the second front image;
with
The image processing apparatus, wherein the second front image is an image obtained using at least part of the three-dimensional motion contrast data and color information according to the position of the subject in the depth direction. 11. The image processing apparatus according to claim 10 , wherein said motion contrast front image is an image without color information. 12. The display control unit causes the display unit to display an image of a portion of the second front image corresponding to the user's instruction in parallel with or superimposed on the motion contrast front image. The image processing device according to any one of the items. an acquisition unit that acquires three-dimensional motion contrast data of a subject;
a generation unit that uses at least a portion of the three-dimensional motion contrast data to generate a two-dimensional front image in which a relative positional relationship in the depth direction of a plurality of intersecting blood vessels of the subject can be identified;
a display control unit for displaying the two-dimensional front image on a display unit;
with
The two-dimensional front image is an image obtained by volume rendering at least part of the three-dimensional motion contrast data from a front view viewpoint in a direction intersecting the depth direction of the subject . Image processing device. 14. The 2D front image according to claim 13 , wherein the generator generates the 2D front image using the 3D motion contrast data in a range corresponding to a depth range for generating the motion contrast front image of the subject. image processing device. an acquisition unit that acquires three-dimensional motion contrast data of a subject;
a generation unit that uses at least a portion of the three-dimensional motion contrast data to generate a two-dimensional front image in which a relative positional relationship in the depth direction of a plurality of intersecting blood vessels of the subject can be identified;
a display control unit for displaying the two-dimensional front image on a display unit;
with
The image processing apparatus, wherein the two-dimensional front image is an image obtained using at least part of the three-dimensional motion contrast data and color information according to the position of the subject in the depth direction. 13. The display control unit causes the display unit to display an image of a portion of the two-dimensional front image corresponding to a user's instruction in parallel with or superimposed on the motion contrast front image of the subject. 16. The image processing apparatus according to any one of items 1 to 15 . generating a first frontal image that is a motion contrast frontal image using at least a portion of the three-dimensional motion contrast data of the subject;
A second frontal image generated by a method different from that of the first frontal image using at least a part of the three-dimensional motion contrast data, the second frontal image obtained in the depth direction of a plurality of intersecting blood vessels of the subject. generating the second front image in which the relative positional relationship is identifiable;
displaying at least one of the first front image and the second front image on a display unit;
including
The second front image is an image obtained by volume rendering at least part of the three-dimensional motion contrast data from a front view viewpoint in a direction intersecting the depth direction of the subject. Processing method. generating a first frontal image that is a motion contrast frontal image using at least a portion of the three-dimensional motion contrast data of the subject;
A second frontal image generated by a method different from that of the first frontal image using at least a part of the three-dimensional motion contrast data, the second frontal image obtained in the depth direction of a plurality of intersecting blood vessels of the subject. generating the second front image in which the relative positional relationship is identifiable;
displaying at least one of the first front image and the second front image on a display unit;
including
The image processing method, wherein the second front image is an image obtained using at least part of the three-dimensional motion contrast data and color information according to the position of the subject in the depth direction. acquiring three-dimensional motion contrast data of a subject;
using at least part of the three-dimensional motion contrast data to generate a two-dimensional front image in which the relative positional relationship in the depth direction of a plurality of intersecting blood vessels of the subject can be identified;
a step of displaying the two-dimensional front image on a display unit;
including
The two-dimensional frontal image is an image obtained by volume rendering at least part of the three-dimensional motion contrast data from a frontal view point in a direction intersecting the depth direction of the subject. Processing method. acquiring three-dimensional motion contrast data of a subject;
using at least part of the three-dimensional motion contrast data to generate a two-dimensional front image in which the relative positional relationship in the depth direction of a plurality of intersecting blood vessels of the subject can be identified;
a step of displaying the two-dimensional front image on a display unit;
including
The image processing method, wherein the two-dimensional front image is an image obtained using at least part of the three-dimensional motion contrast data and color information according to the position of the subject in the depth direction. A program that, when executed by a processor, causes the processor to perform the steps of the image processing method according to any one of claims 17 to 20 .

Images