JP2019195586A - Image processing device, image processing method, and program - Google Patents

Abstract

To enable circulation of a blood flow, for example, to be evaluated by using a plurality of pieces of motion contrast data captured at different time.SOLUTION: An image processing device acquires a plurality of motion contrast images related to the same position of the ocular fundus at different time, calculates a value indicating variations in brightness related to a plurality of pixels corresponding to the plurality of acquired motion contrast images, generates an output image on the basis of the calculated value indicating variations in brightness, and outputs the generated output image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  An ophthalmic tomographic imaging apparatus such as an optical coherence tomography (OCT) can observe the state inside the retinal layer three-dimensionally. Such a tomographic imaging apparatus has attracted attention in recent years because it is useful for more accurately diagnosing diseases in the eye. As a form of OCT, for example, as a method for acquiring an image at high speed, SD-OCT (Spectral domain OCT) is known in which a broadband light source is used and an interferogram is acquired with a spectroscope. Also, SS-OCT (Swept Source OCT) is known in which spectral interference is measured with a single channel photodetector by using a high-speed wavelength swept light source.

  In recent years, an angiography method using OCT (OCTA: OCT Angiography) has been proposed as an angiography method using no contrast agent. In OCTA, a blood vessel image (hereinafter referred to as an OCTA image) is generated by projecting three-dimensional motion contrast data acquired by OCT onto a two-dimensional plane. The motion contrast data is data obtained by repeatedly photographing the same cross section of the measurement object by OCT and detecting a temporal change of the measurement object during the imaging. The motion contrast data can be obtained, for example, by calculating the phase, vector, and intensity temporal change of the complex OCT signal from the difference, ratio, or correlation. Japanese Patent Application Laid-Open No. 2004-228688 discloses a technique for acquiring position information of a blood vessel by processing motion contrast data and acquiring analysis information about the blood vessel based on the position information.

JP, 2006-010658, A

  The motion contrast data acquired by OCT is imaged at locations where there is a change in time, so that it is possible to acquire an image relating to a blood vessel portion where blood flows. Therefore, blood vessels can be analyzed by using OCTA images, and support for diagnosing the degree of illness can be provided. However, conventionally, the physical size of a blood vessel, such as dimensions, area, and volume, is analyzed, and the blood flow in the blood vessel is not evaluated.

  An object of the present invention is to enable evaluation of circulation such as blood flow by using a plurality of motion contrast data photographed at different times.

  In order to achieve the above object, an image processing system according to an aspect of the present invention comprises the following arrangement.

An image processing apparatus according to an aspect of the present invention includes:
Acquisition means for acquiring a plurality of motion contrast images relating to the same position of the fundus at different times,
Calculating means for calculating a value indicating a variation in luminance for a plurality of corresponding pixels of the plurality of motion contrast images;
Generating means for generating an output image based on a value indicating the variation in luminance calculated by the calculating means;
Output means for outputting the output image generated by the generating means.

  According to the present invention, it is possible to evaluate circulation such as blood flow.

1 is a diagram illustrating a configuration example of an image processing system according to a first embodiment. The figure explaining the structure of an eye part, a tomogram, and a fundus image. 6 is a flowchart showing processing by the image processing apparatus. 6 is a flowchart showing processing by the image processing apparatus. The figure for demonstrating the production | generation of motion contrast data. The figure for demonstrating the removal of an artifact. The figure for demonstrating 1st position alignment. The figure for demonstrating 2nd alignment. The figure for demonstrating fluctuation data. The figure for demonstrating the screen which displays an image. The figure for demonstrating the screen which displays an image. The figure which shows an example of the display of data. The figure for demonstrating the analysis of data. The figure for demonstrating the analysis of data. The figure which shows the structural example of the image processing system by 3rd Embodiment. The flowchart which shows the process by 3rd Embodiment. The figure for demonstrating the screen which displays an image.

  Hereinafter, some embodiments of the present invention will be described with reference to the accompanying drawings. The image processing apparatus according to each of the following embodiments aligns a plurality of motion contrast data and calculates temporal variations of the motion contrast data when generating two-dimensional motion contrast data with reduced artifacts. . According to each embodiment, it is possible to acquire high-quality two-dimensional motion contrast data even when an artifact exists in the motion contrast data due to fixation fine movement or the like. Furthermore, changes over time can be displayed. Note that high image quality refers to an image in which the S / N ratio is improved as compared with one-time shooting or an image in which the amount of information necessary for diagnosis is increased. Hereinafter, the details of the image processing system including the image processing apparatus according to each embodiment will be described.

<First Embodiment>
FIG. 1 is a block diagram illustrating a configuration example of an image processing system 100 including an image processing apparatus 300 according to the first embodiment. In the example of FIG. 1, a tomographic imaging apparatus (also referred to as OCT) 200, a fundus imaging apparatus 400, an external storage unit 500, a display unit 600, and an input unit 700 are connected to the image processing apparatus 300 via an interface. .

  The tomographic imaging apparatus 200 is an apparatus that captures a tomographic image of the eye, and includes, for example, SD-OCT and SS-OCT. A known device can be used for the tomographic imaging apparatus 200. In the tomographic imaging apparatus 200, the galvanometer mirror 201 scans the fundus of the measurement light and defines the fundus imaging range by OCT. The drive control unit 202 controls the driving range and speed of the galvanometer mirror 201, thereby defining the imaging range and the number of scanning lines (scanning speed in the plane direction) on the fundus. Here, for the sake of simplicity, the galvanometer mirror 201 is shown as one unit. However, in reality, the galvanometer mirror 201 is configured by an X scan mirror (X scanner) and a Y scan mirror (Y scanner), and a desired range on the fundus. With this, the measuring light can be scanned. The focus 203 uses a focus lens (not shown) to focus measurement light in OCT on the retinal layer of the fundus through the anterior segment of the subject eye.

  The internal fixation lamp 204 includes a display unit 241 and a lens 242. The display unit 241 includes a plurality of light emitting diodes (LD) arranged in a matrix. The lighting position of the light emitting diode is changed according to the part to be photographed under the control of the drive control unit 202. The light from the display unit 241 is guided to the eye to be examined through the lens 242. The light emitted from the display unit 241 is 520 nm, and a desired pattern is displayed by the drive control unit 202. The coherence gate stage 205 is controlled by the drive control unit 202 in order to cope with a difference in the axial length of the eye to be examined. The coherence gate represents a position where the optical distances of the measurement light and the reference light in OCT are equal. By controlling the position of the coherence gate, the imaging position in the depth direction of the tomographic image, such as imaging of the retinal layer or imaging of the deeper side than the retinal layer, is controlled.

  The fundus image capturing apparatus 400 is an apparatus that captures a fundus image of an eye part, and includes, for example, a fundus camera, an SLO (Scanning Laser Ophthalmoscope) and the like.

  Here, the structure of the eye to be imaged by the image processing system 100 and the image acquired by the image processing system 100 will be described with reference to FIG. FIG. 2A shows a schematic diagram of an eyeball. In FIG. 2A, C represents the cornea, CL represents the lens, V represents the vitreous body, M represents the macula (the central portion of the macula represents the fovea), and D represents the optic papilla. In the following, a case where the posterior pole part of the retina including the vitreous body, the macula part, and the optic papilla is mainly illustrated will be exemplified, but the tomographic imaging apparatus 200 may photograph the anterior eye part of the cornea and the lens. It goes without saying that it is possible.

  FIG. 2B shows an example of a tomographic image acquired when the tomographic imaging apparatus 200 images the retina. In FIG. 2B, AS represents an image acquisition unit in an OCT tomographic image called A scan. By scanning this A scan in the x-axis direction in the figure, one B scan is formed. This B-scan is called a tomographic image (or tomographic image). In FIG. 2 (b), Ve represents a blood vessel, V represents a vitreous body, M represents a macular region, D represents an optic papilla, and La represents a posterior surface of a sieve plate. L1 is the boundary between the inner boundary membrane (ILM) and the nerve fiber layer (NFL), L2 is the boundary between the nerve fiber layer and the ganglion cell layer (GCL), and L3 is the joint between the inner and outer segments of photoreceptor cells (ISOS). ), L4 represents the retinal pigment epithelial layer (RPE), L5 represents the Bruch's membrane (BM), and L6 represents the choroid. In the tomographic image, the horizontal axis (OCT main scanning direction) is the x-axis, and the vertical axis (depth direction) is the z-axis.

  FIG. 2C illustrates an example of a fundus image acquired by the fundus image capturing apparatus 400. In FIG. 2C, M represents the macula, D represents the optic papilla, and the thick curve represents the blood vessels of the retina. In the fundus image, the horizontal axis (OCT main scanning direction) is the x-axis, and the vertical axis (OCT sub-scanning direction) is the y-axis. Note that the device configurations of the tomographic imaging apparatus 200 and the fundus imaging apparatus 400 may be an integrated type or a separate type.

  Returning to FIG. 1, the image processing apparatus 300 includes an image acquisition unit 301, a storage unit 302, an image processing unit 303, an instruction unit 304, and a display control unit 305. In the image acquisition unit 301, the tomographic image generation unit 311 acquires signal data of a tomographic image captured by the tomographic imaging apparatus 200, and generates a tomographic image by performing signal processing. The motion contrast data generation unit 312 generates motion contrast data. The image acquisition unit 301 acquires fundus image data captured by the fundus image capturing apparatus 400. The tomographic image generated by the image acquisition unit 301 and the acquired fundus image are stored in the storage unit 302.

  In the image processing unit 303, the preprocessing unit 331 performs processing for removing artifacts from the motion contrast data. The image generation unit 332 generates a two-dimensional motion contrast front image (also referred to as an OCTA image) from the three-dimensional motion contrast data, and a two-dimensional front image (also referred to as an Enfac image) from the three-dimensional tomographic image. The detection unit 333 detects the boundary line of each layer from the retina. The first alignment unit 334 performs alignment of the two-dimensional front image. The selection unit 335 selects reference data from the result of the first alignment unit 334. The second alignment unit 336 performs alignment in the lateral direction (x axis) of the retina using the OCTA image. The calculation unit 340 calculates a statistical value (average value, standard deviation, coefficient of variation, maximum value, minimum value, etc.) between the plurality of OCTA images. The fluctuation image generation unit 341 generates an image based on the statistical value obtained by the calculation unit 340.

  Each image generated by the image processing unit 303 is stored in the storage unit 302. The instruction unit 304 instructs the tomographic imaging apparatus 200 to drive. The display control unit 305 uses the images stored in the storage unit 302 to perform various types of display on the display unit 600 including displays as will be described later with reference to FIGS. 10 and 11. The external storage unit 500 stores information relating to the eye to be examined (patient name, age, sex, etc.), captured image data, imaging parameters, image analysis parameters, and parameters set by the operator in association with each other. The display unit 600 is a liquid crystal display, for example. The input unit 700 is, for example, a mouse, a keyboard, a touch operation screen, and the like, and an operator gives an instruction to the image processing apparatus 300, the tomographic image capturing apparatus 200, and the fundus image capturing apparatus 400 via the input unit 700.

  Next, processing of the image processing apparatus 300 according to the first embodiment will be described with reference to the flowcharts of FIGS. 3 and 4. FIG. 3A shows the overall operation processing of the image processing apparatus 300 according to the first embodiment. FIG. 3B shows a generation process of high-quality data (high quality image) in the first embodiment.

  In step S301, the image processing apparatus 300 acquires a subject identification number from the outside as information for identifying the eye to be examined. The subject identification number may be input by the user from the input unit 700, for example. The image processing apparatus 300 acquires information on the subject eye held by the external storage unit 500 based on the subject identification number and stores it in the storage unit 302.

  In step S <b> 302, the tomographic image capturing apparatus 200 scans the eye to be captured and captures a tomographic image. The scan of the eye to be examined is started in response to a scan start instruction from the operator. The tomographic imaging apparatus 200 controls the drive control unit 202 and operates the galvanometer mirror 201 to scan a tomographic image. As described above, the galvanometer mirror 201 includes the horizontal X scanner and the vertical Y scanner. When the orientations of these scanners are changed, scanning can be performed in the horizontal direction (X) and the vertical direction (Y) in the apparatus coordinate system. By changing the orientations of these scanners simultaneously, scanning can be performed in a direction in which the horizontal direction and the vertical direction are combined. Therefore, the galvano mirror 201 can scan the fundus plane in any direction. .

  Various radiographing parameters are adjusted when taking a tomographic image. For example, the indication unit 304 sets the display position of the individual lamp by the internal fixation lamp 204, the scan range and scan pattern by the galvanometer mirror 201, the position of the coherence gate by the coherence gate stage 205, and the focus position of the focus 203. The drive control unit 202 controls the light emitting diode of the display unit 241 to control the display position of the individual lamp by the internal fixation lamp 204 so as to perform imaging on the center of the macula or the optic disc. Examples of scan patterns that can be set on the galvanometer mirror 201 include raster scans that capture a three-dimensional volume, radial scans, and cross scans. After the adjustment of the shooting parameters, the operator selects shooting start to start shooting.

  In the present embodiment, a raster scan for photographing a three-dimensional volume is used as a scan pattern, and the three-dimensional volume is photographed N times (N ≧ 2) in order to generate high-quality data. In the N repeated shootings, the same shooting range is shot with the same scan pattern. For example, photographing is repeatedly performed at an interval of 300 × 300 (main scanning × sub scanning) in a range of 3 mm × 3 mm. In the three-dimensional volume, the same line portion is repeatedly photographed m times (m ≧ 2) in order to calculate the motion contrast. That is, when m is twice, 300 × 600 data is actually captured, and 300 × 300 three-dimensional motion contrast data is generated therefrom.

  Although detailed description is omitted in the present embodiment, the tomographic imaging apparatus 200 performs tracking of the eye to be imaged in order to image the same place for addition averaging, thereby reducing the influence of fixation micromotion. And scan the eye. Further, when motion that becomes an artifact in detecting an image such as blinking is detected, rescan is automatically performed at the place where the artifact occurs.

  In this embodiment, a plurality of motion contrast data photographed at different times are used. At this time, the tomographic imaging apparatus 200 may control tomographic imaging so that the imaging interval is a constant interval. For example, the shooting interval for acquiring one motion contrast data is automatically controlled according to the scan density, such as 3 seconds, 5 seconds, etc., so that N repeated shootings are automatically performed. May be. Alternatively, the operator may be presented with an index for photographing at regular intervals. The indicator may be any indicator that notifies the operator of the timing to start shooting. For example, a display that allows the operator to recognize the passage of time, such as a countdown at the start of shooting or how many seconds have passed since the previous shooting, can be used as an index.

  In step S303, the tomographic image generation unit 311 generates a tomographic image by performing reconstruction processing on each interference signal. First, the tomographic image generation unit 311 performs fixed pattern noise removal from the interference signal. The fixed pattern noise removal is performed, for example, by extracting fixed pattern noise by averaging a plurality of detected A scan signals, and subtracting this from the input interference signal. Next, the tomographic image generation unit 311 performs desired window function processing in order to optimize the depth resolution and dynamic range that are in a trade-off relationship when Fourier transform is performed in a finite section. Next, the tomographic image generation unit 311 generates a tomographic signal by performing FFT processing.

  In step S304, the motion contrast data generation unit 312 generates motion contrast data. Generation of motion contrast data will be described with reference to FIG. In FIG. 5, MC indicates three-dimensional motion contrast data, and LMC indicates two-dimensional motion contrast data constituting the three-dimensional motion contrast data. Furthermore, FIG. 5 shows that two-dimensional motion contrast data LMC is generated based on m two-dimensional tomographic images (m = 3 in this example). Hereinafter, a method for generating the two-dimensional motion contrast data LMC will be described.

  The motion contrast data generation unit 312 first corrects a positional deviation between a plurality of tomographic images captured in the same range of the eye to be examined. The method for correcting the misalignment may be any method. For example, the motion contrast data generation unit 312 aligns the tomographic image data corresponding to the same location obtained by photographing the same range m times using the features of the fundus shape. Specifically, one of the m pieces of tomographic image data is selected as a template, the degree of similarity with other tomographic image data is obtained while changing the position and angle of the template, and the amount of positional deviation from the template is obtained. . Thereafter, the motion contrast data generation unit 312 corrects each tomographic image data based on the obtained positional deviation amount.

Next, the motion contrast data generation unit 312 obtains a decorrelation value M (x, z) by the following [Equation 1] between two tomographic image data in which the imaging times related to the respective tomographic image data are continuous.
In [Equation 1], A (x, z) represents the luminance at the position (x, z) of the tomographic image data A, and B (x, z) represents the luminance at the same position (x, z) of the tomographic image data B. ing.

  The decorrelation value M (x, z) is a value from 0 to 1, and the value of M (x, z) increases as the difference between the two luminances increases. When the number of tomographic data repeatedly acquired at the same position (number of repetitions m) is 3 or more, the motion contrast data generation unit 312 has a plurality of decorrelation values M (x, z) at the same position (x, z). Can be requested. The motion contrast data generation unit 312 can generate final motion contrast data by performing statistical processing such as the maximum value calculation and the average calculation of the obtained plurality of decorrelation values M (x, z). it can. When the number of repetitions m is 2, the obtained decorrelation value M (x, z) is one. In this case, statistical processing such as the maximum value calculation and the average calculation is not performed, and the decorrelation value M (x, z) between the adjacent tomographic images A and B is the motion contrast at the position (x, z). Value.

  The motion contrast calculation formula shown in [Equation 1] tends to be easily affected by noise. For example, when there is noise in a non-signal portion of a plurality of tomographic image data and the values are different from each other, the decorrelation value becomes high and the noise is also superimposed on the motion contrast image. In order to avoid this, the motion contrast data generation unit 312 may regard tomographic data that falls below a predetermined threshold value as noise and replace it with zero as preprocessing. Thereby, the image generation part 332 can generate | occur | produce the motion contrast image which reduced the influence of noise based on the produced | generated motion contrast data. Then, by repeating the above-described processing for the tomographic images obtained in steps S302 to S303, N pieces of three-dimensional motion contrast data are acquired.

  In step S305, the image processing unit 303 generates high-quality data using the acquired three-dimensional motion contrast data. High-quality data generation processing by the image processing unit 303 will be described with reference to the flowcharts of FIGS. 3B and 4A.

  In step S351 in FIG. 3B, the detection unit 333 detects the boundary lines of the retinal layer in the plurality of tomographic images captured by the tomographic image capturing apparatus 200. The detection unit 333 detects, for example, each boundary of L1 to L6 or a GCL / IPL, IPL / INL, INL / OPL, and OPL / ONL boundary (not shown) in the tomographic image of FIG. An example of the boundary detection process is as follows. First, the detection unit 333 applies the median filter and the Sobel filter to the tomographic image to be processed, and creates an image (hereinafter referred to as a median image and a Sobel image). Next, the detection unit 333 creates a profile for each A scan from the created median image and Sobel image. A brightness value profile is created from the median image, and a gradient profile is created from the Sobel image. Then, the detection unit 333 detects a peak in the profile created from the Sobel image. The detection unit 333 detects the boundary of each region of the retinal layer by referring to the profile of the median image before and after the detected peak and between the peaks. Note that the method of detecting the boundary line in the retinal layer is not limited to the above.

  Next, in step S352, the image generation unit 332 projects motion contrast data corresponding to the range between the upper end of the generation range and the lower end of the generation range specified for the three-dimensional motion contrast data on the two-dimensional plane, and OCTA. Generate an image. For example, the image generation unit 332 performs processing such as average value projection (AIP) or maximum value projection (MIP) on motion contrast data between the upper end of the generation range and the lower end of the generation range of the entire motion contrast data. As a result, an OCTA image that is an Enface image (front image) of the three-dimensional motion contrast data is generated. That is, in step S352, a plurality of (N) motion contrast images (OCTA images) relating to the same position of the fundus at different times are acquired. The OCTA image generation method is not limited to the projection using the average value or the maximum value. An OCTA image may be generated with values such as a minimum value, median value, variance, standard deviation, and sum.

  In this embodiment, the upper end of the generation range is the ILM / NFL boundary line, the lower end of the generation range is the boundary line at the lower end of 50 μm from the GCL / IPL, and the OCTA image is generated by the average value projection method. The motion contrast data generation unit 312 may generate motion contrast data using tomographic data in a range between the upper end of the generation range and the lower end of the generation range. In this case, since motion contrast data between the upper end of the generation range and the lower end of the generation range is obtained, the image generation unit 332 does not need to consider the range between the upper end of the generation range and the lower end of the generation range. That is, the image generation unit 332 performs an average value projection or the like using the entire motion contrast data generated by the motion contrast data generation unit 312, so that the OCTA image based on the tomographic data in the specified depth range. Can be generated.

  Next, in step S353, the first alignment unit 334 determines the horizontal direction (x axis) and the vertical direction (y axis) of the N OCTA images obtained based on the N three-dimensional volumes. , Rotation alignment in the xy plane is performed. This alignment process will be described with reference to the flowchart of FIG.

  In step S3531 in FIG. 4A, a reduction process for reducing artifacts is performed on a plurality of motion contrast images (OCTA images). For example, the preprocessing unit 331 detects an artifact such as a black belt or a white line from the OCTA image generated by the image generation unit 332, and removes the detected artifact. This will be described with reference to FIG. In FIG. 6, the black area of the OCTA image represents a place where the decorrelation value is high, that is, a place where blood flow is detected (corresponding to a blood vessel), and the white area represents a place where the decorrelation value is low. BB in FIG. 6A is an example of a black belt, and WL in FIG. 6B is an example of a white line. The black belt is a phenomenon in which the brightness value of the retinal tomogram decreases and the decorrelation value decreases due to the movement of the retina away from a highly sensitive position due to movement during shooting, and the entire image becomes dark due to blinking, etc. It occurs when the value of decorrelation becomes low. The white line is calculated by aligning M tomographic images in the calculation of decorrelation. If the alignment is not successful or if the position cannot be corrected by the alignment, the entire image is decorrelated. Occurs when the value increases. Since these artifacts occur in the calculation of decorrelation, they occur in units of one line in the main scanning direction. Therefore, the preprocessing unit 331 detects artifacts in units of one line.

For example, black band detection is performed when the average value of decorrelation in one line is _{equal to} or less than the threshold value TH _{AVG_B} . The white line detection is a white line when the average value of the decorrelation values in one line is _{equal to} or greater than the threshold value TH _{AVG_W} and the standard deviation (or variance value) is _{equal to} or less than TH _{SD_W} . Since the correlation value may be high in large blood vessels, etc., if white line detection determination is performed using only the average value, a region containing blood vessels with high correlation values such as these may be erroneously detected as a white line. is there. For this reason, in the present embodiment, it is determined whether or not a white line is combined with an index for evaluating variation in values such as standard deviation and variance. In one line including a blood vessel having a high decorrelation value, the average value is high and the standard deviation is also high. On the other hand, in the white line, the average value is high but the variation in value is small, so the standard deviation is low. Therefore, it is possible to reduce false detection by performing white line detection based on the average value and variation of the decorrelation.

  The preprocessing unit 331 stores the artifact area obtained above in a Mask image corresponding to the OCTA image. In the Mask image of FIG. 6, an example is shown in which a white area is set to 1 and a black area is set to 0. In the OCTA image, the value of the decorrelation value varies depending on whether the eye is a normal eye or an affected eye, and depending on the type of disease of the affected eye. For this reason, it is desirable that the threshold values used for the black band detection and the white line detection described above are set for each image. For example, it is desirable to set a threshold according to the brightness of the OCTA image using a dynamic threshold method such as P-tile or discriminant analysis. When the dynamic threshold method is used, an upper limit threshold and a lower limit threshold are set in advance, and when the value exceeds or falls below these values, the upper limit threshold or the lower limit threshold is set as the threshold.

  In step S3532, the first alignment unit 334 initializes a two-dimensional matrix for storing alignment parameters when the OCTA images are aligned. In each matrix element, information necessary for high image quality such as deformation parameters and image similarity at the time of alignment is collectively stored.

  In step S3533, the first alignment unit 334 selects an OCTA image to be aligned. In the present embodiment, all the N OCTA images are sequentially set as an image serving as a reference for alignment (hereinafter referred to as a target image), and the OCTA image set as the target image and the remaining OCTA images are aligned. Performed (alignment is performed in step S3534). For example, when a Data0 OCTA image is selected as a target image in step S3533, alignment is performed with each of the Data1 to Data (N-1) OCTA images. Next, when an OCTA image of Data1 is selected as a target image, alignment with each of the OCTA images of Data2 to Data (N-1) is performed. Next, when an OCTA image of Data2 is selected as a target image, alignment with each of the OCTA images of Data3 to Data (N-1) is performed. An example of this is shown in FIG. For simplicity, Data0 to Data2 are shown in FIG. 7A. However, when a three-dimensional volume is imaged N times, alignment is performed between N OCTA images.

  When the Data number of the OCTA image that is the target image is incremented by one, the start Data number of the OCTA image that is the target of alignment is also increased by one. For example, when the OC2 image of Data2 is used as a reference, the alignment of the OCTA images of Data0 and Data1, Data0 and Data2, and Data1 and Data2 has already been completed. This is because the alignment for the combination of the OCTA images is performed by the processing up to that point (alignment processing based on the OCTA images of Data0 and Data1). Therefore, when the OCTA image of Data2 is used as a reference, alignment is performed from Data3. Thereby, although it is the alignment of all the OCTA images, a half combination may be calculated.

  In step S3534, the first alignment unit 334 determines that the horizontal direction (x-axis) and the vertical direction (y-axis) of the image between the OCTA image selected as the target image in step S3533 and another OCTA image, xy. Perform in-plane rotational alignment. In the alignment between the OCTA images, in order to perform the sub-pixel alignment in the xy plane, the alignment is performed by enlarging the size of the OCTA image. This is because the subpixel alignment is expected to improve the alignment accuracy over the pixel alignment. For example, when the photographing size of the OCTA image is 300 × 300, it is enlarged to 600 × 600. When enlarging, an interpolation method such as Bicubic or Lanczos (n) method is used. And as an alignment process between images, for example, an evaluation function representing the similarity between two OCTA images is defined in advance, and an evaluation value is calculated while shifting or rotating the OCTA image position, The location where the evaluation value is the best is taken as the alignment result. Examples of the evaluation function include a method of evaluating with a pixel value (for example, a method of performing evaluation using a correlation coefficient). An expression in the case of using a correlation coefficient as an evaluation function representing the similarity S is shown in [Equation 2].

In [Expression 2], the area of the Data0 OCTA image is f (x, y), and the area of the Data1 OCTA image is g (x, y). f _ave and g _ave represent the average of the region f (x, y) and the region g (x, y), respectively. Here, the area is an image area used for alignment, and an area equal to or smaller than the size of the normal OCTA image is set, and the size of the ROI is set.

  The evaluation function is not limited to [Equation 2], and may be SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference), as long as the similarity or difference of images can be evaluated. Alternatively, alignment may be performed by a method such as POC (Phase Only Correlation). Through this processing, global alignment within the XY plane is performed.

  In addition, although the example which expands the size of an OCTA image here and performs the alignment was shown, it does not restrict to this. Further, when the input OCTA image size is a high-density scan such as 900 × 900, enlargement is not necessarily required. Further, in order to perform alignment at high speed, the alignment may be performed by generating pyramid structure data.

Next, in step S3535, the first alignment unit 334 calculates an image evaluation value of the OCTA image selected as the target image. The image evaluation value is calculated using the common area of the image that does not include the invalid area generated by the alignment in the OCTA image that has been two-dimensionally aligned in step S3534. For example, the image evaluation value Q can be obtained by [Equation 3].

In [Expression 3], the area of the Data0 OCTA image is f (x, y), and the area of the Data1 OCTA image is g (x, y). The first term represents the correlation coefficient and is the same as the equation shown in [Expression 2]. Therefore, σ _f and σ _g in the formula correspond to those shown in [Equation 2], respectively. The second term is a term for evaluating brightness, and f _ave and g _ave represent the average of the region f (x, y) and the region g (x, y), respectively. The third term is a term for evaluating contrast. Each term has a minimum value of 0 and a maximum value of 1. For example, if Data0 and Data1 are the same image, the evaluation value is 1. Therefore, the evaluation value is high when an average image among N OCTA images is used as a reference, and the evaluation value is low when an OCTA image that is different from other OCTA images is used as a reference. Here, being different from other OCTA images is a case where the photographing position is different, the image is distorted, the entire image is dark or too bright, and artifacts such as white lines and black bands are included. Since Q based on [Equation 3] is calculated for one target image with other N-1 OCTA images, the sum of these Qs is used as an evaluation value. Note that it is not always necessary to use [Equation 3] as the image evaluation value, and each term may be evaluated independently or the combination may be changed.

  In step S3536, the first alignment unit 334 stores values in a two-dimensional matrix for storing parameters necessary for image quality improvement such as alignment and image similarity initialized in step S3532. . For example, when the target image is Data0 and is aligned with the Data1 image, the horizontal alignment parameter X, the vertical alignment parameter Y, and the XY plane are aligned with the elements (0, 1) of the two-dimensional matrix. The rotation parameter α, the image evaluation value, and the image similarity are stored. In addition to these pieces of information, the Mask image shown in FIG. 6 is stored in association with the OCTA image. Further, although not described in the present embodiment, when magnification correction is performed, the magnification may be stored.

  In step S3537, the first alignment unit 334 determines whether or not all OCTA images have been aligned with the remaining OCTA images as target images. If processing has not been performed on all OCTA images, the process returns to step S3533. If processing has been performed using all OCTA images as a reference, the process proceeds to step S3538.

  In step S3538, the first alignment unit 334 updates the remaining elements of the two-dimensional matrix. As described in steps S3533 to S3537, calculation is performed only for a half combination of N OCTA images. Therefore, these values are copied to the elements that are not calculated. For example, the parameter of the element (0, 1) of the two-dimensional matrix is copied to the element (1, 0). That is, the first alignment unit 334 copies the element (i, j) to the element (j, i). At this time, since the alignment parameters X and Y and the rotation parameter α are reversed, copying is performed by multiplying a negative value. Since the image similarity and the like are not reversed, the same value is copied as it is. The OCTA image alignment is performed by these processes.

Returning to FIG. 3B, in step S354, the selection unit 335 selects a reference image from the N OCTA images based on the result of the alignment performed in step S353. Information necessary for generating a high-quality image is stored in each element of the two-dimensional matrix generated in step S353, and the selection unit 335 uses the information to select a reference image. The selection unit 335 uses, for example, an image evaluation value, an alignment parameter evaluation value, and an artifact region evaluation value in selecting a reference image. As the image evaluation value, the value obtained in step S3535 (for example, the value Q obtained by [Equation 3]) is used. The alignment parameter evaluation value is acquired by, for example, [Equation 4] using X and Y of the alignment result obtained in step S3534. In [Equation 4], the larger the movement amount, the larger the value.

The artifact region evaluation value is calculated by, for example, [Equation 5] using the Mask image obtained in step S3531. In [Equation 5], T _{(x, y)} represents a pixel in a region that is not an artifact in the Mask image, and A _{(x, y)} represents all the pixels in the Mask image. Therefore, when there is no artifact, the artifact area evaluation value NA is the maximum value (NA = 1).

  The image evaluation value Q and the artifact area evaluation value NA should be larger in numerical value, and the alignment parameter evaluation value SV should be smaller. Further, since the image evaluation value and the alignment parameter evaluation value are values obtained in relation to other images when a certain image is used as a reference, they are N−1 total values. Since these evaluation values have different evaluation scales, the selection unit 335 sorts by each value, and selects a reference image based on the total value of the sorted indexes. For example, the selection unit 335 sorts the image evaluation value and the artifact area evaluation value so that the index after sorting decreases as the numerical value increases, and the index after sorting decreases as the numerical value of the alignment parameter evaluation value decreases. Sort as follows. Then, the selection unit 335 selects the OCTA image having the smallest index value after sorting as the reference image.

  In addition, about the method of selecting a reference image, although the example which selects a reference image by totaling a sort value was described, it is not restricted to this. The evaluation values may be calculated by assigning weights to the sorted indexes of the evaluation values. Alternatively, normalization may be performed so that each evaluation value becomes 1 instead of the sort value. For example, the image evaluation value is normalized to 1, but in this embodiment, since it is N−1 total values, an average value may be used.

Further, the alignment parameter evaluation value can be normalized to 1 if defined as in [Equation 6]. In this case, the evaluation value closer to 1 is a better evaluation value.

In Equation 6], SV _n is the N-1 sum of values obtained in Equation 4], n subscript corresponds to the Data number. Therefore, in the case of Data0, a SV _0. SV _max is the maximum alignment parameter evaluation value between Data0 and Data (N-1). α is a weight, and is a parameter for adjusting how many values of NSV _n are set when SV _n and SV _max are the same numerical value. The maximum value SV _max may be determined from actual data as described above, or may be defined in advance as a threshold value. Since the artifact region evaluation value is normalized to 0 to 1, it can be used as it is. As described above, when all the evaluation values are normalized to 1, the image having the largest evaluation value is selected as the reference image.

  As described here, the reference image is an average image among the N images, and the image that satisfies the condition that the movement amount is small and the artifacts are small when the other images are aligned is selected. Is done. An example of the reference image selected by this example is shown in FIG. In this example, Data1 is selected as the reference image. Data0 and Data2 are moved based on the alignment parameters obtained by the first alignment unit 334, respectively.

  Next, in step S355, the second alignment unit 336 performs alignment in the lateral direction (x-axis) of the retina using the OCTA image selected as the reference image in step S354. This will be described with reference to FIG. FIG. 8A illustrates an example in which the horizontal alignment of Data2 is performed when the reference image is Data1 and the alignment target is Data2. Mask sets 0 to an invalid area (vertical black line in the figure) generated by the movement of Data2 as a result of alignment with the artifact (horizontal black line in the figure) included in Data2 and alignment with Data1. It is. The second alignment unit 336 aligns the horizontal direction of each line with respect to the reference image and the alignment target image, and calculates the similarity in line units. For example, [Equation 2] can be used for calculating the similarity. Then, the second alignment unit 336 moves the line to a position where the similarity is maximized. Also, the similarity to the reference image is calculated in line units, and a weight is set to Mask according to the similarity.

  Another example of the alignment result by the second alignment unit 336 is shown in FIG. FIG. 8B shows an example in which the black image 801 and 802 in the horizontal direction are set in the Mask image as lines that are determined not to be similar to the reference image near the upper end of the image and near the center of the image and are not used for superimposition. Yes. Further, an example in which the vicinity of the center of the image and the lower end of the image are shifted to the right near the center and shifted to the left at the lower end of the image as a result of the alignment in line units. Since the invalid area is generated by shifting the image, the second alignment unit 336 sets 0 to the invalid areas 803 and 804 generated in Mask. By this processing, local alignment in the XY plane is performed.

  Note that the rotation parameter α obtained in the first alignment may be applied to each image before the second alignment, or may be applied after the second alignment. May be. In the above description, the second alignment unit 336 has described the alignment in the x direction. However, the second alignment unit 336 may perform the alignment in the y direction. For example, the similarity S according to [Equation 2] is calculated between a certain line of the Data 1 image and each line of the Data 2 image, and one image is arranged in the y direction so that the line having the largest similarity S corresponds. It may be possible to shift to

  In step S356, the calculation unit 340 calculates a statistical value (an average value, a standard deviation, a variation coefficient, a maximum value, a minimum value, etc.) from a plurality of (N) OCTA images. Here, the average value, standard deviation, and variation coefficient will be described. The calculation unit 340 first obtains an average value. The calculation unit 340 includes, for a plurality of OCTA images and a plurality of Mask images, a total value SUM_A obtained by multiplying the corresponding pixel values of the corresponding OCTA image and the Mask image, and a total value of the corresponding pixel values of the plurality of Mask images. Each SUM_B is held. In the Mask image, invalid areas removed as artifacts and invalid areas in which no data exists due to the alignment are stored as 0. Therefore, the total value SUM_B of the Mask image holds a different value for each pixel position. Usually, since movement of several tens of pixels is assumed for each XY in the alignment, when the number of data used for superposition is N, the pixel value of SUM_B near the center of the image is N, and The pixel value of SUM_B is a value smaller than N. Then, the average value of each pixel is obtained by dividing SUM_A by SUM_B. Thereby, an addition average image is generated. In the case of the present embodiment using the fundus OCTA image, the addition average image generated in this way is, for example, an image showing the blood vessel morphology of the fundus.

  Next, the standard deviation and the variation coefficient will be described. In the present embodiment, a variation image as an output image is generated based on a value (for example, standard deviation, variance, etc.) indicating variation in luminance (motion contrast value) in a plurality of corresponding pixels of a plurality of motion contrast images. . In the present embodiment, the value indicating variation is used after being normalized by a representative value of luminance (for example, the above average value or median value) in a plurality of corresponding pixels. In the following, an example will be described in which the luminance variation is obtained from the standard deviation, and the average value is used as a representative value for normalization. Although the data obtained by imaging the variation coefficient can be used as the variation image, in this embodiment, as described in step S306, the variation image from which the blood vessel portion is extracted by the variation image generation unit 341 is generated. .

When calculating the standard deviation and the coefficient of variation, the mask image is also used to exclude pixels that are invalid areas from the calculation. Although the invalid area is excluded from the calculation using the Mask image, the standard deviation and the coefficient of variation cannot be calculated for pixels whose number of pixels used for the calculation is equal to or less than a threshold (for example, 1 or less). Therefore, for example, when it is set to 0, it can be seen that it is invalid so that it can be seen that it is an invalid area when displayed as a final image. The standard deviation can be obtained from the average value and the value of each pixel. The formula of standard deviation obtained from a plurality of OCTA images is shown in [Formula 7].

In Equation 7], SD is standard deviation, n represents the number of pixels used for calculation at each pixel, x _ave is an average value of each pixel, x _i represents each pixel value. Next, the equation of the variation coefficient is shown in [Equation 8].

  In [Equation 8], CV represents a coefficient of variation. That is, the value obtained by dividing the standard deviation by the average value. With standard deviation, pixels with a larger average value tend to be larger, but with the coefficient of variation, even if the average value varies among pixels, the degree of variation must be compared. Is possible.

  Thus, by calculating the coefficient of variation between OCTA images, it is possible to evaluate the variation between images. For blood vessels with poor flow due to circulatory disturbances, there are cases where they appear and do not appear as blood vessels in the OCTA image each time an image is taken. That is, by using an index for evaluating the variation between images, such as a coefficient of variation (or standard deviation), it is possible to grasp whether the circulation is good or bad. FIG. 9 shows an example of an OCTA image relating to a blood vessel in a poor flow portion. FIG. 9A shows an example of images that are a plurality of OCTA images 1 to 3 taken at different times and that have already been aligned. And blood vessels 901-903 of OCTA images 1-3 are shown as an example of a blood vessel with a bad flow. When blood is flowing, it appears in the OCTA image as a blood vessel 901, but when the blood flow deteriorates, the value of motion contrast becomes small, so that it hardly appears like the blood vessels 902 and 903, or there is a reflection. Deteriorate. FIG. 9B shows an example in which the coefficient of variation is calculated using these OCTA images 1 to 3 and imaged. In FIG. 9B, since the value of the portion 910 becomes high as a portion corresponding to the blood vessels 901 to 903 having changes at different times, a change appears in this portion.

  Note that the calculation unit 340 may calculate the histogram after converting the histogram of each OCTA image to the same reference before calculating the statistical value as described above. The histogram of the OCTA image defines a standard deviation and an average value of a reference image, and performs conversion so that the standard deviation and the average value are defined for each OCTA image, thereby obtaining a constant average value and a standard value. The image has a deviation.

  Next, in step S357, the image generation unit 332 generates the average value (addition average image) calculated by the calculation unit 340 as a high-quality OCTA image. The above is the processing in step S305 in FIG.

  Next, in step S306, the fluctuation image generation unit 341 generates a fluctuation image based on the statistical value calculated by the calculation unit 340 in step S356. In the present embodiment, the fluctuation image generation unit 341 performs sharpening processing or enhancement processing on an addition average image (high-quality OCTA image) of a plurality of motion contrast images, and performs binarization processing to thereby perform blood vessel processing. An image showing the form is generated. Then, the variation image generation unit 341 generates a variation image by associating information (CV) indicating variation in motion contrast values in a plurality of motion contrast images with a region indicating a blood vessel included in the image indicating the shape of the blood vessel. To do. This variation image generation process will be described with reference to the flowchart of FIG.

  In step S361, the fluctuation image generation unit 341 performs sharpening processing or enhancement processing (hereinafter collectively referred to as enhancement processing) on the high-quality image (addition average image) generated in step S305. As the enhancement processing, for example, an Unsharp mask or a Hessian filter can be used. Thereby, the emphasis process of the location corresponding to the blood vessel in the OCTA image is performed. Note that one enhancement process may be used, and an AND process of each enhancement image obtained by performing a plurality of enhancement processes may be performed. For example, an enhanced image may be acquired using any one enhancement process of an Unsharp mask and a Hessian filter, or an enhanced image obtained by performing AND processing on each enhanced image obtained by performing these two enhancement processes. May be acquired.

  In step S362, the variation image generation unit 341 performs binarization processing on the enhanced OCTA image to generate a blood vessel mask image. In the binarization process, a fixed threshold value may be used, or a dynamic threshold value may be used. As a method for obtaining a dynamic threshold, it can be obtained by discriminant analysis method, P-tile method, Median, Mean, or the like. In addition, one threshold value may be set for the entire image, or a local threshold value may be set for each ROI by setting an ROI having a size smaller than the image size.

  Note that minute noise may be removed by applying a Deckle process, a Morphology operation, or the like to the mask image obtained by the binarization process. Although alignment is performed when performing high quality processing of an OCTA image, there may be a slight misalignment. In that case, since a variation value may appear in a portion corresponding to the fold (blood vessel wall) of the blood vessel, the portion corresponding to the blood vessel of the blood vessel mask is thinned by about several pixels by the shrinkage processing of the Morphology operation or the like. A mask may be created.

  In step S363, the variation image generation unit 341 corresponds to a blood vessel by taking a logical product (AND processing) of the data obtained by imaging the variation coefficient CV obtained by the calculation unit 340 and the mask image obtained in step S362. The data which imaged the coefficient of variation of the part to perform is generated. As described above, by using a blood vessel mask image, a variation coefficient image (also referred to as a variation image or variation data) of a portion corresponding to a blood vessel can be generated from data obtained by imaging the variation coefficient CV. Thereafter, the processing returns to the flowchart of FIG.

  Returning to FIG. 3A, in step S307, the display control unit 305 outputs the high-quality two-dimensional motion contrast data (addition average image) generated in step S305 and the fluctuation image generated in step S306 as an output image. Is displayed on the display unit 600. Note that the output form of the output image is not limited to display on the display unit 600. For example, an output unit that performs storage in the external storage unit 500 may be provided.

  Here, an example of a screen displayed on the display unit 600 by the display control unit 305 is shown in FIG. Reference numeral 1100 denotes an entire screen, 1101 denotes a patient tab, 1102 denotes an imaging tab, 1103 denotes a report tab, and 1104 denotes a setting tab. The diagonal lines in the report tab 1103 represent the active state of the report screen. In this embodiment, an example of displaying a report screen will be described. Reference numeral 1105 denotes a patient information display unit, 1106 denotes an examination sort tab, 1107 denotes an examination list, and a black frame of 1108 denotes selection of an examination list, and selected examination data is displayed on the screen.

  The inspection list 1107 in FIG. 10 displays SLOs and thumbnails of tomographic images, but is not limited thereto. In the case of OCTA imaging, although not shown, an OCTA thumbnail may be displayed. As an example of the thumbnail display, only the SLO and OCTA thumbnails and the OCTA thumbnail may be used as the tomographic image and the OCTA thumbnail. In the present embodiment, the inspection data acquired by photographing and the inspection data generated by the image quality enhancement process are displayed in a list in the inspection list 1107. In the inspection data generated by the image quality enhancement processing, the image displayed on the thumbnail may be displayed from the data that has been subjected to the image quality enhancement processing.

  Reference numerals 1130 and 1131 denote tabs in the view mode. By selecting the tab 1130, a two-dimensional OCTA image generated from the three-dimensional motion contrast data is displayed. Further, by selecting the tab 1131, a three-dimensional tomographic image and three-dimensional motion contrast data are displayed. In FIG. 10, the tab 1130 is selected.

  Reference numeral 1129 denotes a button for instructing execution of high-quality generation of motion contrast. In the present embodiment, when the button 1129 is pressed, the high-quality data generation process shown in step S305 is executed. This button 1129 is pressed to display data candidates used for improving image quality. Note that, when data that has been repeatedly shot under the same shooting conditions as the selection data is automatically selected, high-quality data generation may be executed without displaying data candidates. By displaying the generated high quality data on the examination list 1107, the user can select the high quality data as a report display. In the present embodiment, the image quality improvement processing has already been completed, and in FIG. 10, data that has undergone the image quality improvement processing is selected and displayed. Therefore, in the following description, data that has been subjected to high image quality processing (addition averaging) will be described.

  1109 is an SLO image, 1110 is a first OCTA image, 1111 is a first tomographic image, 1112 is a front image (Enface image) generated from a three-dimensional tomographic image, 1113 is a second OCTA image, and 1114 is a second image. Represents a tomographic image. Reference numeral 1120 denotes a tab for switching the image type of the Enface image 1112. The Enface image 1112 shows Enface created from the same depth range as the first OCTA image 1110, but it is also possible to switch to Enface created from the same depth range as the second OCTA image 1113 using the tab 1120. is there. Reference numeral 1115 denotes an image superimposed on the SLO image 1109, and 1116 denotes a tab for switching the type of the image 1115. Reference numeral 1117 denotes a tab for switching the type of OCTA image displayed as the first OCTA image 1110, and 1121 denotes a tab for switching the type of OCTA image displayed as the second OCTA image. As types of OCTA images, there are OCTA images created in a shallow layer, a deep layer, a choroid, etc., or an arbitrary range. Note that the upper end of the creation range of the first OCTA image 1110 is indicated by a display 1118 indicating the type of the boundary line and its offset value, and an upper boundary line 1125 displayed superimposed on the tomographic image. Further, the lower end of the creation range of the first OCTA image 1110 is indicated by a display 1119 indicating the type of the lower boundary line and its offset value, and a boundary line 1126 displayed superimposed on the first tomographic image. Similarly, regarding the creation range of the second OCTA image, the display 1123 and the boundary line 1127 indicate the upper end, and the display 1124 and the boundary line 1128 indicate the lower end.

  An arrow 1145 indicates the position of the first tomographic image in the XY plane, and an arrow 1146 indicates the position of the second tomographic image in the XY plane. The positions of the first tomographic image 1111 and the second tomographic image 1114 in the XY plane can be switched by operating the arrows 1145 and 1146 by dragging the mouse.

  In the present embodiment, image quality improvement processing is performed using a plurality of OCTA images, and at the same time, a variation image is generated using a plurality of OCTA images. Therefore, when selecting data that has been subjected to high image quality processing and displaying it on the screen, it is also possible to display a varying image. This example will be described with reference to FIG. FIG. 11 shows an example in which a variation image 1150 is displayed instead of the first OCTA image 1110. In FIG. 11, the variation image 1150 is shown as a substantially black image, but actually, the variation image described in this specification is displayed. The variation image 1150 used for display may be data obtained by imaging the variation coefficient CV calculated by [Equation 8], or data obtained by imaging the variation coefficient of a portion corresponding to a blood vessel by the process of FIG. 4B. But you can. A variation image 1150 is generated by adding a color corresponding to information indicating variation in motion contrast values to a region indicating a blood vessel (a blood vessel region indicated by a mask image) included in the addition average image. That is, the fluctuation image 1150 is a color image. Therefore, a bar 1151 indicating the color scale of the fluctuation image 1150 is displayed. The color scale is expressed in a cold color system (for example, blue) when the value of the variation coefficient is small, and in a warm color system (for example, red) when the value of the variation coefficient is large. When the fluctuation image 1150 is displayed, it may be displayed by switching to the OCTA image 1110 or may be superimposed on the OCTA image 1110. In the case of superimposed display, it is desirable to display the variable image 1150 with the transparency α set. On / Off of the display of the fluctuation image 1150 may be selected by a right-click menu (not shown), or the display may be switched by On / Off of a check box (not shown). In the menu selection or check box, when data for which no variation image is generated is selected and displayed, it is desirable that the selection item is not displayed or is semi-transparent and cannot be selected.

  Although FIG. 11 shows an example in which the fluctuation image is displayed by switching to the OCTA image or superimposed on the OCTA image, the present invention is not limited to this. Another display example of the fluctuation image 1150 is shown in FIG. FIG. 12A shows an example in which only the portion where the first OCTA image 1110 in FIG. 10 is displayed. FIG. 12A shows an example in which a variation portion having a predetermined threshold value or more is displayed in color on the first OCTA image 1210. FIG. 12A shows an example in which only the blood vessel 1201 (only the blood vessel region indicated by the blood vessel mask image generated in step S362) is displayed in color. In this case, information indicating variations in motion contrast values is not associated with regions other than the regions indicating blood vessels included in the addition average image (OCTA image). That is, in the variation image 1150 of FIG. 11, the variation coefficient is displayed for the entire image, but in FIG. 12A, the variation coefficient is displayed in color for a part of the image.

  FIG. 12B shows an example in which only the portion where the first OCTA image 1110 and the first tomographic image 1111 in FIG. 10 are displayed. FIG. 12B shows an example in which the variation coefficient 1240 is superimposed on the tomographic image 1111. The tomographic image 1111 is superimposed and displayed in a color corresponding to the variation image 1150. In the case where the tomographic image 1111 is displayed in a superimposed manner, a portion where the value of the variation coefficient is equal to or greater than a predetermined threshold is displayed in a superimposed manner. In the present embodiment, since the coefficient of variation is calculated with a two-dimensional image on the XY plane, the value of Z is not known when displayed on a tomographic image. Therefore, in this case, the same color is used for the upper and lower sides (boundary lines 1125 and 1126), which are the OCTA image creation range. Alternatively, since the tomographic image is used when obtaining the motion contrast value, the motion contrast value is equal to or greater than a predetermined threshold value and is displayed only at a location that satisfies both of the variation coefficient values equal to or greater than the predetermined threshold value. good.

  In the present embodiment, an example in which an image quality improvement process is performed using an OCTA image generated with the upper end of the generation range as the ILM / NFL boundary line and the lower end of the generation range within the boundary line of 50 μm lower end in the depth direction from GCL / IPL. I explained. Therefore, the fluctuation image 1150 also shows an image in the same range. However, the fluctuation image to be generated is not limited to this range. It is also possible to change the depth range of the OCTA image used for alignment and generate a fluctuation image in accordance with the range. Alternatively, alignment is performed with an OCTA image created in a depth range in which blood vessel characteristics can be easily expressed, and only the alignment parameter is applied to an OCTA image generated in a different depth range, thereby generating a fluctuation image in different depth ranges. Is also possible. That is, in FIG. 11, the fluctuation image 1150 is displayed on the first OCTA image, but the alignment parameter obtained when the first OCTA image is generated can be applied to the second OCTA image 1113. It is. Therefore, it is possible to perform both high image quality and variable image generation in the OCTA image 1113 in different depth ranges.

  Returning to FIG. 3A, in step S <b> 308, the instruction acquisition unit (not shown) acquires an instruction from the outside as to whether or not tomography or analysis of the tomographic image by the image processing system 100 is finished. This instruction is input by the operator using the input unit 700, for example. When an instruction to end the process is acquired, the image processing system 100 ends the process. On the other hand, if the photographing or analysis is continued without ending the process, the process returns to step S302 to continue. Thus, the processing of the image processing system 100 is performed.

  According to the configuration described above, in the present embodiment, the variation between images can be evaluated by calculating the coefficient of variation between OCTA images. With respect to a blood vessel that has a poor flow due to a circulatory disorder or the like, it is possible to grasp the circulatory state by displaying as an image an index for evaluating the variation between a plurality of images.

Second Embodiment
In 1st Embodiment, the example which shows the image for grasping | ascertaining the state of a circulation was shown. In the second embodiment, an example of analyzing a circulation state will be described. Below, a different part from 1st Embodiment is mainly demonstrated.

  The analysis in the second embodiment will be described with reference to FIG. FIGS. 13A to 13D show only portions where the first OCTA image 1110 in FIG. 10 is displayed, and show examples of different shapes of grids for analyzing the coefficient of variation. FIG. 13A shows an example of a grid 1301 that divides the upper, lower, left, and right into four with the macular at the center. For example, the size of the inner circle is 1 mm, and the size of the outer circle is 3 mm. FIG. 13B shows an example of a grid 1302 that divides the macula up and down into two. The size of the circle is the same as in FIG. When a wider range is photographed, the grid size may be 1 mm, 3 mm, or 6 mm.

  FIG. 13C shows an example of a grid 1303 that divides an image into nine. Note that the number of grid divisions is not limited to nine divisions, and may be, for example, 16 divisions. Alternatively, one grid size may be 1 mm, and the number of divisions may be changed according to the angle of view of the captured image. In this case, for example, when the shooting angle of view is 3 × 3 mm, it is divided into nine. FIG. 13D shows an example in which FAZ (Foveal Available Zone) is detected, and a grid 1304 is applied in a range of FAZ + several mm according to the shape. The grid around the FAZ may be a single numerical value as a whole, or the inside of the area may be divided into four parts, top and bottom, left and right, or top and bottom, as shown in FIGS. 13 (a) and 13 (b). FIG. 13E shows an example in which the grid is radially divided into 12 like a clock. 13A to 13E show the OCTA image 1110 as an image for displaying the grid in a superimposed manner, but the present invention is not limited to this. For example, a grid may be superimposed and displayed on the fluctuation image 1150 or the OCTA image 1210 + color blood vessel 1201 shown in FIG. The grid display On / Off may be selected by a right-click menu (not shown), or the display may be switched by On / Off of a check box (not shown).

  Next, statistical values (maximum value, minimum value, average value, median value, standard deviation, variance, etc.) of the coefficient of variation inside the grid as shown in FIG. 13 are calculated and displayed. When calculating a statistical value, it is desirable to use only a blood vessel mask as shown in the first embodiment and target only a portion corresponding to a blood vessel. As a method of displaying the statistical value of the coefficient of variation, numerical values may be displayed inside the grid, or numerical values may be displayed as a graph or a table at a location different from the image.

  Here, an example in which the statistical value of the coefficient of variation inside the grid is displayed in a graph is shown in FIG. FIGS. 14A and 14B are graph examples when the grid shown in FIG. 13E is divided radially. The grid is 1 to 12 clockwise from the top, the horizontal axis is the location of the divided grid, and the vertical axis is the statistical value in the grid. In the graphs of FIGS. 14A and 14B, 0 is the same value as 12. FIGS. 14A and 14B are examples of different eye data, and show examples in which statistical values appear differently for different eyes. On the other hand, FIG. 14C is an example in which the data in FIGS. 14A and 14B are shifted in the horizontal direction and displayed with the minimum statistical value of each grid as a reference. As shown in FIG. 14C, when data is organized based on statistical values instead of locations, it becomes easy to compare the features of blood vessels around it with reference to locations where there are some features in the blood vessels. Therefore, when comparing with a database created from different eyes or normal data, the comparison may be performed with reference to a characteristic place as shown in FIG.

  Furthermore, instead of displaying numerical values in units of grids as shown in FIG. 13, statistical values may be calculated and displayed in units of blood vessels. In that case, for example, by thinning the blood vessel mask, it can be divided into units of blood vessels one by one. As for the end point of the blood vessel, in the thinned blood vessel, in the vicinity of 8 pixels, the connection point of 1 pixel is the end point of the line. Inside a straight line or curved line, there are two connected pixels in the vicinity of 8 pixels. Therefore, the thinned blood vessel can be identified in units of blood vessels using these pieces of information. For this reason, when the operator moves the mouse cursor or the like to an arbitrary blood vessel, the statistical value of the selected blood vessel may be displayed in a pop-up on an OCTA image at an arbitrary location on the screen. Furthermore, numerical values may be displayed not in units of blood vessels but in units of pixels. In that case, since the coefficient of variation is calculated in units of pixels, when the operator moves the mouse cursor to an arbitrary pixel, the numerical value of the selected pixel is set to an arbitrary location on the screen or an OCTA image. It may be displayed in a pop-up above.

  As described above, according to the second embodiment, it is possible to evaluate the variation between images by calculating the coefficient of variation between OCTA images. By displaying an index for evaluating the variation between a plurality of images as a numerical value for a blood vessel that has a poor flow due to a circulatory disorder or the like, it is possible to grasp the state of circulation. In addition, in 2nd Embodiment, although it shows about calculation of the statistical value of a coefficient of variation as an analysis object, it is not restricted to this. For example, the area density or thinning (skeleton) density of blood vessels may be analyzed using a grid area described later.

<Third Embodiment>
In the first and second embodiments described above, the example in which the temporal variation is displayed and calculated using a plurality of two-dimensional motion contrast data has been described. In the third embodiment, an example in which a temporal variation is displayed and calculated using a plurality of three-dimensional motion contrast data will be described. Hereinafter, differences from the first embodiment and the second embodiment will be mainly described.

  FIG. 15 is a diagram illustrating a configuration of an image processing system 1000 including the image processing apparatus 1400 according to the present embodiment. The image processing unit 1403 has a depth alignment unit 1437. The depth alignment unit 1437 performs alignment in the depth direction (z axis) of the retina in the three-dimensional tomographic image and the three-dimensional motion contrast data.

  Next, a processing procedure of the image processing apparatus 1400 according to the present embodiment will be described with reference to FIG. In FIG. 16, the same reference numerals are assigned to the processes similar to those in FIG.

  When the processes in steps S301 to S304 are finished, in step S1505, the image processing unit 303 generates high-quality data using the acquired three-dimensional motion contrast data. FIG. 16B is a flowchart illustrating high-quality data generation processing according to the third embodiment.

  In FIG. 16B, steps S351 to S355 are as described in the first embodiment (FIG. 3). In step S1556, the depth alignment unit 1437 performs alignment in the depth direction. The depth alignment unit 1437 performs alignment in the depth direction within the three-dimensional data (adjacent tomographic images) and in the depth direction between the three-dimensional data (tomographic images of different data). For example, by first performing alignment within the data, the depth direction of the retina and the position of the retina inclination are aligned between adjacent tomographic images. Next, the retina depth direction and the retina inclination are aligned between different data using the data that has already been aligned within the data. XYZ alignment between different three-dimensional data is performed by the first alignment unit 334, the second alignment unit 336, and the depth alignment unit 1437.

  In step S1557, the calculation unit 1440 calculates statistical values (average value, standard deviation, coefficient of variation, maximum value, minimum value, etc.) from the plurality of three-dimensional motion contrast data. The calculation of the statistical value is the same as in step S356, but in the case of three-dimensional data, it is a voxel instead of a pixel. Therefore, a statistical value between different data is calculated in a plurality of aligned 3D motion contrast data. Note that regarding the statistical value calculation method, the statistical value may be calculated not in units of voxels but in units of voxels having a thickness in the depth direction. For example, instead of one voxel, an average or median value of several voxels around the target voxel may be calculated in the Z-axis direction, and a statistical value between the data may be calculated using the value.

  In step S1558, the image generation unit 332 generates the average value calculated by the calculation unit 1440 as high-quality three-dimensional motion contrast data. The image generation unit 332 generates high-quality data of 3D motion contrast data and also generates high-quality data of 3D tomographic images. After performing the above processing, the processing returns to the flowchart of FIG.

  In step S1506, the fluctuation image generation unit 341 generates a fluctuation image based on the statistical value calculated by the calculation unit 1440 in step S1557. More specifically, first, the image generation unit 332 uses the three-dimensional motion contrast data that has been averaged and the three-dimensional statistical value (for example, three-dimensional variation coefficient data) obtained from the three-dimensional motion contrast data, respectively. A two-dimensional front image is generated. The variation image generation unit 341 generates a variation image using the two-dimensional image generated by the image generation unit 332. The variation image generation method is the same as that in the first embodiment (step S306 in FIG. 3), and thus description thereof is omitted.

  In step S1507, the high-quality three-dimensional motion contrast data, the high-quality three-dimensional tomographic image, the two-dimensional high-quality motion contrast data, and the fluctuation image that are created by averaging are displayed.

  Here, an example of a screen displayed on the display unit 600 is shown in FIG. In FIG. 17, three-dimensional tomographic images and two-dimensional motion contrast data (OCTA) display high-quality data. Further, high-quality three-dimensional motion contrast data 1640 is superimposed on the tomographic image. In the third embodiment, three-dimensional variation coefficient data may be displayed instead of the three-dimensional motion contrast data 1640. When the three-dimensional variation coefficient data is displayed in a superimposed manner, it is desirable to superimpose and display a portion where each of the three-dimensional motion contrast data 1640 and the three-dimensional variation coefficient data is equal to or greater than a predetermined threshold. As a result, it is possible to display the three-dimensional variation coefficient data of a portion three-dimensionally corresponding to the blood vessel. When the three-dimensional variation coefficient data is displayed in a superimposed manner, it is displayed in color and displayed in the same color as the bar 1151 of the two-dimensional variation image.

  Although not shown in FIG. 17, it is also possible to perform the analysis as shown in FIG. 13 and display numerical values for the variation image created from the three-dimensional variation coefficient data.

  According to the configuration described above, in the third embodiment, the variation between the data can be evaluated by calculating the coefficient of variation between the three-dimensional motion contrast data. By displaying an index for evaluating a variation between a plurality of data as an image or a numerical value for a blood vessel having a poor flow due to a circulatory disorder or the like, the state of circulation can be grasped.

(Modification 1)
In each of the above-described embodiments, the example in which the variation information in a plurality of data is displayed as one image is shown, but the present invention is not limited to this. You may display as a moving image instead of making it one image. For example, the OCTA images 1 to 3 taken at different times and aligned as shown in FIG. 9A are displayed at the same place (for example, the first OCTA image) at a predetermined time interval (for example, 1 frame / second). To the place where is displayed). Thereby, the operator can confirm the change of the OCTA image.

(Modification 2)
In the third embodiment, an example in which three-dimensional variation coefficient data is obtained from three-dimensional motion contrast data has been described. However, the present invention is not limited to this. For example, in the three-dimensional alignment, high-quality three-dimensional motion contrast data and high-quality tomographic images are generated, and the variation image is obtained from the two-dimensional OCTA image as shown in the first and second embodiments. May be.

(Modification 3)
In each of the above embodiments, an example in which a circulation state is evaluated using a coefficient of variation has been described, but the present invention is not limited to this. For example, a standard deviation, a difference between data, or the like may be used, or by calculating with Log, it is possible to suppress the influence of variation in a portion having a large value. Any index can be used as long as it can evaluate the variation among a plurality of data.

(Modification 4)
In each of the above embodiments, an example in which a blood vessel mask is created by binarization processing from an OCTA image with high image quality is shown, but the present invention is not limited to this. For example, thinning processing may be performed after binarization processing, and the thinned image may be used as a mask image. In the case of a thinned image, a blood vessel is represented by one pixel, and thus is not affected by an artifact caused by an alignment error that may appear in the fold of the blood vessel. For this reason, it is also possible to calculate blood vessel fluctuation using a thinned image as a mask image. When expressing the variation value of the blood vessel with a color image, the value obtained by the thinning process may be extended to the blood vessel equivalent region in the binarized image before thinning.

(Modification 5)
In each of the above embodiments, an example in which an OCTA image is generated from motion contrast data has been described. However, a variation image may be generated using an image from which projection artifacts have been removed as an OCTA image. The projection artifact is a phenomenon in which the shadow of the blood vessel in the upper layer is reflected in the lower layer, and the artifact is generated in a place other than the blood vessel by changing the shadow due to a change due to the flow of blood. Such artifacts may exist in motion contrast data. For this reason, even when the range for generating the OCTA image is arbitrarily designated by operating the UI, the fluctuation image may be generated and displayed using the image from which the artifact is removed.

(Modification 6)
In each of the above-described embodiments, the process from shooting to display is shown in a series of flows, but the present invention is not limited to this. For example, the variation image generation process may be performed using data that has already been shot. In that case, steps S302 to S304 of the processing relating to imaging are skipped, and instead, a plurality of 3D motion contrast data and 3D tomographic images already acquired are acquired. In step S306 or step S1506, the variation image generation process is performed. As a result, the fluctuating image generation process can be executed when necessary, without processing the data that has been shot a plurality of times without taking a process at the time of shooting. Therefore, when shooting, you can concentrate on shooting.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  For example, the CPU of the image processing apparatus controls the entire computer using computer programs and data stored in a RAM or ROM. In addition, the execution of software corresponding to each unit of the image processing apparatus is controlled to realize the function of each unit. Also, the user interface such as buttons and the display layout are not limited to those shown above.

100: Image processing system, 200: Tomographic imaging apparatus, 300: Image processing apparatus, 301: Image acquisition unit, 303: Image processing unit, 305: Display control unit, 311: Tomographic image generation unit, 312: Motion contrast data generation Unit, 331: preprocessing unit, 332: image generation unit, 333: detection unit, 334: first alignment unit, 335: selection unit, 336: second alignment unit, 340: calculation unit, 341: fluctuation Image generation unit, 400: fundus image capturing apparatus, 500: external storage unit, 600: display unit, 700: input unit, 1440: calculation unit, 1437: depth alignment unit

Claims (23)

Acquisition means for acquiring a plurality of motion contrast images relating to the same position of the fundus at different times,
Calculating means for calculating a value indicating a variation in luminance for a plurality of corresponding pixels of the plurality of motion contrast images;
Generating means for generating an output image based on a value indicating the variation in luminance calculated by the calculating means;
An image processing apparatus comprising: an output unit that outputs an output image generated by the generation unit.   The image processing apparatus according to claim 1, wherein the value indicating the luminance variation is a value of a standard deviation of luminance or a value of variance in a plurality of corresponding pixels of the plurality of motion contrast images.   The generation unit normalizes a value indicating the luminance variation using a representative value of luminance of a plurality of corresponding pixels in the plurality of motion contrast images, and uses the normalized value indicating the luminance variation. The image processing apparatus according to claim 1, wherein the output image is generated.   The image processing apparatus according to claim 3, wherein the representative value is an average value or a median value of luminances of the plurality of motion contrast images.   5. The image processing apparatus according to claim 1, wherein the generation unit generates, as the output image, an image obtained by extracting a blood vessel portion from an image based on the value indicating the variation in luminance. . The generating means includes
Generating a blood vessel mask image based on an averaged image of the plurality of motion contrast images;
The image processing apparatus according to claim 1, wherein the output image is generated from a logical product of the mask image and an image based on a value indicating the luminance variation.   The image processing apparatus according to claim 6, wherein the generation unit generates the mask image by performing binarization processing on the addition average image.   The said generation means produces | generates the said mask image by performing the binarization process with respect to the image after performing the sharpening process or the enhancement process with respect to the said addition average image, The said mask image is produced | generated. The image processing apparatus described.   The image processing apparatus according to claim 1, wherein the generation unit generates an image colored according to a value indicating the luminance variation.   The image processing apparatus according to claim 1, wherein the plurality of motion contrast images and the output image are Enface images. Processing means for performing reduction processing to reduce artifacts for the plurality of motion contrast images;
The said calculating means calculates the value which shows the dispersion | variation in the said brightness | luminance using the said several motion contrast image in which the said reduction process was performed by the said processing means. An image processing apparatus according to 1.   The image processing apparatus according to claim 1, wherein the output unit displays the output image on a display unit. Acquisition means for acquiring a plurality of motion contrast images relating to the same position of the fundus at different times,
First generating means for generating a first image showing a shape of a blood vessel of the fundus obtained by processing at least one of the plurality of motion contrast images;
Second generation means for generating a second image by associating information indicating variation in motion contrast values in the plurality of motion contrast images with an area indicating the blood vessel included in the first image;
An image processing apparatus comprising: output means for outputting the second image generated by the second generation means.   The second generation unit generates the second image by giving a color corresponding to information indicating variation in the motion contrast value to a region indicating the blood vessel included in the first image. The image processing apparatus according to claim 13.   15. The image processing apparatus according to claim 13, wherein the information indicating the variation in the motion contrast value is information indicating a standard deviation or a variance of the motion contrast value in the plurality of motion contrast images.   The information indicating the variation in the motion contrast value is information obtained by normalizing a standard deviation or a variance of the motion contrast values in the plurality of motion contrast images with a representative value of the motion contrast values in the plurality of motion contrast images. The image processing apparatus according to claim 13 or 14.   The image processing apparatus according to claim 16, wherein the representative value is an average value or a median value of motion contrast values of the plurality of motion contrast images.   The image processing apparatus according to claim 13, wherein the first image is an averaged image of the plurality of motion contrast images.   The first generation unit performs a sharpening process or an emphasis process on the addition average image of the plurality of motion contrast images, and generates a first image indicating a blood vessel shape by performing a binarization process. The image processing apparatus according to claim 13, wherein the image processing apparatus is an image processing apparatus.   20. The second generation unit does not associate information indicating variation in the motion contrast value with a region other than the region indicating the blood vessel included in the first image. The image processing apparatus according to any one of the above. An acquisition step of acquiring a plurality of motion contrast images relating to the same position of the fundus at different times,
A generation step of generating an output image based on a value indicating luminance variation in a plurality of corresponding pixels of the plurality of motion contrast images;
And an output step of outputting the output image generated by the generation step. An acquisition step of acquiring a plurality of motion contrast images relating to the same position of the fundus at different times,
A first generation step of generating a first image showing a shape of a blood vessel of the fundus obtained by processing at least one image of the plurality of motion contrast images;
A second generation step of generating a second image by associating information indicating variations in motion contrast values in the plurality of motion contrast images in a region indicating the blood vessel included in the first image;
And an output step of outputting the second image generated by the second generation step.   A program for causing a computer to execute each step of the image processing method according to claim 21 or 22.