JP5998493B2 - Ophthalmic image processing apparatus and program - Google Patents

Description

  The present invention relates to an ophthalmologic image processing apparatus and a program for performing alignment between different image data acquired by an ophthalmologic photographing apparatus.

  In an ophthalmic imaging apparatus such as an optical coherence tomography (OCT), fundus camera, or laser scanning ophthalmoscope (SLO), images at the same site are acquired at different dates and times. May be compared, and the captured images may be compared. For example, in the case of OCT for ophthalmology, a tomographic image of the fundus is acquired a plurality of times, and the progress of the lesion is observed from the change in the tomographic image (Patent Document 1).

  For alignment of the captured image, for example, the CPU detects a characteristic portion such as a blood vessel extraction or a branch point of a part from one captured image, and detects a similar characteristic portion in the other captured image. Then, the image alignment is performed by matching these characteristic portions.

JP 2010-246904 A

  When comparing between captured images, it is important to accurately align the captured images. Conventionally, alignment is often performed automatically. However, automatic alignment may fail.

  For example, when performing follow-up over several years, if a plurality of years have passed, the specification of the apparatus and the imaging method may be significantly changed. Further, even in the case of a common device, when the version is upgraded or in a successor, the function mounted on the device and the means for acquiring a predetermined image may be changed. For this reason, alignment between captured images may not be performed with high accuracy.

  In addition, when the shooting position is extremely different or the quality of the shot image is poor, automatic alignment may fail. For example, when the image quality of a captured image is poor, it is difficult to detect a feature portion, and positioning may fail due to setting different feature portions between captured images to be aligned.

  In this way, when the automatic alignment fails, there is a method in which the examiner designates a characteristic portion in each captured image and performs alignment. However, since manual positioning designates a characteristic part by visual inspection by an examiner, there is a possibility that a different part is selected as a characteristic part in each captured image. In addition, when there are a plurality of captured images to be aligned, it is a heavy burden on the examiner to specify a characteristic portion on each captured image.

  In view of the above problems, the present invention relates to an ophthalmic image processing apparatus and a program that can easily and accurately align image data.

  In order to solve the above problems, the present invention is characterized by having the following configuration.

(1) The ophthalmic image processing apparatus according to the first aspect of the present disclosure includes image data of a reference front image acquired by the ophthalmologic photographing apparatus and image data of another front image acquired at a timing different from the reference front image. Are acquired from the storage unit and displayed on the monitor.
A feature portion on the reference front image can be selected based on an operation signal input from the operation input unit , and a feature portion corresponding to the feature portion selected in the reference front image is imaged in the other front image. Image processing for searching by processing and performing matching processing between the reference front image and the other front image by image processing based on the feature portion in the reference front image and the feature portion searched in the other front image It has a section, an ophthalmologic image processing apparatus that performs positioning between the different image data obtained by the ophthalmic photographing apparatus,
The reference front image is a front image associated with a first three-dimensional tomographic image acquired by an optical coherence tomography,
The other front image is a front image associated with a second three-dimensional tomographic image acquired by an optical coherence tomography and acquired at a different timing from the first three-dimensional tomographic image,
The image processing unit
Based on the matching result between the reference front image and the other front image, the first map showing the analysis result of the first three-dimensional tomographic image and the analysis result of the second three-dimensional tomographic image are shown. Alignment with the second map is performed .
(2) An ophthalmic image processing program according to a second aspect of the present disclosure is an ophthalmic image processing apparatus program for performing alignment between different image data acquired by an ophthalmologic photographing apparatus.
Acquiring from the storage unit image data of the reference front image acquired by the ophthalmologic photographing apparatus and image data of another front image acquired at a different timing from the reference front image;
A step of selecting a feature portion on the reference front image based on an operation signal input from an operation input unit;
Searching for a feature portion corresponding to the feature portion selected in the reference front image by image processing in the other front image;
An image processing step of performing matching processing between the reference front image and the other front image by image processing based on the feature portion in the reference front image and the feature portion searched in the other front image;
An ophthalmologic image processing program causing a computer to execute the,
The reference front image is a front image associated with a first three-dimensional tomographic image acquired by an optical coherence tomography,
The other front image is a front image associated with a second three-dimensional tomographic image acquired by an optical coherence tomography and acquired at a different timing from the first three-dimensional tomographic image,
The image processing step includes
Based on the matching result between the reference front image and the other front image, the first map showing the analysis result of the first three-dimensional tomographic image and the analysis result of the second three-dimensional tomographic image are shown. Alignment with the second map is performed.

  According to the present invention, it is possible to easily and accurately align a captured image.

  DESCRIPTION OF EMBODIMENTS An embodiment for carrying out an apparatus according to the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating an example of a system according to the ophthalmic image processing apparatus of the present embodiment, and FIG. 2 is a block diagram illustrating a configuration of the ophthalmic image processing apparatus.

<Overview>
An outline of an ophthalmologic image processing apparatus according to an embodiment of the present invention will be described. In the present embodiment, the ophthalmic image processing apparatus 1 includes an operation input unit (operation unit) 90, a monitor 75, a display control unit, and an image processing unit. The control unit (CPU) 70 is used as, for example, a display control unit and an image processing unit.

  The control unit 70 acquires from the storage unit (memory) 72 the image data of the reference front image acquired by the ophthalmologic photographing apparatus and the image data of other front images acquired at a timing different from the reference front image.

  For example, as a display control unit, the control unit 70 displays a reference front image and a pointer 60 that can move on the reference front image based on an operation signal input from the operation input unit 90 on the monitor 75.

  For example, as an image processing unit, the control unit 70 enables the examiner to select a characteristic portion (for example, a reference point a) in the reference front image 20 based on position information of the pointer 60 on the reference front image 20. The control unit 70 searches the other front image 30 for a feature portion (for example, the corresponding point a ′) corresponding to the feature portion selected in the reference front image 20 by image processing. The control unit 70 performs matching processing between the reference front image and the other front image based on the position coordinates of the feature portion in the reference front image and the position coordinates of the feature portion searched for in the other front image. Perform by processing. In the matching processing, for example, alignment between the reference front image and another front image, detection of a position shift direction and a position shift amount, and the like are performed.

  As a result, the image feature portion in the reference front image is surely selected by the examiner, and the feature portion corresponding to the feature portion selected in the reference front image is searched for in the other front image by image processing by the control unit 70. Then identified. For this reason, even if automatic alignment is insufficient, good matching between images can be smoothly performed using the characteristic portion in the reference front image selected by the examiner.

  The operation unit 90 is operated by an examiner. For the operation unit 90, for example, a user interface such as a mouse, a trackball, or a touch panel is used.

  A plurality of front images are displayed on the monitor 75. As the monitor 75, for example, a display provided in a PC or a display provided in an ophthalmologic photographing apparatus is used. The monitor 75 may be a touch panel.

  Note that the selection of the feature portion on the reference front image using the operation unit 90 is not limited to the method using the pointer 60. A feature portion on the reference front image may be selectable by the examiner based on an operation signal input from the operation unit 90. For example, when the monitor 75 is a touch panel, it is conceivable that the feature portion is selected by the examiner touching the feature portion on the monitor.

  The storage unit (memory) 72 stores the image data of the reference front image acquired by the ophthalmologic photographing apparatus and the image data of other front images acquired at a timing different from the reference front image. As the storage unit (memory) 72, for example, a hard disk provided in a PC, an external server, a USB memory, or the like is used. The memory 72 stores, for example, image data of a plurality of front images acquired by at least one ophthalmologic photographing apparatus. For example, the plurality of front images are acquired at different timings.

  As the ophthalmologic photographing apparatus, for example, a laser scanning optometry apparatus (SLO), an optical coherence tomography (OCT photographing apparatus), a fundus camera, or the like can be considered. The laser scanning optometry apparatus is an apparatus that scans a laser beam on an eye to be examined and receives reflected light from the eye through a confocal aperture to obtain a front image. The OCT imaging apparatus is an apparatus that scans OCT measurement light with an eye to be examined, receives light obtained by combining reflected light from the eye and reference light, and obtains a front image. A fundus camera is a device that obtains a front image by integrally irradiating illumination light to a subject's eye and receiving reflected light from the eye with a two-dimensional imaging device.

  The examiner operates the operation unit 90 to compare a plurality of front images stored in the memory 72. For example, a plurality of front images stored in the memory 72 are displayed on the monitor 75. The examiner operates the operation unit 90 to select a front image to be aligned from a plurality of front images displayed on the monitor 75. For example, when the examiner selects two front images, two front images are displayed on the monitor 75. At this time, for example, one front image is set as a reference front image, and the other front image is set as another front image. Note that the ophthalmologic photographing apparatus for photographing the reference front image and other front images may be different.

  First, the control unit 70 displays a plurality of front images (reference front image and other front images) stored in the memory 72 on the monitor 75. The examiner operates the operation unit 90 on the reference front image 20 displayed on the monitor 75, and features on the front image (for example, an image region including a feature point based on a branch of a blood vessel, a nipple portion). Is specified. When the feature portion is selected by the examiner, the control unit 70 determines the reference front image based on the operation signal input from the operation unit 90 in the feature portion on the reference front image 20 displayed on the monitor 75. Is selected (for example, reference point a) (see FIG. 4).

  For example, based on the position coordinates of the feature portion selected on the reference front image 20 and the position coordinates of the feature portion searched on the other front image 30, the control unit 70 and the other reference front image 20 By aligning the characteristic portions in the front image 30 by image processing, the reference front image 20 and another front image 30 are aligned.

  For example, the control unit 70 performs affine transformation processing based on the position coordinates of the reference point a and the position coordinates of the corresponding point a ′, and aligns the front images. Of course, as image processing, non-rigid transformation may be used instead of affine transformation which is rigid transformation. Note that as the image processing for alignment, for example, processing for synchronizing the contrast of the front image may be performed.

  It is more preferable to perform alignment by setting more reference points when performing alignment. Based on the position coordinates of the plurality of feature portions selected on the reference front image 20 and the position coordinates of the plurality of feature portions searched on the other front image 30, the control unit 70. And other front images 30 are matched by image processing. This makes it possible to calculate the positional relationship (parallel movement amount, rotation angle, magnification, etc.) between the front images, so that alignment can be performed with high accuracy.

  When using a feature point as the feature part, the control unit 70, for example, the position coordinates of a plurality of reference points specified on the reference front image and a plurality of correspondences corresponding to the plurality of reference points on the other front image. Based on the position coordinates of the points, the reference front image and another front image are aligned by image processing so that the reference point matches the corresponding point or the error is minimized.

  When searching for a feature portion, the control unit 70 searches (detects) a corresponding feature portion on the other front image 30 based on the position coordinates of the feature portion selected on the reference front image 20. To do. As the matching method, for example, template matching or phase only correlation method can be considered.

  Note that the front image to be aligned may be a plurality of front images. For example, when aligning three or more front images, when a reference front image serving as a reference is selected and a reference point is set on the reference front image, a reference point on a plurality of other front images Corresponding points corresponding to are respectively set. Then, based on the information, the plurality of front images are aligned.

  Note that the system in the present embodiment is more preferably performed after automatic alignment. For example, the control unit 70 performs a first matching process between the reference front image 20 and the other front image 30 by image processing before the feature portion is selected by the examiner.

  After the first matching process, the control unit 70 can select the feature portion in the reference front image 20 based on the position information of the pointer 60 on the reference front image 20 and the feature portion selected in the reference front image 20. A feature portion corresponding to is searched by image processing in the other front image 30, and based on the position coordinates of the feature portion in the reference front image 20 and the position coordinates of the feature portion searched in the other front image 30 A second matching process between the front image 20 and the other front image 30 is performed by image processing.

  The above method is advantageous when the alignment fails when the alignment is automatically performed (when the alignment accuracy is not good or when the alignment cannot be performed). Thus, only when automatic alignment fails, by using the semi-automatic alignment of the present embodiment, alignment can be performed efficiently, and alignment failures can be reduced. .

  As described above, alignment can be easily performed with high accuracy. Then, by performing alignment between front images with high accuracy, comparison (for example, follow-up observation) between image data acquired at different timings can be performed with high accuracy.

  The control unit 70 may directly compare the front images with each other, or may compare OCT image data corresponding to each front image.

  The comparison process between the OCT image data is performed, for example, by creating an analysis map or the like and using it. For example, each layer thickness map showing a two-dimensional distribution related to the layer thickness is created based on the fundus layer information in each OCT image data acquired at different timings, and a difference map that is a difference result between the layer thickness maps is created. The

  Note that the information on the thickness of the fundus layer to be mapped (graphed) may be information regarding the thickness of the fundus layer in the depth direction (Z direction). For example, in addition to the thickness of each layer, a plurality of layers It may be the thickness of the layer when the thicknesses are added. For example, the thickness of the nerve fiber layer and the thickness from the retina surface to the choroid can be considered.

  The analysis map is not limited to the difference map. A map that shows the position of the abnormal part on the fundus in two dimensions can be considered. For example, the layer thickness at each position is measured, and it is determined whether or not the measurement result is within a predetermined range in the normal eye database (for example, a normal range corresponding to a normal eye measurement value). That is, layer thickness determination, shape determination of each layer, size determination of a predetermined part (for example, nipple, macular), etc. are performed, and normal / abnormality of the fundus of the fundus is determined. And it is good also as a structure which detects the two-dimensional distribution data regarding the normal / abnormal part of the fundus of the fundus.

  Acquisition of layer information needs to be detected from a three-dimensional tomographic image acquired by an optical coherence tomography. For this reason, it is necessary to acquire the three-dimensional tomographic image acquired by the optical coherence tomography with acquisition of each front image. That is, the acquisition of the layer information of each front image is performed by analyzing a three-dimensional tomographic image associated with each front image. Note that the front image and the 3D tomographic image are associated with each other by generating an OCT front image from the 3D tomographic image used for the analysis and superimposing the generated OCT front image and the front image. It is done.

  Then, by comparing the positions of the front images, each layer information can be compared, and an analysis map can be created. For example, the reference front image is associated with the first three-dimensional tomographic image acquired by the optical coherence tomography. The other front image is associated with the second three-dimensional tomographic image acquired by the optical coherence tomography. The control unit 70 associates the first three-dimensional tomographic image with the second three-dimensional tomographic image based on the alignment result between the reference front image and the other front image. As a result, since the three-dimensional tomographic image of each optical coherence tomography and the front image are matched, it becomes possible to associate each front image with layer information. It can be carried out.

  As described above, by acquiring analysis data in a state where alignment is performed with high accuracy, it is possible to reduce acquisition of an erroneous analysis map or failure to acquire an analysis map.

  Note that the present invention can be applied as long as at least a common part is acquired between the front images. For example, it can be used for a panoramic image that acquires a wide range of front images by arranging a plurality of front images. In this case, it can be used as an auxiliary tool when manually performing panorama synthesis processing of the front image.

  Note that, by checking the alignment result confirmation screen after the alignment is completed, the examiner can confirm whether or not the alignment between the front images has been performed with high accuracy. For example, when the operation unit 90 is operated by the examiner, the control unit 70 shifts the display on the monitor 75 to the alignment result confirmation screen.

  In the alignment result confirmation screen, the control unit 70 has a timing different from that of the reference front image 20 with respect to the image data of the reference front image (first front image) 20 acquired by the ophthalmologic imaging apparatus and the same imaging region as the reference front image 20. The image data of the other front image (second front image) acquired in step S2 is acquired from the memory 72. Then, the control unit 70 extracts the first image data, which is a part of the image data in the reference front image 20 that has been aligned, from the image data of the reference front image, and is different from the image data extracted from the reference front image. The second image data of the part is extracted from the image data of the other front image 30. The control unit 70 synthesizes each extracted image data to create a synthesized front image for confirming the positional deviation between the reference front image 20 and the other front image 30, and creates the created synthesized front image 25. It is displayed on the monitor 75. That is, the control unit 70 performs image synthesis processing as an image processing unit, and displays the synthesized front image 25 on the monitor 75 as a display control unit (see FIG. 5). For example, still image data is used as the first image data and the second image data. Of course, moving image data or the like may be used as each image data.

  As the synthesized front image 25, for example, the control unit 70 extracts the first image data and the second image data so that the first image data and the second image data are alternately and repeatedly arranged and synthesized. It is conceivable to create a composite front image 25 by processing. For example, the control unit 70 determines that the first image data and the second image data are a check pattern pattern as a pattern in which the first image data and the second image data are alternately and repeatedly arranged in the composite front image 25 of one frame. It is conceivable that the first image data and the second image data are extracted and combined to be arranged alternately in the display format. Further, it is conceivable that the first image data and the second image data are alternately and repeatedly arranged in a display format such as a border pattern. In the case of the check pattern pattern display format, the first image data and the second image data are composed of 5 × 5 divided regions (see FIG. 5). Note that the number of divisions can be changed (for example, 3 × 4, 6 × 6, etc.). Moreover, the structure which an examiner can set the number of divisions arbitrarily may be sufficient.

  In the alignment result confirmation screen, after the image data of the reference front image 20 and the image data of the other front image 30 are aligned by image processing, the control unit 70 performs alignment by image processing. Although the first image data and the second image data are extracted from the reference front image 20 and the other front image 30 in a later state to create a composite front image, the present invention is not limited to this. A synthesized front image may be created by synthesizing the front images before the alignment is completed. In this case, image data at a position corresponding to each pixel position of the synthesized front image is extracted from each front image, and a synthesized front image is created.

  When combining the color front image and the grayscale front image in the composite front image, it is easier to compare by changing the color of one of the front images to the other color. . For example, a color front image is changed to a gray-scale front image, and a synthesis process is performed.

  Note that the synthesized front image may be created by a synthesis process of a plurality of front images. For example, when three or more front images are combined, three or more front images are sequentially arranged.

  Note that the plurality of front images forming the composite front image may be colored with different colors and displayed on the composite front image. For example, in the synthesized front image, the image data portion of the reference front image is not colored. In the combined front image, the image data portion of the other front image is colored. Each image data is displayed in a different color. As a result, when it is difficult to understand the boundary between the image data, such as when there is little deviation between the image data, the boundary between the image data can be clearly recognized, making it easy to check the alignment.

  In addition, although it was set as the structure which displays the synthesized image of a front image as a synthesized image, it is not limited to this. The synthesized 3D tomographic image may be displayed by synthesizing the 3D tomographic image and the 3D tomographic image. In this case, the image data of the first three-dimensional tomographic image (hatched portion) and the image data of the second three-dimensional tomographic image (blank page portion) are alternately arranged (see FIG. 7).

<Example>
FIG. 1 is a diagram illustrating an example of an ophthalmologic image processing apparatus according to the present embodiment. In the present embodiment, a case where follow-up observation is performed by the ophthalmic image processing apparatus 1 will be described as an example.

  The ophthalmic image processing apparatus 1 includes, for example, an image processing apparatus 10, a first ophthalmic imaging apparatus A (imaging apparatus A), and a second ophthalmic imaging apparatus B (imaging apparatus B). Note that each unit of each device is connected via a network (bus, LAN, etc.), and can transmit and receive image data and the like to each other. Note that the ophthalmic image processing apparatus 1 does not necessarily include the imaging apparatus A and the imaging apparatus B at the same time, and in the long-term follow-up observation, the tomographic imaging apparatus is changed from one imaging apparatus to another imaging apparatus. Is also included.

  The imaging apparatus A and the imaging apparatus B are OCT devices each having a tomographic imaging optical system (for example, optical coherence tomography (OCT). Each OCT device measures a light beam emitted from a light source as measurement light. And the measurement light beam is guided to the fundus of the eye to be examined, the reference light is guided to the reference optical system, and then the interference state between the measurement light reflected from the fundus and the reference light is detected by the detector. A three-dimensional tomographic image is acquired by image processing based on the received light signal output from the detector.

  In the present embodiment, each of the imaging devices A and B has a different front imaging optical system.

  For example, the imaging apparatus A includes an optical scanner that two-dimensionally scans the fundus of measurement light (for example, infrared light) emitted from a light source, and a confocal aperture that is disposed at a position substantially conjugate with the fundus. It has a light receiving element that receives fundus reflected light and has a so-called laser scanning opthalmoscope (SLO) device configuration. When acquiring the fundus front image, the fundus front image (SLO image) is acquired based on the light reception signal output from the light receiving element of the SLO. Note that the OCT scanning position and the scanning position when acquiring the SLO image are set in advance so as to scan a common area. Thereby, a three-dimensional tomographic image at a position corresponding to the SLO image is acquired by OCT, and the fundus front image and the three-dimensional tomographic image are matched.

  When acquiring the fundus front image, the imaging apparatus B acquires the fundus front image based on the OCT interference signal using the OCT front image imaging optical system. For example, the present apparatus scans the measurement light two-dimensionally, and obtains an OCT front image by integrating the spectral intensity of the interference signal from the light receiving element for each XY point. Since the fundus front image is acquired based on the tomographic image, each position of the fundus front image is associated with the tomographic image acquired by OCT.

  Further, by analyzing the three-dimensional data acquired by OCT, image data including a distribution state related to a predetermined tissue of the fundus (for example, a blood vessel, a nipple, a macula, etc.) is created by image processing, and the created image data May be used as a front image. For example, the technique of the present embodiment is applied even to a configuration in which layer thickness maps having different acquisition timings are acquired and alignment is performed using portions corresponding to blood vessels in those layer thickness maps.

  For example, the image processing apparatus 10 acquires image data captured by each imaging apparatus via a network, and stores and manages each image data (for example, three-dimensional tomographic image data, front image data). Then, the image processing apparatus 10 analyzes the acquired image by different image acquisition methods in each apparatus, and displays the analysis result of the image (details will be described later).

  Note that each imaging device transfers the acquired image data automatically or manually via a network.

  FIG. 2 is a block diagram illustrating the configuration of the image processing apparatus 10.

  As shown in FIG. 2, the image processing apparatus 10 includes an image processing unit (control unit) 70, an operation unit 90, a display unit (such as a monitor) 75, and a storage unit (memory) 72.

  The control unit 70 comprehensively controls operations of the control unit 70, the monitor 75, and the memory 72 in data management of image data acquired from each device. Image data acquired by each device is stored in the memory 72 (details will be described later).

  The control unit 70 also has an image analysis function, and performs arithmetic processing based on the first image data and the second image data stored in the memory 72. Each image data is analyzed, and after analyzing each image data, a comparative analysis between the image data is performed for follow-up observation. In the comparative analysis, the image data is aligned based on the front images in the image data, and the analysis results are compared. The control unit 70 creates, for example, a map (hereinafter referred to as a layer thickness map) that two-dimensionally shows layer information at each position on the fundus.

  The operation unit 90 includes various switches, a mouse, and the like for taking in various instructions from the examiner when an operation is performed by the examiner.

  The monitor 75 can display the analyzed image data.

  The memory 72 stores image data and various control programs for controlling the operation of each unit. The memory 72 stores image data including a three-dimensional tomographic image associated with the front image acquired by the ophthalmologic photographing apparatus.

<Control action>
FIG. 3 is a flowchart showing the control operation of the ophthalmic image processing apparatus 1. Hereinafter, the control operation of the ophthalmic image processing apparatus 1 will be described more specifically with reference to flowcharts.

  In the present embodiment, in the past, when fundus photographing has been performed by the photographing apparatus A and the fundus front image has been acquired, the follow-up observation of the fundus is performed by another apparatus thereafter. I will explain.

<Acquisition of image>
First, image data including a front image and a three-dimensional tomographic image is acquired by the imaging device A and the imaging device B and transferred to the image processing device 10. For example, a first front image of the eye to be examined photographed using the SLO by the photographing apparatus A and a three-dimensional tomographic image of the eye to be examined photographed using the optical interference technique, which are associated with the first front image. First image data including the first three-dimensional tomographic image is acquired and transferred to the image processing apparatus 10. Further, a second front image of the eye to be examined photographed by the photographing apparatus B using the OCT front image capturing optical system, and a three-dimensional tomographic image of the eye to be examined photographed using the optical interference technique, Second image data including the second three-dimensional tomographic image associated with the image is acquired and transferred to the image processing apparatus 10. Each image data transferred to the image processing apparatus 10 is stored in the memory 72.

<Automatic alignment>
Next, the image processing apparatus 10 automatically aligns the front images. The control unit 70 of the image processing apparatus 10 uses the image data stored in the memory 72 to perform alignment processing between captured images. For example, when a plurality of image data is stored in the memory 72, the examiner operates the operation unit 90 and selects image data for follow-up observation from the plurality of image data. In the present embodiment, it is assumed that the first image data and the second image data described above are acquired.

  The control unit 70 calls the image data selected by the examiner from the memory 72 and performs alignment. For alignment, a front image of image data is used. The control unit 70 sets a reference front image among the image data selected by the examiner, and detects a positional deviation amount of another front image with respect to the reference front image. In the present embodiment, the first front image of the first image data out of the first image data and the second image data is set as a reference front image.

  The control unit 70 detects the amount of positional deviation between the first front image (reference front image) and the second front image (other front images), and aligns the other front images with the reference front image. That is, the reference front image and another front image are compared, and the position shift direction and the amount of position shift of the other front image with respect to the reference front image are detected by image processing.

  Various image processing methods (a method using various correlation functions, a method using Fourier transform, and a method based on feature point matching) can be used as a method for detecting the shift amount.

  For example, the reference front image or another front image is shifted by one pixel, the reference front image and another front image are compared, and when both data are the best (when the correlation is the highest), A method for detecting the direction of displacement and the amount of displacement is conceivable. Further, a method of extracting a common feature point from a predetermined reference front image and another front image and detecting a position shift direction and a position shift amount of the extracted feature point is conceivable.

  In this embodiment, the correlation value (the larger the value, the higher the correlation between the images (maximum 1)) is sequentially calculated while shifting another front image with respect to the reference front image in units of one pixel. Then, the control unit 70 detects the displacement amount (the number of displaced pixels) of the pixel when the correlation value is maximized as the displacement amount, and detects the shifted direction as the displacement direction.

  The control unit 70 aligns the reference front image and another front image based on the detected displacement amount and displacement direction. After the alignment between the front images is completed, the control unit 70 compares the front images and acquires an analysis map.

  For example, the control unit 70 analyzes the three-dimensional tomographic images acquired by the imaging device A and the imaging device B, and creates a layer thickness map that two-dimensionally indicates the layer thickness on the fundus (details will be described later). . At this time, since the three-dimensional tomographic image and the front image of each device are matched, the front image and the layer thickness map can be associated with each other. Thereby, alignment of the layer thickness map acquired by each imaging device can be performed. The control unit 70 creates an analysis map (difference map) indicating a difference result between the layer thickness maps in a common area in the front images acquired by the imaging device A and the imaging device B.

  Here, when the shooting positions between the front images are extremely different or when the image quality of the front images is poor, the alignment accuracy may be deteriorated or the alignment may fail. When comparing front images, there are cases where an analysis map cannot be acquired or an analysis map with an abnormal result is acquired.

<Semi-auto alignment>
Hereinafter, the alignment process when the automatic alignment fails will be described. For example, the examiner observes the acquired analysis map and switches the mode to the semi-auto alignment mode by operating the operation unit 90 when the analysis map cannot be acquired or when the result of the analysis map is abnormal. I do. Of course, the control operation related to mode switching may be performed by the control unit 70. For example, the control unit 70 switches the mode to the semi-automatic alignment mode when the analysis map cannot be acquired or when the analysis map shows an abnormal result. In addition, the control unit 70 may display on the monitor 75 that it is better to perform mode switching.

  Hereinafter, semi-automatic alignment will be described. When the mode is switched to the semi-auto alignment mode, the control unit 70 displays a semi-auto alignment screen for performing semi-auto alignment. FIG. 4 is a diagram illustrating an example of a semi-automatic alignment screen displayed on the monitor 75.

  On the monitor 75, a first front image (reference front image) 20, a second front image (other front image) 30, and a pointer 60 are displayed. The control unit 70 moves the pointer 60 on the monitor 75 based on the operation signal from the operation unit 90. The pointer 60 is displayed by, for example, a cross mark or a pen mark.

  First, the examiner operates the operation unit 90 to move the pointer 60 on the reference front image 20, and designates a feature point (characteristic portion) of the fundus region. As a feature point, a site with a large number of blood vessels, a branch point of blood vessels, or the like is preferable. When the feature point is designated by the examiner, the control unit 70 sets the feature point as a reference point a for alignment. Then, the control unit 70 searches for the corresponding point a ′ corresponding to the reference point a from other front images.

  The control unit 70 sets a template image T for searching for the corresponding point a ′ on the reference front image 20 in order to search for the corresponding point a ′. The template image T is set by extracting a predetermined area around the reference point a based on the position coordinates of the reference point a. As the predetermined area, for example, 5% of the image size is set in advance. For example, the control unit 70 searches the correlation value (the correlation between the images increases as the value increases (maximum 1)) while shifting the set template image in units of one pixel on the other front images. To do. And the control part 70 determines the position when a correlation value becomes the maximum, and sets corresponding point a 'from a determination result. Based on the position coordinates of the reference point a, a predetermined search area is set on another front image, the correlation value of the template image is detected within the set search area, and the corresponding point a ′ is set. You may go.

  It is also possible to change the set corresponding point a ′. The examiner operates the operation unit 90, moves the pointer 60 onto another front image, and aligns the pointer 60 with the set corresponding point a ′. Then, the examiner can adjust the position of the corresponding point a ′ by dragging and moving the corresponding point a ′ to move the corresponding point a ′.

  As described above, the reference point a and the corresponding point a ′ are set. The examiner similarly designates a plurality of reference points. In this embodiment, it is assumed that three reference points are designated. When the reference point and the corresponding point are set, the control unit 70 matches each reference point with each corresponding point based on the position coordinates of each reference point and the position coordinates of each corresponding point, or minimizes the error. The affine transformation (parallel movement, rotational movement, relative magnification change) of the other front image 30 is performed. Of course, only one of the movements or transformations of the affine transformation may be performed. Thereby, alignment between front images can be performed.

  A setting for starting the alignment may be provided. For example, when a predetermined number of reference points are set, the control unit 70 starts alignment. For example, the predetermined number is set to a number by which information for calculating the conversion process can be calculated. In the case of this embodiment, at least three or more reference points are set and the alignment process is performed.

As described above, since a part having a large feature amount can be selected and specified by semi-automatic alignment, the possibility of setting an incorrect corresponding point when setting corresponding points can be reduced. Further, the same position as the position set as the reference point can be easily set as the corresponding point, and it is possible to save the operator from failing to select the corresponding point and the trouble of the examiner. For this reason, alignment between images can be easily performed with high accuracy.
<Confirmation of alignment result>
In this embodiment, there is an alignment confirmation mode for confirming whether or not the alignment is performed with high accuracy. The examiner switches the mode to the alignment confirmation mode by operating the operation unit 90.

  Hereinafter, the alignment confirmation mode will be described. When the mode is switched to the alignment confirmation mode, the control unit 70 displays an alignment confirmation screen for confirming whether the alignment has been performed with high accuracy. FIG. 5 is a diagram showing an example of the alignment confirmation screen displayed on the monitor 75.

  On the monitor 75, a reference front image 20, another front image 30, a pointer 60, and a composite front image 25 are displayed. On the reference front image 20 and the other front image 30, reference points a1 to a3 and corresponding points a′1 to a′3 used as alignment references are displayed. The composite front image 25 is an image created by extracting at least a part of image data from the reference front image 20 and another front image 30 and performing a composite process.

  Hereinafter, creation of a composite front image will be described. The control unit 70 synthesizes the reference front image and the other front image so that the still image data of the reference front image that has been aligned and the still image data of the other front image are alternately and repeatedly arranged. . For example, as an alternately repeated pattern, there is a check pattern pattern display format in which the reference front image 20 and the other front image 30 are extracted in a grid-like region and are alternately arranged.

FIG. 6 is a diagram illustrating the arrangement of image data when creating a composite front image that is a display format of a check pattern. As illustrated in FIG. 6, for example, the control unit 70 combines the front image F <b> 1 and the front image F <b> 2 that have been aligned with each other to create a composite front image C that is a display format of a check pattern. . The control unit 70 extracts and arranges a part of the image data from the still image data of the front image F1. Further, the control unit 70 extracts and arranges a part of the image data from the still image data of the front image F2. At this time, the image data is extracted so that there is no overlapping portion in the image data extracted from the front image F1 and the front image F2. For example, as shown in FIG. 6, the image data constituting the composite front image C is that the image data of the front image F1 (shaded portion) and the image data of the front image F2 (blank page portion) are on the check pattern display format. Are arranged alternately. A composite front image is created as described above.
In the synthesized front image, two image data are alternately arranged. If there is a misalignment between the front images, the running position of the blood vessel is shifted between adjacent images. The For this reason, the examiner compares adjacent images in the synthesized front image 25 and observes the deviation of the running state of blood vessels or the like, thereby shifting the deviation between the front images obtained by performing blood alignment from the synthesized front image 25. Can be confirmed.

  In addition, you may make it display the shift | offset | difference between images quantitatively. The control unit 70 acquires image similarity information from the combined front image and displays it on the monitor 75. For example, the image similarity information includes normalized cross-correlation and mutual information amount. By doing so, the examiner can visually observe and quantitatively determine the composite front image, and therefore, it becomes easy to grasp the deviation between the front images.

  Note that the arrangement position of the image data can be changed. The examiner can change the display state of the check pattern by operating the operation unit 90 and moving the slider bar 35. For example, the check pattern is reversed as the slider bar 35 is moved in the left-right direction by the examiner. In this case, the arrangement positions of the image data of the front image F1 and the front image F2 are reversed.

  When image data is arranged, a part (space) where an image is not displayed is generated depending on the alignment and the image size. In this case, the non-displayed state may be maintained, or the size of the image data may be adjusted by changing the relative magnification so that the image data is interpolated in the space.

  Hereinafter, the control operation on the alignment confirmation screen will be described. When the mode is switched to the alignment confirmation mode, the control unit 70 creates a composite front image 25 from each front image used for alignment and displays it on the monitor 75. The examiner observes the created composite front image 25 and confirms the alignment result. At this time, when the alignment result is not preferable, the alignment is adjusted by operating the pointer 60 on any one of the reference front image 20, the other front image 30, and the composite front image 25.

  For example, when the examiner operates the operation unit 90 and moves the corresponding point a ′ 1 in the other front image 30, an operation signal is input to the control unit 70. The control unit 70 changes the position of the corresponding point a′1 by outputting a display signal to the monitor 75. The control unit 70 moves the position of the corresponding point a′1 and changes the image of the part corresponding to the image data extracted from the other front image 30 on the composite front image 25 displayed on the monitor 75. .

  The change of the image corresponding to the image data extracted from the other front image 30 will be described more specifically. The control unit 70 redoes the alignment from the relationship of the position coordinates between the corresponding point a′1 and the reference point a1 after the movement. That is, when the corresponding point is changed, the control unit 70 determines the affine of another front image based on the position coordinates of each reference point and the position coordinates of each corresponding point (the changed corresponding point and other corresponding points). Conversion is performed and the still image data is stored in the memory 72. Then, the control unit 70 creates a new composite front image 25 based on the other front image after the new alignment and the reference front image, and displays it on the monitor 75. Thus, by changing the corresponding points, the image data of the other front image portion in the composite front image 25 is changed.

  In the above description, the composite front image is changed by changing the corresponding points. However, the present invention is not limited to this. For example, the reference point may be changed. When the reference point is changed, the other front image is affine transformed based on the position information of the reference point and the corresponding point, and the part of the other front image on the composite front image is changed. Of course, the reference front image may be affine transformed. In addition, another front image or the reference front image may be moved by dragging and moving a region in the composite front image.

  The examiner moves the corresponding point while observing the synthesized front image, and adjusts the alignment so that the images in the synthesized front image match. For example, the examiner performs alignment so that the deviation of the blood vessel portion or the like in the composite front image is reduced.

As described above, alignment can be performed with high accuracy by performing alignment confirmation and adjustment in the alignment confirmation mode. In addition, since the shift between the front images can be easily observed from the shift of the blood vessel or the like by the synthesized front image, alignment can be performed with high accuracy.
<After completion of alignment>
After the alignment between the front images is completed, the control unit 70 compares the front images and acquires an analysis map. The control unit 70 analyzes the three-dimensional tomographic images acquired by the imaging device A and the imaging device B, and creates layer thickness maps that two-dimensionally indicate the layer thickness on the fundus. In the present embodiment, the analysis map is created after the alignment is completed, but the present invention is not limited to this. An analysis map may be created during alignment.

  Hereinafter, creation of the layer thickness map will be described. The control unit 70 detects fundus layer information in the acquired fundus front image by image processing. The detection result is stored in the memory 72 together with the image data.

  When detecting a layer, for example, a tomographic image on the XY plane in the image data of a three-dimensional tomographic image is used. Then, the luminance level of the tomographic image is detected, and a layer boundary corresponding to a predetermined retinal layer (for example, the retinal surface and the retinal pigment epithelium layer) is extracted by image processing. Then, the layer thickness is measured by measuring the interval between the layer boundaries.

  Then, the control unit 70 calculates the thickness of each layer of the retina (for example, the retina surface layer and the retinal pigment epithelium layer) for each tomographic image. In the present embodiment, the control unit 70 creates a layer thickness map that graphically shows the analysis result of each tomographic image based on the acquired analysis result. The layer thickness map indicates, for example, two-dimensional distribution data related to the layer thickness of the fundus. After creating the layer thickness map, the control unit 70 stores the layer thickness map in the memory 72. The control unit 70 creates a layer thickness map for each front image.

  As described above, when the layer thickness map is created, the control unit 70 creates a difference map indicating a comparison result between the layer thickness maps. Of course, the control unit 70 may transmit the layer thickness map data of each apparatus to the monitor 75 and display it without creating the difference map.

  Here, since the three-dimensional tomographic image and the front image of each device are matched, the front image and the layer thickness map can be associated with each other. Thereby, alignment of the layer thickness map can be performed by performing alignment of the front image acquired by each imaging device. The control unit 70 creates a difference map indicating a difference result between the layer thickness maps in a common area in the front images acquired by the imaging device A and the imaging device B.

  As described above, even when the imaging device A and the imaging device B have different fundus front image acquisition means, a difference map of the fundus front image can be acquired, and observation between the imaging devices can be performed. it can. This makes it possible to detect changes in the fundus and perform follow-up observations even if there are cases where multiple years have passed since the previous observation and the specifications and measurement principles of the device have changed significantly. .

  Note that the data used as the front image as described above may be used for adjustment of the scanning position between different photographing apparatuses. For example, the imaging apparatus B acquires the SLO image acquired by the imaging apparatus A and the scanning position information corresponding to the three-dimensional tomographic image from the memory (database). Then, a positional shift between the OCT front image acquired by the imaging apparatus B and the SLO image is detected by image processing. Then, the optical scanner is controlled so as to correct the scanning position information acquired from the memory based on the detected positional deviation, thereby aligning the same part.

  In the present embodiment, for the ophthalmologic image processing apparatus 1, the tomographic image acquisition means is acquired by means common to the respective apparatuses, and the fundus front image acquisition means is acquired by different means. However, the present invention is not limited to this. The present invention is applicable to any device that acquires a common image type. For example, it is not only applied between spectral domain OCTs using the spectrum meter used in the present embodiment, but also Swept-source OCT (SS-OCT) or Time-domain OCT (TD-OCT) including a wavelength tunable light source. May be.

  In this case, for example, the actual size of the first 3D tomographic image and the second 3D tomographic image are the same size using the actual size data (depth direction and lateral direction) per unit pixel forming each 3D tomographic image. Thus, the magnification is adjusted in the depth direction, and the magnification is adjusted in the XY two-dimensional direction perpendicular to the depth direction.

  In the present embodiment, a fundus imaging apparatus that captures a fundus front image and a fundus tomographic image has been described as an example, but the present invention is not limited thereto. For example, the present invention can also be applied to an anterior segment imaging device that captures an anterior segment front image and an anterior segment tomographic image. In this case, the SLO scans the laser light on the anterior eye portion of the eye to be examined, and receives the reflected light from the anterior eye portion through the confocal aperture to obtain an anterior eye portion front image. Further, in the OCT front image capturing optical system, the OCT measurement light is scanned on the anterior eye portion of the eye to be examined, and the light obtained by combining the reflected light from the anterior eye portion and the reference light is received to obtain the front image of the anterior eye portion. obtain. Further, in the fundus camera optical system, illumination light is integrally irradiated to the anterior ocular segment to be examined, and reflected light from the anterior ocular segment is received by a two-dimensional image sensor to obtain an anterior ocular segment front image.

  Note that the present invention is not limited to the apparatus described in this embodiment. For example, ophthalmic image processing software (program) that performs the functions of the above embodiments is supplied to a system or apparatus via a network or various storage media. A computer of the system or apparatus (for example, a CPU) can also read and execute the program.

It is a figure explaining an example of the ophthalmic image processing apparatus which concerns on this embodiment. It is a block diagram explaining the structure of an image processing apparatus. It is a flowchart which shows the control operation of an ophthalmologic image processing apparatus. It is a figure which shows an example of the semi-automatic alignment screen displayed on a monitor. It is a figure which shows an example of the alignment confirmation screen displayed on a monitor. It is a figure explaining arrangement | positioning of the image data at the time of creating the synthetic | combination front image which is a display format of a check pattern pattern. It is a figure which shows an example of a synthetic | combination three-dimensional tomographic image.

DESCRIPTION OF SYMBOLS 1 Ophthalmic image processing apparatus 10 Image processing apparatus 25 Composite front image 70 Control part 72 Memory | storage part (memory)
75 Monitor 90 Operation input section

Claims (4)

The image data of the reference front image acquired by the ophthalmologic photographing apparatus and the image data of another front image acquired at a timing different from the reference front image are acquired from the storage unit and displayed on the monitor.
A feature portion on the reference front image can be selected based on an operation signal input from the operation input unit , and a feature portion corresponding to the feature portion selected in the reference front image is imaged in the other front image. Image processing for searching by processing and performing matching processing between the reference front image and the other front image by image processing based on the feature portion in the reference front image and the feature portion searched in the other front image It has a section, an ophthalmologic image processing apparatus that performs positioning between the different image data obtained by the ophthalmic photographing apparatus,
The reference front image is a front image associated with a first three-dimensional tomographic image acquired by an optical coherence tomography,
The other front image is a front image associated with a second three-dimensional tomographic image acquired by an optical coherence tomography and acquired at a different timing from the first three-dimensional tomographic image,
The image processing unit
Based on the matching result between the reference front image and the other front image, the first map showing the analysis result of the first three-dimensional tomographic image and the analysis result of the second three-dimensional tomographic image are shown. An ophthalmologic image processing apparatus characterized by performing alignment with a second map . The ophthalmic image processing apparatus according to claim 1.
The first map is a first layer thickness map showing a layer thickness analysis result of the first three-dimensional tomographic image,
The second map is a second layer thickness map showing a layer thickness analysis result of the second three-dimensional tomographic image,
The image processing unit
An ophthalmologic image processing apparatus that creates a difference map indicating a comparison result between the first map and the second map. In the ophthalmic image processing apparatus according to claim 1,
The other front image is an ophthalmologic image processing apparatus acquired by an ophthalmologic imaging apparatus different from the ophthalmologic imaging apparatus that acquired the reference front image. In an ophthalmic image processing apparatus program for performing alignment between different image data acquired by an ophthalmic imaging apparatus,
Acquiring from the storage unit image data of the reference front image acquired by the ophthalmologic photographing apparatus and image data of another front image acquired at a different timing from the reference front image;
A step of selecting a feature portion on the reference front image based on an operation signal input from an operation input unit;
Searching for a feature portion corresponding to the feature portion selected in the reference front image by image processing in the other front image;
An image processing step of performing matching processing between the reference front image and the other front image by image processing based on the feature portion in the reference front image and the feature portion searched in the other front image;
An ophthalmologic image processing program causing a computer to execute the,
The reference front image is a front image associated with a first three-dimensional tomographic image acquired by an optical coherence tomography,
The other front image is a front image associated with a second three-dimensional tomographic image acquired by an optical coherence tomography and acquired at a different timing from the first three-dimensional tomographic image,
The image processing step includes
Based on the matching result between the reference front image and the other front image, the first map showing the analysis result of the first three-dimensional tomographic image and the analysis result of the second three-dimensional tomographic image are shown. An ophthalmologic image processing program characterized by performing alignment with a second map .

Images