JP6927715B2 - Ophthalmic image display device, ophthalmic image display method and program - Google Patents

Description

The present invention relates to an ophthalmic image display device, an ophthalmic image display method and a program used for diagnosing an eye to be inspected.

In the field of ophthalmology, various images are acquired in order to observe the condition of the eye to be inspected. Images acquired in the field of ophthalmology in this way are called ophthalmic images.

Ophthalmologic images are acquired by various ophthalmologic imaging devices. Examples of ophthalmologic imaging devices include an optical coherence tomography that obtains a tomographic image using optical coherence tomography (OCT), a fundus camera that photographs the fundus, and an image of the fundus by laser scanning using a confocal optical system. There are scanning laser ophthalmoscopes (Scanning Laser Ophthalmoscope, SLO), slit lamps that obtain images by cutting out optical sections of the cornea using slit light, and the like.

Further, a thickness map showing a two-dimensional distribution of the layer thickness is generated based on the layer thickness information of the fundus obtained from the OCT signal. This thickness map and the thickness change map showing the time-series change of the layer thickness are also included in the ophthalmologic image.

By displaying these ophthalmic images on the display means, the state of the eye to be inspected can be observed.

For example, Patent Document 1 describes a device for simultaneously displaying frontal images and tomographic images of the fundus acquired at different dates and times on a monitor in a follow-up mode for performing follow-up observation. There is.

Japanese Unexamined Patent Publication No. 2013-27443

However, there is a drawback that it is difficult to visually understand the temporal change and it is difficult to recognize a minute change only by displaying the front image and the tomographic image side by side at the same time.

The present invention has been made in view of such circumstances, and is an ophthalmologic image display device and an ophthalmologic image that makes it possible to visually easily grasp temporal changes in ophthalmic images obtained at different times of the same patient. The purpose is to provide a display method and a program.

In order to achieve the above object, one aspect of the ophthalmic image display device is an image acquisition unit that acquires a plurality of captured images relating to the same eye to be inspected, or a plurality of analysis images relating to the shape of the same eye to be inspected, and a plurality of acquired images. It is provided with an alignment unit for aligning images, a video creation unit for creating a moving image by connecting a plurality of aligned images in chronological order, and a display unit for displaying the created moving image.

According to this aspect, a plurality of captured images relating to the same eye to be inspected or a plurality of analysis images relating to the shape of the eye to be inspected are acquired, the acquired plurality of images are aligned, and the plurality of aligned images are time-series. Since the moving images are created by connecting them in order and the created moving images are displayed, it is possible to easily visually grasp the temporal change of the ophthalmic images obtained at different times of the same patient.

The alignment unit includes a feature point extraction unit that extracts a plurality of feature points from the first image, which is one image of the plurality of images, and a plurality of second images other than the first image among the plurality of images. A corresponding point detection unit that detects a plurality of corresponding points corresponding to the feature points, and an image deformation unit that deforms the second image to match the positions of the plurality of corresponding points with the positions of the corresponding plurality of feature points. It is preferable to prepare. As a result, it is possible to appropriately align the acquired plurality of images.

It is preferable that the corresponding point detection unit detects a plurality of corresponding points by using template matching. Thereby, a plurality of corresponding points corresponding to the plurality of feature points can be appropriately detected from the second image.

It is preferable that the image deformation portion uses an affine transformation to match the positions of the plurality of corresponding points with the positions of the plurality of corresponding feature points. As a result, it is possible to appropriately align the acquired plurality of images.

The alignment unit includes a rotational movement amount calculation unit that calculates a rotational movement amount between a first image that is one of a plurality of images and a second image other than the first image, and a rotational movement amount calculation unit. An image rotation unit that aligns the rotation direction between the first image and the second image based on the rotation movement amount calculated by the above, and the first image and the second image that are aligned by the image rotation unit. Based on the translation amount calculation unit that calculates the translation amount between the first image and the second image and the translation amount calculated by the translation amount calculation unit by performing phase-limited correlation processing on the subject. It is preferable to include an image translation unit for aligning the first image and the second image in the parallel direction. As a result, it is possible to appropriately align the acquired plurality of images.

It is preferable that the rotation movement amount calculation unit calculates the rotation movement amount between the first image and the second image by performing phase-limited correlation processing on the first image and the second image. Thereby, the amount of rotational movement between the acquired plurality of images can be appropriately calculated.

The rotational movement amount calculation unit includes a first conversion processing unit that performs a discrete Fourier transform on the second image, a polar coordinate conversion unit that performs polar coordinate transformation on the calculation result of the first conversion processing unit for the second image, and a second. The second conversion processing unit that performs the discrete Fourier transform on the calculation result by the polar coordinate conversion unit for the image, the first data obtained in advance for the first image, and the calculation result by the second conversion processing unit for the second image. A first phase-limited compositing unit that performs a phase-limited compositing process that synthesizes the second data obtained based on the above, and a first inverse transform processing unit that performs an inverse discrete Fourier transform on the calculation result by the first phase-limited compositing unit. It is preferable to calculate the rotational movement amount based on the calculation result by the first inverse transform processing unit. Thereby, the amount of rotational movement between the acquired plurality of images can be appropriately calculated.

It is preferable to provide a brightness changing unit that matches the brightness of a plurality of images. Thereby, the temporal change of the ophthalmologic image can be grasped more easily visually.

It is equipped with an interpolated image creation unit that creates an interpolated image that interpolates between two images that are adjacent in chronological order among a plurality of images that are joined in chronological order, and the moving image creation unit includes the interpolated image. It is preferable to create a moving image. As a result, it is possible to easily visually grasp the temporal change of the ophthalmic image obtained at different times of the same patient.

The captured image is preferably a frontal image of the fundus or a tomographic image of the fundus. Further, the analysis image is preferably a thickness map showing a two-dimensional distribution of the layer thickness of the fundus, or a thickness change map showing a time-series change in the layer thickness. As a result, it is possible to easily visually grasp the temporal change of the photographed image and the analyzed image.

One aspect of the ophthalmic image display method for achieving the above object is an image acquisition step of acquiring a plurality of captured images relating to the same eye to be inspected or a plurality of analysis images relating to the shape of the same eye to be inspected, and a plurality of acquired images. It is provided with an alignment step of performing alignment, a moving image creation step of connecting a plurality of aligned images in chronological order to create a moving image, and a display step of displaying the created moving image on a display unit.

According to this aspect, a plurality of captured images relating to the same eye to be inspected or a plurality of analysis images relating to the shape of the eye to be inspected are acquired, the acquired plurality of images are aligned, and the plurality of aligned images are time-series. Since the moving images are created by connecting them in order and the created moving images are displayed, it is possible to easily visually grasp the temporal change of the ophthalmic images obtained at different times of the same patient.

One aspect of the program for achieving the above object is an image acquisition step of acquiring a plurality of captured images relating to the same eye to be inspected or a plurality of analysis images relating to the shape of the same eye to be inspected, and a plurality of acquired images. The alignment step of performing the alignment, the moving image creation step of connecting a plurality of aligned images in chronological order to create a moving image, and the display step of displaying the created moving image on the display unit are executed.

According to this aspect, a plurality of captured images relating to the same eye to be inspected or a plurality of analysis images relating to the shape of the eye to be inspected are acquired, the acquired plurality of images are aligned, and the plurality of aligned images are time-series. Since the moving images are created by connecting them in order and the created moving images are displayed, it is possible to easily visually grasp the temporal change of the ophthalmic images obtained at different times of the same patient.

A computer-readable non-temporary recording medium on which a program is recorded is also included in this embodiment.

According to the present invention, it is possible to visually easily grasp the temporal change of the ophthalmic image obtained at different times of the same patient.

Block diagram showing the configuration of an ophthalmic image display device A flowchart showing a moving image display process executed by an ophthalmic image display device. Diagram showing an example of a base image Diagram showing an example of an image for creating a video The figure which shows the shooting date of the base image and the image for moving image, and the order when arranged in chronological order. Block diagram showing the configuration of an ophthalmic image display device A flowchart showing a moving image display process executed by an ophthalmic image display device. The figure which showed the shooting interval of two adjacent images in chronological order Block diagram showing the configuration of an ophthalmic image display device Block diagram showing an example of the alignment unit A flowchart showing a moving image display process executed by an ophthalmic image display device.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
[Configuration of ophthalmic image display device]
FIG. 1 is a block diagram showing a configuration of an ophthalmic image display device 10 according to the first embodiment. As shown in FIG. 1, the ophthalmic image display device 10 includes a control unit 12, a communication unit 14, an image memory 16, an alignment unit 18, an image selection unit 25, a brightness change unit 26, a moving image creation unit 28, and an operation unit 30. , And a display unit 32 and the like.

The control unit 12 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory) (not shown). Various programs and control data are stored in the ROM. In addition, programs and data executed by the CPU are temporarily stored in the RAM.

The control unit 12 functions as a control means for controlling each part of the ophthalmic image display device 10 by executing a program read from the ROM in the CPU, and also functions as a calculation means for performing various calculation processes.

The communication unit 14 includes a communication interface (not shown), and transmits / receives data to / from the communication interface and an external server 100 connected to the communication interface. The server 100 constitutes a database of various ophthalmic images and diagnostic data.

The image memory 16 is a temporary storage means for various data including image data, and data is read / written through the control unit 12. The image data taken in from the server 100 via the communication unit 14 (an example of the image acquisition unit) is stored in the image memory 16.

The alignment unit 18 is an image processing means for aligning the positions of a plurality of input images, and includes a feature point extraction unit 20, a corresponding point detection unit 22, and an image deformation unit 24.

The feature point extraction unit 20 extracts the feature points included in the input image and specifies the position of the extracted feature points. Here, the feature point is a place in the image where the feature amount is relatively large, and corresponds to, for example, an edge or a vertex. The feature point extraction unit 20 calculates the feature amount of the entire image, and extracts pixels in which the calculated feature amount exceeds the threshold value as feature points.

The corresponding point detection unit 22 extracts the corresponding points included in the input image and specifies the position of the extracted corresponding points. The corresponding points are points in the image having the same features as the feature points extracted by the feature point extraction unit 20. The corresponding point detection unit 22 calculates the feature amount of the entire image, and extracts points having a high degree of similarity between the calculated feature amount and the feature amount of the feature point as corresponding points.

The image deformation unit 24 transforms the input image by performing translation processing, rotation movement processing, enlargement processing, and reduction processing on the input image. Here, in particular, based on the positional relationship between the feature points of the image in which the feature points are extracted by the feature point extraction unit 20 and the corresponding points in the image in which the corresponding points are detected by the corresponding point detecting unit 22, the corresponding point detecting unit 22 Transforms the image in which the corresponding point is detected by.

The image selection unit 25 selects a desired ophthalmic image from the ophthalmic images recorded on the server 100.

The brightness changing unit 26 changes the brightness of the input image.

The moving image creation unit 28 creates a moving image in which a plurality of images aligned by the alignment unit 18 are combined in chronological order and each image is displayed in order.

The operation unit 30 includes operation members such as operation buttons and a keyboard, and outputs operation information input from the operation members to the control unit 12. The control unit 12 executes various processes according to the operation information input from the operation unit 30.

The display unit 32 displays various images including the moving image created by the moving image creating unit 28, diagnostic information about the patient, and the like. As the display unit 32, for example, a liquid crystal display, an organic EL display, or the like can be applied. A touch panel in which the operation unit 30 and the display unit 32 are integrated may be used.

[Video display method]
FIG. 2 is a flowchart showing a moving image display process (an example of an ophthalmic image display method) executed by the ophthalmic image display device 10 according to the present embodiment.

In the present embodiment, a plurality of ophthalmic images of the same eye of the same patient and the same eye to be inspected are combined to create a moving image, and the created moving image is displayed so that the observer can visually recognize the temporal change of the ophthalmic image. Easy to grasp.

The ophthalmic image includes a photographed image of the eye to be inspected by various ophthalmologic imaging devices and an analysis image regarding the shape of the eye to be inspected created from the analysis result. Captured images also include a frontal image of the fundus, a two-dimensional tomographic image of the fundus, a three-dimensional tomographic image, and a fluorescence image that captures fluorescence excited by a fluorescent substance. Further, the analysis image includes a thickness map showing the two-dimensional distribution of the layer thickness of the fundus and a thickness change map showing the time-series change of the layer thickness.

Here, it is assumed that all the ophthalmic images are recorded on the server 100.

First, the control unit 12 initializes the variable n to 0 (step S1).

_{Next, the observer selects the base image P 0} (an example of the first image) relating to the eye to be inspected for which the moving image is to be created (step S2, an example of the image acquisition step).

To select the base image P ₀ , the control unit 12 displays a list of ophthalmic images recorded on the server 100 on the display unit 32, and the observer selects a desired image from the list display using the operation unit 30. Alternatively, the observer may input the patient's name to automatically select the ophthalmologic image associated with the input name in the control unit 12.

As a result, the selected image is acquired by the ophthalmic image display device 10 via the communication unit 14, and is stored in the image memory 16 as _{the base image P 0.} The base image P ₀ is not limited to the shooting time or the acquisition time, and the observer can select _{any image as the base image P 0.}

Subsequently, the feature point extraction unit 20 extracts a plurality of feature points from the base image P ₀ (step S3).

Figure 3 is a diagram showing an example of the base image P _0, which is stored in the image memory 16. Here, an example in which a fundus photograph taken by a fundus camera is selected as the base image P _{0 is shown.}

As shown in FIG. 3, the feature point extraction unit 20 extracts the _{feature point A 1} including the optic nerve head, and the _{feature points A 2 and A 3} including the bifurcation of the blood vessel _{from the base image P 0} . The plurality of feature points may be determined by the observer designating an arbitrary point.

When the extraction of the feature points from the base image P ₀ is completed, the control unit 12 increments the variable n (step S4).

Next, the image selection unit 25 selects an image _Pn for creating a moving image (an example of a second image other than the first image) for creating a moving image (step S5, an example of an image acquisition step).

The image selection unit 25 selects an ophthalmic image of the same eye as the base image P ₀ and the same type of ophthalmic image as _{the base image P 0 among the ophthalmic images recorded on the server 100.} As a result, the selected image is acquired by the ophthalmic image display device 10 via the communication unit 14, and is stored in the image memory 16 as the _{moving image creation image Pn.}

The same type of ophthalmic image refers to a photographed image taken by the same type of device or an analysis image represented by the same expression method based on the same index. The same type of device does not have to be the same device up to the manufacturer and model number, as long as the imaging principle is the same.

Figure 4 is a diagram showing an example of a moving image creating images P ₁ to create a video. The moving image creation image P ₁ is a fundus photograph of the same eye to be inspected as in the base image P _0.

Subsequently, the corresponding point detection unit 22 detects a plurality of feature points from the _{moving image creation image Pn (step S6).} For the detection of the corresponding points, for example, template matching, a phase-limited correlation method, or the like can be used.

Here, as shown in FIG. 4, corresponding points B _1, the corresponding point as the corresponding point corresponding to the feature point A ₂ B _2, and the corresponding points corresponding to the characteristic point A ₃ as the corresponding point corresponding to the feature point A ₁ detects a corresponding point B ₃ are as.

Then, the image deforming unit 24, based on the image P ₀ by modifying the image P _n for movie created as a reference, the alignment of the base image P ₀ and the moving creating image P _n performs (step S7, positioning step An example).

Here, the coordinate position of each corresponding point B _n of video creating image P _n to deform the image P _n for moving created to match the coordinate position of each feature point A _n of the base image P _0. For example, when the position of the corresponding point B _n is moving parallel to the X direction, Y direction or XY directions relative to the position of each feature point A _n, the image deformation unit 24, the image P _n for creating video Is translated. Further, when the position of the corresponding point B _n is rotating moves the XY plane relative to the position of each feature point A _n, the image deformation unit 24 rotates moving the image P _n for creating video.

Further, if the positional relationship between the corresponding points B _n is shrinking relative positional relationship of each feature point A _n, the image deformation unit 24 expands the image P _n for creating video. Further, if the positional relationship between the corresponding points B _n is enlarged relative positional relationship of each feature point A _n, the image deformation unit 24 reduces the image P _n for creating video.

For these image deformations, the movement vector may be determined so that the sum of the distances _{between each feature point An} and each corresponding point B _{n is minimized.} The parallel movement process, the rotational movement process, the enlargement process, and the reduction process can be performed by the affine transformation.

The moving image creation image P _n deformed in this way is an image in which the position is aligned with the _{base image P 0.}

Subsequently, the brightness change unit 26, the brightness of the moving image creating image P _n to change the brightness of the moving image creating image P _n to match the brightness of the base image P ₀ (step S8).

Here, the brightness correction is performed by matching the average luminance value of _{each moving image creation image P n with} the average luminance value of the base image P _0. Further, the dynamic range of each moving image P _n may be changed so as to match the dynamic range of the base image P _0.

Note that, although change the brightness of the moving image creating images P _n in accordance with the brightness of the base image P ₀ is, if it is possible to match the base image P ₀ and the brightness of the moving image creating images P _n, The reference brightness may be arbitrarily determined by the observer.

Next, the control unit 12 _{determines whether or not the image selection unit 25 has completed the selection of the moving image creation image Pn} for creating the moving image (step S9).

The server 100, the base image P ₀ and an ophthalmologic image of the same eye to be examined, and in the base image P ₀ and the same type of ophthalmologic image, if there is an ophthalmologic image not yet been selected, the process proceeds to step S4, The same process is repeated. That is, n is incremented (step S4), the moving image creation image _Pn is selected (step S5), and the same processing is performed (steps S6 to S9).

_{The image Pn} for creating a moving image may be manually selected by the observer using the operation unit 30.

If selection of the image is finished, moving image creating unit 28, the base image _{P 0} and the image _P 1 to P _n for moving image creating, when sorted in chronological order (step S10).

In the header portion of the ophthalmic image stored in the server 100, the shooting date and time of the captured image is recorded, and if it is the analysis image, the shooting date and time of the captured image that is the basis of the analysis is recorded.

FIG. 5 is a diagram showing the shooting dates of the base image P ₀ and the moving image creation images P _{1 to} P ₅ and the order when arranged in chronological order. Here, they are arranged in chronological order in chronological order.

As shown in FIG. 5, the base image P ₀ and the shooting date of the moving image creating images P ₁ to P ₅ each base image P ₀ June 22, 2013, the moving image creating image P ₁ July 2014 1st, video creation image P ₂ is December 15, 2013, video creation image P ₃ is January 12, 2016, video creation image P ₄ is July 21, 2016, and video creation image P ₅ is a January 31, 2015. Thus, when arranging the image to the old order of the shooting date, the base image P _0, image P for creating video _2, video created for the image P _1, the video created for the image P _5, the video created for the image P _3, and videos for creating image made in the order of P _4.

Subsequently, the moving image creating unit 28, when the base image P ₀ and the image P ₁ to P _n for the moving image creating sorted in chronological order chronologically combine to create a time-lapse video (step S11, the moving image creating process One case).

Finally, the control unit 12 displays the moving image created by the moving image creating unit 28 on the display unit 32 (step S12, an example of the display process). The control unit 12 may record this moving image on the server 100.

In this way, by using a moving image in which a plurality of ophthalmic images are continuously displayed in chronological order, the observer can easily visually grasp the temporal change of the same eye to be inspected. Here, an example in which the frontal image of the fundus is made into a moving image has been described, but the same effect can be obtained by making a moving image of a two-dimensional tomographic image, a three-dimensional tomographic image, a fluorescence image, a thickness map, a thickness change map, etc. of the fundus. Obtainable.

<Second embodiment>
[Configuration of ophthalmic image display device]
FIG. 6 is a block diagram showing the configuration of the ophthalmic image display device 40 according to the second embodiment. The parts common to the block diagram shown in FIG. 1 are designated by the same reference numerals, and detailed description thereof will be omitted.

As shown in FIG. 6, the ophthalmic image display device 40 includes an interpolated image creating unit 34.

The interpolated image creation unit 34 creates an interpolated image that interpolates between the two input images in chronological order. Here, the interpolated image creation unit 34 detects a motion vector from the two images by block matching processing, and creates an interpolated image based on the detected motion vector.

[How to create a video]
FIG. 7 is a flowchart showing a moving image display process executed by the ophthalmic image display device 40 according to the present embodiment. The parts common to the flowchart shown in FIG. 2 are designated by the same reference numerals, and detailed description thereof will be omitted.

In the present embodiment, the temporal change of the ophthalmic image is more easily grasped visually by creating the morphing by interpolating based on the shooting date of the image for which the moving image is created.

The processes from step S1 to step S10 are the same as those in the first embodiment. Base image P ₀ and the image P _n for the moving image creating selected in the present embodiment is assumed to be the same as the base image P ₀ and the moving image creating images P ₁ to P ₅ shown in FIG.

In step S10, the moving image creation section 28 sorts the base image P ₀ and the image P ₁ to P _n for moving created in chronological order, time series between the two images interpolated image forming unit 34, which when adjacent in chronological order An interpolated image to be interpolated is created (step S21). Here, a number of interpolated images are created according to the shooting interval of two adjacent images in chronological order.

Figure 8 arranges the base image P ₀ and the moving image creating images P ₁ to P ₅ in chronological order, is a diagram illustrating a photographing interval of the two images adjacent in time series order. As shown in FIG. 8, _{the shooting interval between the base image P 0} and the moving image creation image P ₂ is about 6 months, the shooting interval between the moving image creation image P ₂ and the moving image creation image P ₁ is about 6 months, and the moving image. shooting interval between creating image P ₁ and the moving image shooting interval between creating image P ₅ is from about 6 months, moving creating image P ₅ and video creating image P ₃ is about 12 months, and moving creating image P ₃ shooting interval of the video created for the image P ₄ and is about 6 months.

Here, it is assumed that an interpolated image is created every about 3 months. Accordingly, the number of required interpolated images, interpolated images to interpolate between the base image P ₀ and the moving creating image P ₂ is one, between the moving image creating image P ₂ and moving creating image P ₁ interpolation image to interpolate is one interpolation image for interpolating between the moving image creating image P ₁ and the moving image creating image P ₅ is one, between the moving image creating image P ₅ and video creating image P ₃ interpolation image for interpolating three, and the interpolation image for interpolating between the moving image creating image P ₃ and moving creating image P ₄ is one.

The interpolated image creation unit 34 creates a required number of interpolated images. For example, in the case of _{an interpolated image of the base image P 0} and the moving image creation image P ₂ , an interpolated image that divides the motion vector detected from these two images into two equal parts is created. Further, if the interpolated image and the moving image creating image P ₅ and video creating image P _3, generating an interpolation image to 4 equally dividing the detected motion vector from the two images.

After creating the interpolated image, moving image creating unit 28, when the sorted based image P ₀ and the image P ₁ to P _n and the respective interpolated image for a moving image created in chronological order chronologically combine to create a time-lapse video (Step S11). Further, the control unit 12 displays the moving image created by the moving image creating unit 28 on the display unit 32 (step S12).

The moving image created in this way smoothly reproduces how the previous image changes in chronological order to the later image in chronological order. Therefore, the observer can more easily visually grasp the temporal change of the same eye to be inspected.

Further, by creating a number of interpolated images according to the shooting interval of two adjacent images in chronological order, it is possible to accurately reproduce the temporal change of the same eye to be inspected. The number of interpolated images to be created is not particularly limited, and generally, the number of interpolated images that can be easily visually recognized as a moving image may be appropriately created.

<Third embodiment>
[Configuration of ophthalmic image display device]
FIG. 9 is a block diagram showing the configuration of the ophthalmic image display device 50 according to the third embodiment. The parts common to the block diagram shown in FIG. 6 are designated by the same reference numerals, and detailed description thereof will be omitted.

The alignment unit 18 of the ophthalmic image display device 50 _{calculates the amount of rotational movement between the base image P 0} and the moving image P _n at the sub-pixel level, and based on this amount of rotational movement, the base image P ₀ and aligning the rotational direction between the moving image creating image P _n. After that, the alignment unit 18 _{calculates the amount of translation between the aligned base image P 0} and the moving image creation image P _n at the sub-pixel level, and based on this translation amount, the base image P ₀ And the moving image creation image _Pn are aligned in the parallel direction.

In order to realize such processing, as shown in FIG. 9, the alignment unit 18 includes a rotation movement amount calculation unit 52, an image rotation amount 54, a translation amount calculation unit 56, and an image translation amount calculation unit 58. ing.

The alignment unit 18 calculates the amount of rotational movement and the amount of translation between the _{base image P 0} and the moving image P _n by performing phase-limited correlation processing described later. Here, the alignment unit 18 calculates in advance the POC (Phase Only Correlation) data (first data) of the base image used for the phase-limited correlation processing. Hereinafter, the POC data for the base image P ₀ may be referred to as the base POC data, and the POC data for the moving image creation image P _n may be referred to as the target POC data. When the target image POC data (second data) is calculated, the alignment unit 18 continues the phase-limited correlation processing using the target POC data and the base POC data obtained in advance. As a result, the processing load is reduced and the processing time is shortened, and a minute amount of misalignment is calculated.

(Phase-limited correlation processing)
In the phase-limited correlation processing in the present embodiment, a known phase-limited correlation function is used.

_{First, the base image P 0} having an image size of N ₁ × N ₂ (N ₁ and N ₂ are positive integers) is f (n ₁ , n ₂ ), and the moving image P _n is g (n ₁ , n). ₂ ). _{Here, n 1} = −M ₁ , ···, M ₁ in the discrete space, N ₁ = 2M ₁ + 1 (M ₁ is a positive integer), and two-dimensional discrete _{f (n 1} , n _2). Assuming that the calculation result of the Discrete Fourier Transformation (DFT) is F (k ₁ , k ₂ ), F (k ₁ , k ₂ ) is expressed by Eq. (1).

In equation (1), AF (k ₁ , k ₂ ) is the amplitude component of f (n ₁ , n ₂ ^{), and e jθF} (k 1, k 2) is the phase component of f (n ₁ , n _2). ..

Similarly, in a discrete space, n ₂ = -M ₂ , ..., M ₂ , N ₂ = 2 M ₂ + 1 (M ₂ is a positive integer), and g (n ₁ , n ₂ ) two-dimensional DFT. Assuming that the calculation result of is G (k ₁ , k ₂ ), G (k ₁ , k ₂ ) is expressed as in Eq. (2).

In equation (2), _AG (k ₁ , k ₂ ) is the amplitude component of g (n ₁ , n ₂ ^{), and e jθG} (k 1, k 2) is the phase component of g (n ₁ , n _2). be.

The phase-limited synthesis function used in the phase-limited synthesis processing is defined as in equation (3) using equations (1) and (2).

The phase-limited correlation function in the present embodiment is as shown in the equation (4) by applying a two-dimensional inverse discrete Fourier transform (IDFT) to the phase-limited composite function of the equation (3). expressed.

The 2-dimensional images _s c of the continuous space is defined _(x 1, _{x 2),} by a minute amount of movement [delta] ₁ in _{x 1} direction and the image obtained by shifting the _{x 2} direction by a minute amount of movement [delta] ₂ is is expressed as _{s c (x 1 -δ 1,} x 2 -δ 2). The two-dimensional image f (n ₁ , n ₂ ) on the discrete space sampled at the sampling interval T ₁ is defined by Eq. (5).

Similarly, the two-dimensional image g (n ₁ , n _{2 ) on the discrete space sampled at the sampling interval T 2} is defined by Eq. (6).

In equations (5) and (6), n ₁ = -M ₁ , ..., M ₁ and n ₂ = -M ₂ , ..., M ₂ . As a result, the phase-limited correlation function for the two-dimensional images f (n ₁ , n ₂ ) and g (n ₁ , n ₂ ) in the discrete space is expressed in the general form as in Eq. (7). In equation (7), α = 1.

The alignment unit 18 that performs the above-mentioned phase-limited correlation processing will be described. FIG. 10 is a block diagram showing an example of the alignment unit 18.

(Rotation movement amount calculation unit)
The rotation movement amount calculation unit 52 calculates the rotation movement amount between the _{base image P 0} and the moving image creation image P _n. Specifically, the rotational movement amount calculating section 52, by contrast with the base image P ₀ and the moving creating image P _n subjected to phase-only correlation processing between the base image P ₀ and the moving creating image P _n Calculate the amount of rotational movement of. As shown in FIG. 10, such a rotational movement amount calculation unit 52 includes a first conversion processing unit 60, a logarithmic conversion unit 62, a polar coordinate conversion unit 64, and a second conversion processing unit 66. It includes a first phase limited synthesis unit 68 and a first inverse conversion processing unit 70.

The first conversion processing unit 60 performs two-dimensional DFT processing on _{the base image P 0.} Further, the first conversion processing unit 60 performs two-dimensional DFT processing on _{the moving image creation image Pn.} The two-dimensional DFT process performed by the first conversion processing unit 60 includes a two-dimensional DFT and a known shift process for shifting the quadrant with respect to the calculation result of the two-dimensional DFT. Hereinafter, this shift process may be referred to as "shift". The two-dimensional DFT performed by the first transformation processing unit 60 may be a two-dimensional FFT (Fast Fourier Transformation).

The logarithmic conversion unit 62 _{performs logarithmic conversion on the calculation result of the base image P 0} by the first conversion processing unit 60. Further, the logarithmic conversion unit 62 performs logarithmic conversion on the calculation result of the first conversion processing unit 60 _{for the moving image creation image Pn.} The logarithmic conversion by the logarithmic conversion unit 62 has the effect of compressing the amplitude spectrum that tends to be concentrated in the low frequency region of the spatial frequency in the natural image.

The polar coordinate conversion unit 64 performs polar coordinate conversion on the calculation result of the logarithmic conversion unit 62 for the _{base image P 0.} Further, the polar coordinate conversion unit 64 performs polar coordinate conversion on the calculation result by the logarithmic conversion unit 62 _{for the moving image P n.} When the logarithmic conversion by the logarithmic conversion unit 62 is not performed, the polar coordinate conversion unit 64 performs polar coordinate conversion on the calculation result by the first conversion processing unit 60 for the _{base image P 0 , and the moving image P n} is subjected to polar coordinate conversion. Polar coordinate conversion is performed on the calculation result by the first conversion processing unit 60. Polar conversion by polar coordinate transformation unit 64 is a process of converting the movement amount of the rotational direction to the amount of movement of the parallel direction (n ₁ direction and n ₂ direction) in the formula (1) to (7).

As shown in the equation (1), the second conversion processing unit 66 performs two-dimensional DFT processing (two-dimensional DFT + shift) on the calculation result of the polar coordinate conversion unit 64 for the _{base image P 0.} The processing result of the base image P ₀ by the second conversion processing unit 66 is controlled as, for example, the base POC data (first data) normalized by the amplitude component prior to the arithmetic processing by the first phase limited synthesis unit 68. It is stored in advance in a ROM (not shown) of the unit 12. Further, as shown in the equation (2), the second conversion processing unit 66 performs two-dimensional DFT processing (two-dimensional DFT + shift) on the calculation result by the polar coordinate conversion unit 64 _{for the moving image creation image Pn.} The two-dimensional DFT performed by the second conversion processing unit 66 may also be a two-dimensional FFT.

As shown in the equation (3), the first phase limited synthesis unit 68 is a second conversion processing unit for the base POC data (first data) obtained in advance for _{the base image P 0 and the moving image creation image P n.} A phase-limited synthesis process for synthesizing the target POC data (second data) normalized by the amplitude component based on the calculation result by 66 is performed.

The first inverse transformation processing unit 70 performs two-dimensional IDFT processing on the calculation result by the first phase limited synthesis unit 68. The two-dimensional IDFT process performed by the first inverse transformation processing unit 70 includes a two-dimensional IDFT and a known shift process for shifting the quadrant with respect to the calculation result of the two-dimensional IDFT. The two-dimensional IDFT may be calculated by a two-dimensional Inverse Fast Fourier Transformation (hereinafter referred to as IFFT).

The rotation movement amount calculation unit 52 calculates the rotation movement amount based on the calculation result by the first inverse conversion processing unit 70. Specifically, the rotational movement amount calculation unit 52 specifies the peak position based on the calculation result by the first inverse transformation processing unit 70, so that the rotational movement amount (rotational movement amount, misalignment amount) at the pixel level. ) Is calculated. After that, the rotation movement amount calculation unit 52 specifies the pixel position when the correlation value of the phase-limited correlation function shown in the equation (7) becomes maximum in the vicinity of the peak position specified at the pixel level, thereby substituting. The amount of movement in the rotation direction (rotational movement amount, misalignment amount) is calculated at the pixel level.

The rotation movement amount calculation unit 52 has been described as calculating the movement amount of the moving _{image P n in} the rotation direction with reference to _{the base image P 0, but the present invention is not limited to this.} The rotation movement amount calculation unit 52 may calculate the movement amount in the rotation direction of the base image with reference to the target image.

Further, the rotational movement amount calculation unit 52 does not have to calculate the rotational movement amount by phase-limited correlation processing, calculates the rotational movement amount by a known method, and uses the calculated rotational movement amount as the image rotation unit 54. It may be output to.

(Image rotation part)
_{The image rotation unit 54 aligns the base image P 0} and the moving image creation image P _n in the rotation direction based on the rotation movement amount calculated by the rotation movement amount calculation unit 52. _{Specifically, the image rotation unit 54 uses the base image P 0} as a reference based on the rotation movement amount calculated by the rotation movement amount calculation unit 52, so that the rotation movement amount becomes zero. _{Aligning in the} rotation direction with respect to n.

(Translation amount calculation unit)
The translation amount calculation unit 56 calculates the translation amount between the _{base image P 0} aligned by the image rotation unit 54 and the moving image creation image P _n. Specifically, the translation amount calculation unit 56 performs phase-limited correlation processing on _{the base image P 0} aligned by the image rotation unit 54 and the moving image creation image P _{n , thereby causing the base image P 0.} The amount of translation between the _{image P n} and the moving image P n is calculated. As shown in FIG. 10, such a translation amount calculation unit 56 includes a third conversion processing unit 72, a second phase limited synthesis unit 74, and a second inverse conversion processing unit 76.

As shown in the equation (1), the third conversion processing unit 72 performs two-dimensional DFT processing (two-dimensional DFT + shift) _{on the base image P 0.} The processing result of the base image P ₀ by the third conversion processing unit 72 is stored, for example, as base POC data (third data) normalized by the amplitude component prior to the arithmetic processing by the second phase limited synthesis unit 74. Pre-stored in section 212. Further, as shown in the equation (2), the third conversion processing unit 72 performs a two-dimensional DFT process (two-dimensional DFT + shift) _{on the moving image creation image Pn.} The two-dimensional DFT performed by the third conversion processing unit 72 may be a two-dimensional FFT.

As shown in the equation (3), the second phase limited synthesis unit 74 is a third conversion processing unit for the base POC data (third data) obtained in advance for _{the base image P 0 and the moving image creation image P n.} Based on the calculation result according to 72, a phase-limited synthesis process for synthesizing the target POC data (fourth data) normalized by the amplitude component is performed.

The second inverse transformation processing unit 76 performs two-dimensional IDFT processing (two-dimensional IDFT + shift) on the calculation result by the second phase limited synthesis unit 74. The two-dimensional IDFT performed by the second inverse transform processing unit 76 may be a two-dimensional IFFT.

The translation amount calculation unit 56 calculates the translation amount based on the calculation result by the second inverse transformation processing unit 76. Specifically, the translation amount calculation unit 56 specifies the peak position based on the calculation result by the second inverse transformation processing unit 76, so that the translation amount in the parallel direction (translation amount, misalignment amount) at the pixel level. ) Is calculated. After that, the translation amount calculation unit 56 subordinates by specifying the pixel position when the correlation value of the phase-limited correlation function shown in the equation (7) becomes maximum in the vicinity of the peak position specified at the pixel level. Calculate the amount of movement in the parallel direction (translation amount, misalignment amount) at the pixel level.

The translation amount calculation unit 56 has been described as calculating the movement amount of the moving _{image P n in} the parallel direction with reference to _{the base image P 0, but the present invention is not limited to this.} The translation amount calculation unit 56 may calculate the movement amount of the base image P _{0 in} the parallel direction _{based on the moving image P n.}

The translation amount calculated by the translation amount calculation unit 56 is output to the image translation amount calculation unit 58.

(Image translation part)
_{The image translation unit 58 aligns the base image P 0} and the moving image creation image P _n in the parallel direction based on the translation amount calculated by the translation amount calculation unit 56. Specifically, the image translation unit 58 is an image for creating a moving image so that the translation amount becomes zero _{based on the base image P 0} based on the translation amount calculated by the translation amount calculation unit 56. Align in the direction parallel to P _n.

[How to create a video]
FIG. 11 is a flowchart showing a moving image display process executed by the ophthalmic image display device 50 according to the present embodiment. The parts common to the flowchart shown in FIG. 7 are designated by the same reference numerals, and detailed description thereof will be omitted.

In this embodiment, phase-limited correlation processing is used for image alignment when creating a moving image.

Similar to the second embodiment, the variable n is initialized to 0 (step S1), and the observer _{selects the base image P 0} (step S2).

Next, the variable n is incremented (step S4), and the moving image creation image P _n for creating the moving image is selected (step S5).

Subsequently, the rotation movement amount calculation unit 52 calculates the rotation movement amount between the _{base image P 0} and the moving image creation image P _{n (step S31). Further, the image rotation unit 54 aligns the base image P 0} and the moving image creation image P _n in the rotation direction based on the rotation movement amount calculated by the rotation movement amount calculation unit 52 (step S32). ..

Next, the translation amount calculation unit 56 calculates the translation amount between the _{base image P 0} aligned by the image rotation unit 54 and the moving image creation image P _{n (step S33). Further, the image translation unit 58 aligns the base image P 0} and the moving image creation image P _n in the parallel direction based on the translation amount calculated by the translation amount calculation unit 56 (step S34). ).

Hereinafter, as in the second embodiment, by changing the brightness of the moving image creating images P _n as the brightness of the moving image creating images P _n coincides with the brightness of the base image P ₀ (step S8), and The same process is repeated until the selection of the moving image creation image _{Pn for creating the moving image is completed (step S9).}

As described above, according to the present embodiment, a more appropriate moving image can be created by using the phase-limited correlation process for the alignment of the image when creating the moving image.

<Others>
In the embodiments described so far, for example, the control unit 12, the alignment unit 18, the brightness change unit 26, the moving image creation unit 28, the interpolation image creation unit 34, the rotation movement amount calculation unit 52, the image rotation unit 54, and the translation unit. Quantity calculation unit 56, image translation unit 58, first conversion processing unit 60, logarithmic conversion unit 62, polar coordinate conversion unit 64, second conversion processing unit 66, first phase limited synthesis unit 68, first inverse conversion processing unit 70. The hardware structure of the processing unit that executes various processes such as the third conversion processing unit 72, the second phase limited synthesis unit 74, and the second inverse conversion processing unit 76 is as shown below. Various processors. For various processors, the circuit configuration can be changed after manufacturing the CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), etc., which are general-purpose processors that execute software (programs) and function as various processing units. Includes a dedicated electric circuit, which is a processor having a circuit configuration specially designed for executing a specific process such as a programmable logic device (PLD), an ASIC (Application Specific Integrated Circuit), etc. Is done.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). You may. Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a server and a client. There is a form in which a processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. be. As described above, the various processing units are configured by using one or more various processors as a hardware-like structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

The ophthalmic image display method according to the present embodiment is configured as a program for causing a computer to execute each of the above steps, and is non-temporary such as a CD-ROM (Compact Disk-Read Only Memory) storing the configured program. It is also possible to configure a recording medium.

The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined between the respective embodiments without departing from the spirit of the present invention.

10 Ophthalmology image display device 12 Control unit 14 Communication unit 16 Image memory 18 Alignment unit 20 Feature point extraction unit 22 Corresponding point detection unit 24 Image deformation unit 25 Image selection unit 26 Brightness change unit 28 Video creation unit 30 Operation unit 32 Display Unit 34 Interpolated image creation unit 40 Ophthalmology image display device 50 Ophthalmology image display device 52 Rotational movement amount calculation unit 54 Image rotation unit 56 Translation amount calculation unit 58 Image parallel movement unit 60 First conversion processing unit 62 Logistic conversion unit 64 Polar coordinate conversion Part 66 Second conversion processing unit 68 First phase limited synthesis unit 70 First inverse conversion processing unit 72 Third conversion processing unit 74 Second phase limited synthesis unit 76 Second inverse conversion processing unit 100 Server A ₁ Feature point A ₂ Features point _{a 3} characteristic point _{a n} feature points _{B 1} corresponding point _{B 2} corresponding point _{B 3} corresponding point _{B n} corresponding point S1~S34 video display processing step

Claims (13)

An image acquisition unit that acquires a plurality of captured images relating to the same eye to be inspected or a plurality of analysis images relating to the shape of the same eye to be inspected.
An alignment unit that aligns the acquired plurality of images, and an alignment unit.
A video creation unit that creates a video by connecting a plurality of aligned images in chronological order,
A display unit that displays the created video and
An interpolation image creation unit that creates an interpolated image that interpolates in chronological order between two images adjacent to each other in chronological order among a plurality of images that are joined in chronological order.
Equipped with a,
The interpolated image creation unit creates a number of the interpolated images according to the shooting interval of two adjacent images in the time series order.
The moving image creation unit is an ophthalmic image display device that creates the moving image including the interpolated image. The alignment part is
A feature point extraction unit that extracts a plurality of feature points from the first image, which is one of the plurality of images, and a feature point extraction unit.
A correspondence point detection unit that detects a plurality of correspondence points corresponding to the plurality of feature points from a second image other than the first image among the plurality of images.
An image deformation unit that deforms the second image to match the positions of the plurality of corresponding points with the positions of the corresponding plurality of feature points.
The ophthalmic image display device according to claim 1. The ophthalmic image display device according to claim 2, wherein the corresponding point detection unit detects the plurality of corresponding points by using template matching. The ophthalmic image display device according to claim 2 or 3, wherein the image transforming unit uses an affine transformation to match the positions of the plurality of corresponding points with the positions of the corresponding plurality of feature points. The alignment part is
A rotational movement amount calculation unit that calculates a rotational movement amount between a first image that is one of the plurality of images and a second image other than the first image.
An image rotation unit that aligns the rotation direction between the first image and the second image based on the rotation movement amount calculated by the rotation movement amount calculation unit.
The amount of translation between the first image and the second image is calculated by performing phase-limited correlation processing on the first image and the second image that have been aligned by the image rotating portion. Translation amount calculation unit and
An image translation unit that aligns the first image and the second image in the parallel direction based on the translation amount calculated by the translation amount calculation unit.
The ophthalmic image display device according to claim 1. The claim that the rotational movement amount calculation unit calculates the rotational movement amount between the first image and the second image by performing phase-limited correlation processing on the first image and the second image. 5. The ophthalmic image display device according to 5. The rotational movement amount calculation unit
A first transform processing unit that performs a discrete Fourier transform on the second image,
A polar coordinate conversion unit that performs polar coordinate conversion on the calculation result of the first conversion processing unit for the second image, and a polar coordinate conversion unit.
A second conversion processing unit that performs a discrete Fourier transform on the calculation result of the polar coordinate conversion unit for the second image,
A first phase that performs a phase-limited synthesis process for synthesizing the first data obtained in advance for the first image and the second data obtained based on the calculation result of the second conversion processing unit for the second image. Limited synthesis section and
A first inverse transform processing unit that performs an inverse discrete Fourier transform on the calculation result by the first phase limited synthesis unit is provided.
The ophthalmologic image display device according to claim 6, wherein the rotational movement amount is calculated based on the calculation result by the first inverse transformation processing unit. The ophthalmologic image display device according to any one of claims 1 to 7, further comprising a brightness changing unit for matching the brightness of the plurality of images. The ophthalmic image display device according to any one of claims 1 to 8, wherein the interpolated image creating unit creates the interpolated image that equally divides motion vectors detected from two adjacent images in chronological order. The ophthalmologic image display device according to any one of claims 1 to 9, wherein the captured image is a frontal image of the fundus or a tomographic image of the fundus. The ophthalmic image according to any one of claims 1 to 10, wherein the analysis image is a thickness map showing a two-dimensional distribution of the layer thickness of the fundus, or a thickness change map showing a time-series change in the layer thickness. Display device. An image acquisition step of acquiring a plurality of captured images relating to the same eye to be inspected or a plurality of analysis images relating to the shape of the same eye to be inspected.
The alignment step of aligning the plurality of acquired images and the alignment step.
A video creation process for creating a video by connecting a plurality of aligned images in chronological order, and
A display process for displaying the created moving image on the display unit, and
An interpolation image creation step of creating an interpolated image that interpolates in time series between two images adjacent to each other in the time series order among a plurality of images joined in the time series order.
Equipped with a,
In the interpolated image creation step, a number of the interpolated images corresponding to the shooting intervals of two adjacent images in the time series order are created.
The moving image creation step is an ophthalmic image display method for creating the moving image including the interpolated image. On the computer
An image acquisition step of acquiring a plurality of captured images relating to the same eye to be inspected or a plurality of analysis images relating to the shape of the same eye to be inspected.
The alignment step of aligning the plurality of acquired images and the alignment step.
A video creation process for creating a video by connecting a plurality of aligned images in chronological order, and
A display process for displaying the created moving image on the display unit, and
An interpolation image creation step of creating an interpolated image that interpolates in time series between two images adjacent to each other in the time series order among a plurality of images joined in the time series order.
To run ,
In the interpolated image creation step, a number of the interpolated images corresponding to the shooting intervals of two adjacent images in the time series order are created.
The moving image creation step is a program for creating the moving image including the interpolated image.

Images