JP2002058647A - Transforming and compositing method of eyeground image, and recording medium with the program recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method of compositing a seamless eyeground image with little sense of incompatibility for grasping, etc., of an affected part from plural eyeground images of identical subject, which enables to grasp the vessel course of the eyeground and the chorioidea. SOLUTION: Two images of the eyeground are superposed making cross points and branch points of blood vessels, etc., as feature points and linear transformation of image is carried out (S112) only on prescribed width inside areas of images abutting on a joining border line on both sides centering the joining border line of eyeground images passing through the positions of feature points. Thus a seamless composited image is obtained from small amounts of calculation and memories. In the linear transformation, blood vessels, etc., are refracted in not higher than 40 deg. against the original image so that impression of seamlessly joining on the joining border is not lost. Because of that, a searching range to find out corresponding points within the length of a local area with prescribed width is limited to both sides centering on the joining border line (S111).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to image synthesis in which a plurality of fundus images having different photographing angles are input. More specifically, the present invention relates to a technique for obtaining a composite image in which blood vessels, choroids, and the like of the fundus oculi do not match on a junction boundary line.

[0002]

2. Description of the Related Art Conventionally, when one whole image is created from a plurality of fundus images having different photographing angles, a plurality of images obtained by photographing the same non-planar object from various angles are obtained. Since the images are synthesized, various problems occur.

For example, as in the technique described in “Digital Mosaic Processing of Remote Sensing Images”, simple geometric transformations such as affine transformation are performed so that the same points on an object to be photographed coincide with each other and connected on a plane. Methods are known.

However, the above-described method has a problem that fundus images having peripheral distortions cannot be perfectly matched. This is because the fundus has a three-dimensional curved surface structure, and in a photograph in which a part of the fundus is projected on a two-dimensional plane, distortion increases from the center to the periphery.

Further, when the fundus image is the target, the eyeball is not a perfect sphere and the refractive index of the crystalline lens through which the light rays pass is not uniform, so that a plurality of images are accurately joined to form an image. As a countermeasure against the problem that it is difficult to solve the problem, there has been proposed a method of calculating an original curved surface by detecting a plurality of locations from a plurality of image data and joining the curved surfaces with high accuracy (Japanese Patent Laid-Open No. Hei 6-243232).

However, in order to deform the entire image, it is necessary to set a large number of corresponding points on a spherical surface, so that a large amount of memory and a large amount of calculation time are required. There is a problem that it remains.
For this reason, there is a problem that images of blood vessels and the like at the junction boundary of the images are not completely connected. Furthermore, an equation that satisfies multiple corresponding points can result in a sphere different from the original surface if the number of corresponding points is small, and can satisfy the convergence condition that minimizes the error of the algorithm to be obtained if the number of corresponding points is large. Instead, they diverge and fall into an infinite loop.

A method for improving accuracy by repeating affine transformation, for example, Japanese Patent Application Laid-Open No. 7-152895 has been proposed. However, this does not solve the problem that blood vessels and the like other than the two corresponding points, that is, blood vessels that are not selected as corresponding points, do not match at the joint boundary.

Further, Japanese Patent Application Laid-Open No.
No. 00-23921. The present invention synthesizes a fundus image by matching corresponding points on the fundus image, but deforms the image at a boundary portion of the fundus image, that is, at a joint boundary (hereinafter, joint line), It is not intended to connect blood vessels seamlessly.

[0009]

Providing a seamless fundus image is important in explaining a medical condition from an ophthalmologist to a patient, in a situation of informed consent, or when explaining a medical condition between doctors. .

An object of the present invention is to grasp the most important condition of the blood vessel running of the fundus and the state of the choroid in a single image for grasping the position of a lesion site before laser treatment of the fundus.
It is an object of the present invention to provide a method for synthesizing a seamless fundus image with less discomfort for an ophthalmologist based on a plurality of images.

[0011]

In order to solve the above problems, the present invention employs the following means.

One of the means is to input a plurality of fundus images of the same subject having different gazes, determine a combining position of two of the fundus images A and B, match the combining positions, and match the two positions. A superimposition process of synthesizing two fundus images and a process of joining the fundus image A and the fundus image B with a joining line passing through the matched synthesis position (for example, if the fundus image A and the fundus image B are side by side, When the fundus image A and the fundus image B are vertically aligned, they are joined horizontally, etc.), and in the fundus image A and the fundus image B, the blood vessels passing through the respective joint lines are detected, and the joint lines are detected. A process of obtaining an intersection with a blood vessel from each of the fundus image A and the fundus image B, and obtaining a peripheral area of a fixed size centered on the intersection of the joint line and the blood vessel from the fundus image A before the joining, One intersection of image A In order to find a corresponding intersection pair of the fundus image B, all the joint lines and blood vessels extracted on the joint line of the fundus image B or at a certain distance from the intersection point of the fundus image A or limited by the number of blood vessels And the correlation with the peripheral area on the fundus image B before joining centered on the intersection point with the vicinity point thereof, and when the correlation value is equal to or greater than a certain value and is the maximum, one intersection point of the blood vessel of the fundus image A is obtained. All the other joints on the fundus image A using the intersection correlation calculation process on the joining line and the intersection correlation calculation process on the joining line performed to find an intersection pair to be the position of the corresponding fundus image B After finding a pair of the intersection line of the vascular line and the blood vessel on the fundus image B with respect to the intersection point of the line and the blood vessel, each correlation value is compared, and a corresponding blood vessel determination processing step of determining each corresponding blood vessel; Between the joint line and the blood vessel Points and, at the junction line of the midpoint between the intersection of the joining line and the blood vessel of the eye fundus image B, and for connecting the corresponding blood vessel between the fundus image A,
B, each continuous two intersections on the joining line are defined as one side, and a rectangle having a certain length from the joining line as the other side is parallel to the joining line separated by a certain length from the corresponding joining line. The length of one side is the same, and an image deformation processing step of deforming the image into a trapezoid having two consecutive midpoints on the corresponding joining line as one side, and an image deformation synthesis processing step of synthesizing the deformed images. And / or a step of storing the combined image and a step of storing the combined image.

Alternatively, in the superimposing process,
The above fundus image deforming and synthesizing method, wherein two fundus images A and B are overlapped at an arbitrary position, and an arbitrary point in an overlapping area is set as a synthesis position.

Alternatively, in the superposition process,
The above fundus image deformation synthesizing method is characterized in that processing is performed on two fundus images A and B for which an operator has manually input an instruction using a pointing device.

Alternatively, in the superposition process,
A fundus image inputting step of inputting a plurality of fundus images having different eyes of the same subject, a step of binarizing the input plurality of fundus images by image processing, and extracting a blood vessel image,
A step of thinning the extracted blood vessel image to obtain a branch point and a cross point, which are intersections of blood vessels, and a step of obtaining a fundus image of two fundus images A and B to be synthesized from the plurality of fundus images. The correlation between the peripheral area centered at one intersection of a certain blood vessel on the image A and the peripheral area centered on the intersection of all blood vessels extracted on the fundus oculi image B with respect to the peripheral area of a certain size centered on one blood vessel is calculated. An intersection correlation calculation process for finding an intersection pair in which the maximum value is a position of the fundus image B corresponding to one intersection of the blood vessels of the fundus image A, and an intersection on the fundus image B with respect to each other intersection on the fundus image A Performing the intersection correlation process to find a pair, a combination position determination processing step of setting the intersection pair having a correlation value equal to or greater than a certain threshold value among the found intersection pairs as a combination position of the two fundus images, The synthesis position Is the above fundus image transformation synthesis method characterized by having a process of synthesizing the two fundus images by matching the position of the intersection point pairs determined by constant process.

Alternatively, a program for causing a computer to execute the steps in the above fundus image deformation synthesizing method is provided by:
A fundus image deformation synthesizing program recorded on the computer-readable medium.

One whole image synthesized from fundus images obtained from various angles is obtained as a seamless image as if it were taken as a single image from the beginning. After examining the main factors that give an impression, it is enormous that the images of blood vessels or choroids are synthesized without inconsistency near the joint line of the images, rather than performing the deformation operation on each captured image evenly. I found that it was affecting me.

Accordingly, the present invention provides a method in which an intersection or a branch point of a blood vessel or the like is used as a feature point, and a fixed width is provided on both sides of the fundus image passing through the feature point with a fixed width on each side. By performing linear transformation of an image only on the inner region of each image having a width, that is, on a local region, a seamless composite image can be obtained with a small amount of computation and a small amount of memory.

In the above-mentioned linear transformation, if the blood vessels and the like of the input image are greatly deformed so as to exceed 45 degrees, the impression that they are seamlessly joined at the boundary is almost lost. In the conversion, it is preferable that blood vessels and the like have a refraction of not more than 40 degrees with respect to the original image.

For this purpose, the present invention employs a means for limiting the scanning range when finding a corresponding blood vessel in image conversion. Specifically, a search range for finding a blood vessel corresponding to a local area having a certain width or less on both sides of a blood vessel on the joint boundary line on the joint boundary line is limited. That is, in order to suppress excessive refraction of an image of a blood vessel or the like, it is necessary to make the width of local regions on both sides around the joint boundary line a certain length or more.

In the present invention, when three or more images are overlapped, deformation is performed at a joining line joining the two overlapped images. The corresponding two cards may be determined in any manner. For example, processing is performed using the two highest joining lines in accordance with the priority in display. Alternatively, a method in which the deformation process is performed on the two images at the joining line first, and then the deformation process is sequentially performed at the joining line with the next image, and so on may be repeated.

The joining line joining the fundus images does not need to be a straight line, but may be a curved line.

It is assumed that the input fundus image has an area overlapping at least one or more fundus images. Therefore, when a fundus image having no overlap is input, the fundus image can be processed by the present invention, but since no overlapping place is found, no combination is performed.

[0024]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIGS. 1, 2 and 3 show a flowchart of an embodiment of a fundus image deformation / synthesis method of the present invention, and FIG. 4 shows an example of a configuration for implementing the fundus image deformation / synthesis method of the present invention.

In FIG. 1, S101 is a start, S1
02 is a process of inputting a plurality of fundus images, S103 is a process of binarizing the fundus image and extracting a blood vessel image, S104 is a process of thinning the blood vessel image and finding intersections (branch points and intersection points) of blood vessels, S105. Is the blood vessel 1 on the fundus image A
A process of finding a correlation with a peripheral area of a certain size centered on the intersection and all the intersections on the fundus image B to be superimposed or a point near the intersection and finding an intersection pair having the maximum correlation value, S106 is the fundus image S105 is processed for all the intersections of the blood vessels on A, and the intersection pair having the maximum and a certain threshold or more among the intersection pairs is set to two fundus images (A,
B) The process of setting as a synthesis position, S107 is a process of synthesizing two fundus images (A, B) by matching the position of the intersection pair determined in S106, and S108 is all the images processed? Is shown. In the case of No in S108, S10 is performed until all images are processed.
5 is repeated, and in the case of Yes, the following S
The processing shifts to processing from 109.

In step S109, the image is cut at a joining line passing through the intersection pair determined in step S107, and the images are joined.
110 is a process of obtaining the intersection of the blood vessel passing through the joint line (the blood vessel thinned in S104) and the joint line from the two fundus images (A, B), and S111 is a fundus image A obtained in S110.
A process of obtaining a correlation with respect to a peripheral region of a certain size centered on the intersection of the image and B, and considering an intersection pair having a maximum correlation value that is equal to or greater than a certain value as a corresponding intersection. In order to connect the corresponding blood vessel at the midpoint on the joining line between the two intersections obtained in S111 of the fundus images A and B on the joining line of the blood vessels, each two consecutive intersection points of the joining line in the fundus images A and B respectively Is a process of transforming a rectangle having one side as a side and a certain length from the joining line as another side into a trapezoid having two corresponding midpoints as one side, S
Reference numeral 113 denotes a process of joining the deformed fundus images on a joining line, S114 denotes a process of displaying a synthesized image, S115 denotes a process of storing a synthesized image, and S116 denotes an end.

FIG. 2 is a flowchart showing the details of the process of S111.

In FIG. 2, S111-1 is started,
S111-2 is performed on the fundus images A and B in S110.
For obtaining a peripheral area (template) of a fixed size M centered on the intersection obtained in step S111-2.
Processing for calculating a correlation between a certain template TAi and a template TBj on the fundus oculi image B, and extracting the one having a maximum correlation value equal to or higher than a certain threshold value as a corresponding blood vessel intersection (however, a certain number or more) , Objects separated by a certain distance (refraction of 45 degrees or less) are excluded from search targets),
Has S111-3 processed all templates? Is shown.

In the case of No at S111-3,
The processing of S111-3 is repeated until all the templates are processed, and in the case of Yes, S111-4 is ended and the processing of S111 is ended.

FIG. 3 is a flowchart showing the details of the process of S112.

In FIG. 3, S112-1 is started,
S112-2 is valid for each of the two images A and B
Labels P and Q for the corresponding blood vessel intersections,₀, P₁,…,
P_n, P_{n + 1}: Sequence of blood vessel intersection points in image A, Q₀, Q₁, ..., Q
_n, Q_{n + 1}: A sequence of blood vessel intersection points in image B (where n is the n-th
Processing corresponding to the corresponding blood vessel intersection), S1
12-3 is the middle point with each blood vessel intersection as R,
R₀, R₁, ..., R_n, R_{n + 1}Processing, S112-4 is
Two adjacent points P from the blood vessel intersection point sequence of image A_n, P_{n + 1}In order
2 points P_n, P_{n + 1}Height with line segment connecting
w, vertex A_{n + 1}Isosceles triangle (P_n, P_{n + 1}, A_{n + 1})
And define two adjacent midpoints (R_n, R_{n + 1})
Height w, vertex A with line segment as base_{n + 1}Isosceles triangle of
(R_n, R_{n +1}, A_{n + 1}), S112-5
Is a triangle (P_n, P_{n + 1}, A_{n + 1}) To the triangle (R_n,
R_{n + 1}, A_{n + 1}), A process for obtaining a transformation equation to be transformed into
Has 12-6 changed n and found all the conversion equations? of
The determination process will be described.

In the case of No at S112-6,
The process of S112-6 is repeated until all the conversion formulas are obtained, and when the result is Yes, the process proceeds to S112-7. S11
2-7 is a process for sequentially executing a process of transforming a rectangle into a trapezoid in accordance with the obtained conversion formula.
Have you performed processing to transform all target areas from rectangles to trapezoids for each of the images A and B? Is shown.

In the case of No at S112-8,
The process of S112-8 is repeated until all the target areas have been processed. If the determination is Yes, the process ends in S112-9 and ends the process in S112.

In FIG. 4, reference numeral 201 denotes a plurality of input images (fundus images); 202, an input unit; 203, a control unit;
4 is a feature extraction unit, 205 is a correlation processing unit, 206 is an image synthesis unit, 207 is an image joining unit, 208 is an image transformation unit, 20
Reference numeral 9 denotes an image storage unit, 210 denotes a data storage unit, and 211 denotes a display unit.

The input unit 202 receives a plurality of fundus images of the same subject having different gazes from the outside, and outputs a feature point extracting unit 204.
Extracts feature points from the plurality of fundus images,
Reference numeral 05 indicates a correlation between peripheral regions of the extracted feature points, an image synthesizing unit 206 synthesizes an image by matching feature points of an area having a high correlation, and an image joining unit 207 generates an image of the fundus image to be synthesized. The images are joined at the joining line, and the corresponding blood vessel extracting unit included in, for example, the feature extracting unit 204 extracts a corresponding blood vessel on the joining line, and the image deforming unit 208 performs a corresponding operation to match the corresponding blood vessel on the joining line. The area around the blood vessel is deformed, the display unit 211 displays the synthesized image after the deformation, the image storage unit 209 stores the synthesized image, and the control unit 2
03 controls the above-mentioned units. The present invention is a fundus image deformation / synthesis method using an image processing means having at least these components. In addition, the corresponding blood vessel peripheral area is the joint line L
The corresponding two intersections (P _i , P _{i + 1} ) of each blood vessel are defined as one side,
It indicates a rectangular area having a fixed distance w as another side from the joining line L.

FIGS. 5 and 6 (partially enlarged views of FIG. 5) show an example of a method for synthesizing a fundus image according to the present invention, and FIG. 7 shows an example of the state of the corresponding blood vessel at the joint line.

5 and 6, reference numeral 301 denotes a fundus image A, 302 denotes a fundus image B, 303 denotes an intersection image A, 304.
Is an intersection image B, 305 is a branch point, 306 is an intersection point, 30
7 is a composite image, 308 is a peripheral area T _KAi , 309 is an intersection KA _i , 310 is a peripheral area T _KBj , 311 is an intersection KB _j ,
_{Reference numeral} 312 denotes a neighboring region D _KBj , 313 denotes a joining line L, 314 denotes a blood vessel, M denotes the size of the peripheral region T _KAi (or the peripheral region T _KBj ), and N _denotes the size of the neighboring region D _KBj .
It should be noted that the vicinity region of the intersection of the blood vessels or simply the vicinity region D
_KBj 313 is the intersection (branch or intersection) of blood vessels
(P _i , Q _i, etc.) refers to a rectangular area of size N × N. Here, N is set smaller than M as shown in FIG.

In FIG. 7, reference numeral 401 denotes a composite image;
402 is image A, 403 is image B, T_P1Is in image A
Heart P₁, A peripheral area of size M × M, T_Q1Is in image B
Heart Q₁, A peripheral area of size M × M, P₁, P_Two, P_Three,
P_Four, P_FiveRepresents the joint line L and each thinned blood vessel on the image A.
Each intersection of₁, Q_Two, Q_Three, Q_Four, Q_FiveIs image B
Each intersection of the joining line L and each thinned blood vessel above
And T_P5Is the center P on the image A _Five, Peripheral area of size M × M
Area, T_Q5Is the center Q on the image B_Five, Peripheral area of size M × M
Each area is represented.

FIG. 8 shows an example of a composite image, and FIG. 9 shows an example of a fundus image modification / combination process.

In FIG. 8, A₁, A_Two, A_Three, A_Four, A_Five
Represents two successive intersections of each blood vessel on the joint line L on the image A.
Point (P_iAnd P_{i + 1}) Is defined as one side and a fixed distance w from the joining line L
Another side (S_iAnd S_{i + 1}) And the joining line L and the opposite side in the rectangular area
Indicates the position of the midpoint of₁, B_Two, B_Three, B_Four, B
_FiveRepresents two consecutive blood vessels on the joining line L on the image B
Intersection (Q_iAnd Q_{i + 1}) As one side and a fixed distance w from the joining line L
To another side (U_iAnd U_{i + 1}) And the joint line L
Indicates the position of the midpoint of the side,₁, P_Two, P _Three, P_Four,
P_FiveRepresents the connection between the joining line L and each thinned blood vessel on the image A.
Each intersection is Q₁, Q_Two, Q_Three, Q_Four, Q_FiveIs on image B
The intersection of each of the joining line L and each thinned blood vessel is represented by R
₁, R_Two, R_Three, R_Four, R_FiveOn the joining line L with the images A and B
Each midpoint (R₁Is P₁And Q₁The middle point,
R_TwoIs P_TwoAnd Q_TwoThe midpoint of R_ThreeIs P_ThreeAnd Q_ThreeThe midpoint of R_FourIs P_Four
And Q_FourThe midpoint of R_FiveIs P_FiveAnd Q_FiveMidpoint). However
Then P₀, P₆, Q₀, Q₆Is the joining of images A and B respectively
Indicates both end points where the line and the rectangular area intersect. Note that in the example of FIG.
In the P_Three, Q_Three, R_ThreeAre the same points because of the corresponding points
You.

In FIG. 9, reference numeral 601 denotes a processed image, and 602 denotes a deformed composite image.

First, as a process S102 of inputting a plurality of fundus images of the same subject having different eyes, an input image 201 photographed by using a digital fundus camera is input through the input unit 202, and the image storage unit 20 is controlled through the control unit 203.
7. Store in. Further, the control unit 203 grasps the positional relationship (up, down, left, right, etc.) of each fundus image, and stores the data in the data storage unit 208.
To accumulate. At this time, each fundus image is photographed so as to overlap with at least one fundus image. Next, the control unit 20
Step 3 causes the feature extracting unit 204 to execute a process S103 for binarizing each fundus image and extracting a blood vessel image and a process S104 for thinning the blood vessel image and obtaining intersections (branch points and intersection points) of blood vessels. .

Here, as an example, an example of processing for synthesizing a fundus image A301 and a fundus image B302 which are two pairs as shown in FIG. 5 will be described. When the branch point 305 and the intersection point 306 obtained as a result of executing the processing up to the processing S104 for obtaining the branch point and the intersection point of the blood vessel are plotted in the fundus image,
It is represented as an intersection image A303 and an intersection image B304. Then, based on the positional relationship of the fundus image, a series of processes (S105 to S107) described below
A set of the fundus images (the fundus image A301 in the example of FIG. 5)
And the fundus image B302). At this time, depending on the positional relationship, the fundus image at a certain position limits the position of the pair of fundus images whose overlapping position is to be checked (for example, although the fundus image A301 checks the overlapping position with the fundus image B302, It is also possible to check the position where the image overlaps with the upper and lower fundus images. Such limitation of the position is performed in consideration of the case where the image is not deformed and the priority of display. Further, when the position is limited, there is an effect that the calculation time is reduced.

The control unit 203 instructs the correlation processing unit 205 to make one intersection of the blood vessel on the fundus image A and all intersections or neighboring points on the fundus image B to be superimposed a fixed-size periphery. A series of processes from a process of finding an intersection pair having a maximum correlation value to finding a correlation with respect to an area, to a process of synthesizing two fundus images (A, B) by matching the positions of the intersection pairs determined from S105. To run. The point near the blood vessel intersection is the blood vessel intersection (branch point or cross point).
A plurality of points (N ² ) within the range of a rectangular area (neighboring area D _KBj ) of size N × N centered on (P _i , Q _i, etc.)
Refers to a certain point. In addition, peripheral regions (T _KAi , T _KBj , T _Pi ,
T _Qj ) refers to a rectangular area of size M × M centered on the intersection of blood vessels or a point near the intersection. This size M × M
Is movable in a range in which the center point of this peripheral region is a point near a blood vessel or an intersection.

The outline of the processing is as follows. The basic idea is that, based on the intersection (branch point or intersection) of all blood vessels in the fundus image A301, the fundus image B302 expected to be at or near the intersection of blood vessels corresponding to each intersection point That is, a corresponding point is searched. It is _assumed that a point in the vicinity of the blood vessel intersection is a point in the vicinity area D _KBj . In addition, the corresponding point is, in two overlapping fundus images, a point on one fundus image and a corresponding point on the other fundus image, that is, the same point in the fundus image,
For example, it indicates the same intersection. One intersection point KA among the intersection points (branch points or intersection points) of all blood vessels in the fundus image A301
and _i, to determine a similarity degree between neighborhood points of all intersections of blood vessels (branch points or intersections) KB _j or intersection in the fundus image B302, peripheral centering an intersection of blood vessels of the fundus image A301 calculating a correlation coefficient r of the peripheral region T _KBj around the neighborhood points of intersection or intersection of all blood vessels in the region T _KAI and the fundus image B302. Find the peripheral region T _KBj around the intersection or intersection point near on the peripheral region T _KAI highest correlation fundus image B302 centered an intersection of blood vessels of the fundus image A301, the fundus image A301 of the time One intersection of blood vessels and fundus image B30
2, the intersection (or a point near the intersection) is defined as a synthesis point, and the two fundus images 301 and 3
02 is synthesized. Thereafter, the control unit 203 sets the image joining unit 2
At 07, an instruction is given to cut the image at the joining line passing through the intersection pair determined at S107 and to perform the process S109 of joining the images. At this time, data of the images 301 and 302 before cutting are also held.

Next, when the image is cut and joined at the joint line L passing through the synthesis point, a process of deforming the joint line L so that the corresponding blood vessels can be smoothly connected to each other is performed by the image deforming unit. This is performed at 208. In this case, the combination point may be automatically obtained by the processing up to S108 as described above, or may be a combination point arbitrarily designated by a person.

First, from the control unit 203 to the feature extraction unit 204
And the blood vessels passing on the joint line (the thinned
Two images of the fundus (A, B)
Is performed at step S110. Then, in S110
Of fixed size around the intersection of the fundus images A and B
Calculate the correlation for the side area, and the maximum correlation value above a certain value
Processing S111 in which an intersection pair having
To find the corresponding set of vessels. As shown in FIG.
At the intersection P of the blood vessel of the image A on the joining line L₁And image B
Intersection Q of blood vessels₁Find out if corresponds. Intersection P₁Around
Rectangular area T _P1Is taken on image A402 and its rectangle
Area T_P1And the intersection Q₁Rectangular area T centered at_Q1
Is taken on the image B403, and the rectangular area T_Q1Correlation value with
Find out. Similarly, the rectangular area T_P1And rectangular area T_Qi(I =
2,3,4,5). Compare each correlation value,
The one with the maximum and a correlation value equal to or greater than a certain value
Point pairs.

At this time, an intersection where a correlation value equal to or more than a certain value is not obtained is deformed and excluded from the connected blood vessels.
That is, only the blood vessels that clearly correspond are deformed, and the blood vessels for which no correspondence is found are not processed. This policy has an effect that large deformation can be performed without erroneous deformation.

Next, the control unit 203 sends the image transformation unit 208
To join the blood vessel of the fundus image A and the blood vessel of the fundus image B
On the line, the two obtained in S111 of the fundus images A and B
To connect the corresponding vessels at the midpoint on the junction line of the intersection of
In each of the fundus images A and B, each of the joining lines is continuous
Two intersections are defined as one side, and a certain length from the joining line is defined as another side.
, And the corresponding two consecutive midpoints as one side
Step S112 for transforming into a trapezoid is performed. In FIG. 8, P
_i, Q_i(I = 2, 3, 4, 5) are the corresponding blood vessels.
Also, P₀And Q₀, P₆And Q₆It is assumed that it also corresponds. Fig. 8
If P₁And Q₁Midpoint R of₁To P₁And Q₁Transform to overlap
I do. That is, image A is triangle P₀A ₁P₁The triangle
P₀A₁R₁The image B is transformed into a triangle so that
Q₀B₁Q₁To the triangle Q₀B₁R₁Deform so that it becomes. Less than
Similarly, in image A, triangle P_iA_{i + 1}P_{i + 1}To the triangle P_i
A_{i + 1}R_iIn image B, triangle Q_iB_{i + 1}Q_{i + 1}To the triangle Q_i
B_{i + 1}R_i, Respectively.

The control unit 203 controls the image joining unit 20
7, and executes processing S113 for joining the deformed fundus images together on the joining line.

With the above, the transformation synthesis process is completed, and the control unit 20
3, the synthesized image is stored in the image storage unit 207, and the synthesized data such as the position coordinates of the synthesis point of each image, the image n priority information, and the synthesis method is stored in the data storage unit 208. At this time, when combining with a plurality of images is possible due to the positional relationship, the priority of the position is determined, and the joining of the images on the joining line at the joining line for the images at the front and the overlapping portion of the image according to the present invention is performed. By using a synthesis method, smooth blood vessels and peripheral regions are joined. The reason why the deformation is required is that the degree of distortion is different between the vicinity of the center and the vicinity of the periphery of the fundus photograph by mapping the fundus having a spherical three-dimensional structure on a two-dimensional plane.

A process S for displaying a composite image for displaying the finally composited image on an output medium such as a display or a printer.
Step 116 is performed.

Referring to FIG. 10, a description will be given of a conversion formula used in the transformation processing performed in the present invention. The pixel value at each coordinate after the conversion is obtained by obtaining an inverse mapping of the conversion equation. Since the coordinate values after the conversion are quantized to integers, an inverse mapping is used to calculate a pixel value at the coordinate position after the conversion.

Three sample points in the figure before conversion
｛(X₁, Y₁), (X_Two, Y_Two), (X _Three, Y_Three)｝ G
写 (x₁', Y₁'), (X_Two', Y_Two'),
(X_Three', Y_Three')｝, The inverse map G of g is a parameter
It is expressed as follows using a to f.

[0056]

(Equation 1)

L = (x ₁ ′ −x ₂ ′) (y ₂ ′ −y ₃ ′) − (x
If _{2 '-x 3') (y} 1 '-y 2'), the parameter a~f
Is as follows.

A = {(x ₁ −x ₂ ) (y ₂ ′ −y ₃ ′) − (x
₂ −x ₃ ) (y ₁ ′ −y ₂ ′) / Lb = {(x ₂ −x ₃ ) (x ₁ ′ −x ₂ ′) − (x ₁ −x ₂ ) (x
_{2 '-x 3')} /} L c = {x 1 (x 2 'y 3' -x 3 'y 2') + x 2 (x 3 'y 1' -x
_{1 'y 3') + x} 3 (x 1 'y 2' -x 2 'y 1')} / L d = {(y 1 -y 2) (y 2 '-y 3') - (y 2 - y ₃₎ (y
_{1 '-y 2')} /} L e = {(y 2 -y 3) (x 1 '-x 2') - (y 1 -y 2) (x
_{2 '-x 3')} /} L f = {y 1 (x 2 'y 3' -x 3 'y 2') + y 2 (x 3 'y 1' -x
_{1 'y 3') + y} 3 (x 1 'y 2' -x 2 'y 1')} / L points obtained by the inverse mapping G (x _1, y ₁₎ is a rational number, not an integer. That is, since it represents the position of the sub-pixel, it is an integer coordinate position 4 surrounding this point.
A value obtained by interpolating the luminance values of the four points as the internally dividing point of the coordinate values of the points is set as the luminance value of the converted coordinate point (x ₁ ′, y ₁ ′).

Next, the deformation processing for seamlessly joining the boundary portions between images at the joining line L will be described in detail. First, in determining the cutting position (joining line L), a straight line passing through a corresponding point (or in the vicinity thereof) is set as a cutting plane for two images. The direction of the joining line L is determined to be either the vertical direction or the horizontal direction as shown in FIG. 11 depending on the positional relationship between the two fundus images. Deformation processing of a boundary portion between images is performed in the following procedure.

After determining the joint line L of the image, a deformation process is performed on the peripheral region of the joint line L so that the corresponding blood vessels are connected to each other.

First, information on the intersection of blood vessels on the joint line is extracted. A scan is performed on the cut plane (straight line) for two target images (images A and B) to detect an intersection point sequence (information on the presence or absence of a blood vessel on the joint line L). The intersection point sequence is detected by scanning from the corresponding point of each image to both ends of the straight line. Further, an index is assigned in the order in which the intersections at both ends are detected. Prepare a template of a mask size M with the detected individual intersections as the middle points.

Next, a corresponding blood vessel is determined. -For each template on one image, calculate the correlation value with the template of the other image, and extract the one with the highest correlation as the corresponding blood vessel. If the correlation value is equal to or less than a certain value, it is excluded from the corresponding blood vessel candidates. -In selecting a template, the correlation information is calculated using the index information of both images in a range where the difference between the index values is equal to or less than a predefined value, and the difference between the index values is large (at a long distance). Is not calculated. Further, scanning is not performed in a range longer than the length w in FIG. This is because when connected to a blood vessel separated by w or more, the refraction angle of the blood vessel becomes 45 degrees or more, and it is considered that the blood vessel is not an actual blood vessel, and the connected blood vessel cannot be smoothly connected. In the present invention, w is a length that is m times the average value of the blood vessel thickness on the fundus image.

The transformation process (see FIG. 8) is performed as follows.

Labels P and Q are given to the corresponding corresponding blood vessel intersections effective for each of the two images A and B as follows.

P: (P ₀ , P ₁ ,..., P _n , P _{n + 1} ,...)
Sequence of blood vessel intersection points in image A Q: (Q ₀ , Q ₁ ,..., Q _n , Q _{n + 1} ,...) Sequence of blood vessel intersection points in image B where n is the nth corresponding blood vessel intersection point.

Further, the middle point between the edge points P _n and Q _n is R _n, and the point sequence of the middle points is R: (R ₀ , R ₁ ,..., R _n , R _{n + 1} ,...).

Two adjacent points from the blood vessel intersection point sequence of image A
(P_n, P_{n + 1}) In order, and 2 points (P_n, P_{n + 1})
Height w, vertex A with base line segment as base_{n + 1}Isosceles triangle of
(P_n, P_{n + 1}, A_{n + 1}). Further adjacent 2
Point (R_n, R_{n + 1}), Height w with the line segment connecting
A_{n + 1}Isosceles triangle (R_n, R_{n + 1}, A_{n + 1})
You. The conversion process is a triangle (P_n, P_{n + 1}, A_{n + 1}) The triangle
(R_n, R_{n + 1}, A_{n + 1}), Where n is
In accordance with the conversion equation described above, the rectangle (P_n, P_n
+1, S_n, S_{n + 1}) To a trapezoid (R_n, R_{n + 1}, S_n, S_{n + 1})
The deforming processes are sequentially executed. Note that the converted individual loci
The pixel value at the target is obtained by calculating the inverse mapping of the conversion formula.
The corresponding coordinates before the transformation
Plot the values. However, as mentioned earlier,
The coordinates of the sub-pixel are
Interpolate and determine the coordinate values of the four surrounding points according to the ratio of the internally dividing points
You.

The above conversion processing is performed for each of the two images A and B.

Further, when the luminance difference between the image A and the image B is large (generally large) on the joint line, the value of 1/2 of the luminance difference on the joint line is attenuated in inverse proportion to the distance. While acting on each pixel in the partial area, the luminance difference absorption processing is performed.

FIGS. 12 and 13 show two examples of fundus images embodied by the fundus image deformation synthesizing method of the present invention described above.

It is needless to say that the steps of the processing shown in the flowcharts of FIGS. 1, 2 and 3 can be executed by a computer, and a program for causing the computer to execute the steps of the processing can be read by the computer. It is possible to record, provide, and distribute on various storage media, for example, FD (floppy disk: registered trademark), MO, ROM, memory card, CD, DVD, removable disk, and the like.

[0072]

As described above, the present invention corrects the distortion of the two-dimensional fundus image because the corresponding blood vessels are deformed and combined so as to match each other on the joint line in the overlapping fundus images. In addition, there is an effect that synthesis with less discomfort can be realized. If a plurality of fundus images can be synthesized on a computer and the fundus image can be output to a high-quality printer or the like as a single montage image, there is an effect that it is not necessary to manually attach papers. Further, by using a computer, there is an effect that the work time required for manual work can be reduced, the burden on the manual work of the worker can be reduced, and the free time can be used for other tasks. In addition, since the electronic data on the computer is used, if the fundus images are separately stored, it becomes possible to repeat the fundus image synthesis many times. In addition, although you can only roughly overlay the fundus image by hand,
According to the present invention, there is an advantage that fine adjustment can be performed, such as shifting the position of each fundus image vertically and horizontally by one pixel with respect to the automatically synthesized image.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating an embodiment of a fundus image deformation synthesizing method according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating details of S111 in the embodiment.

FIG. 3 is a flowchart showing details of S112 of the embodiment.

FIG. 4 is a diagram showing a configuration example for implementing a fundus image deformation synthesizing method of the present invention.

FIG. 5 is a diagram illustrating a synthesis example of a fundus image deformation synthesis method according to the present invention.

FIG. 6 is a partially enlarged view of FIG. 5;

FIG. 7 is a diagram illustrating an example of a state of a corresponding blood vessel at a joint line in the fundus image deformation synthesizing method of the present invention.

FIG. 8 is a diagram illustrating an example of a combined image according to the fundus image deformation combining method of the present invention.

FIG. 9 shows an example of a fundus image deformation synthesis process in the fundus image deformation synthesis method of the present invention.

FIG. 10 is a diagram illustrating a conversion formula used for a deformation process in the fundus image deformation synthesis method of the present invention.

FIGS. 11A and 11B are diagrams illustrating a deformation process for seamlessly joining a boundary portion between images at a joining line L in the fundus image deformation synthesizing method of the present invention.

FIG. 12 is a diagram simulating an example of a fundus image embodied by the fundus image deformation synthesizing method of the present invention.

FIG. 13 is a diagram simulating another example of a fundus image embodied by the fundus image deformation synthesizing method of the present invention.

[Explanation of symbols]

S101: Start S102: Processing for inputting a plurality of fundus images S103: Processing for binarizing the fundus image and extracting a blood vessel image S104: Processing for thinning the blood vessel image to obtain intersections (branch points and intersection points) of blood vessels S105 ... A cross-section having a maximum correlation value is obtained by calculating a correlation between a single intersection point of a blood vessel on the fundus image A and all intersection points or neighboring points on the fundus image B to be superimposed and a peripheral area of a certain size centered on the intersection point. Processing for finding a pair S106: S is performed for all intersections of blood vessels on the fundus image A
105 is processed, and the maximum intersection pair among the intersection pairs and a certain threshold value or more is set as the combined position of the two fundus images (A, B). The positions of the intersection pairs determined in S107. Processing for synthesizing two fundus images (A, B) by doing this S108: Have all images been processed? S109: A process of cutting an image by a joining line passing through the intersection pair determined in S107 and joining the images S110: Identifying an intersection between a blood vessel passing on the joining line (the blood vessel thinned in S104) and the joining line Two fundus images (A, B)
S111: A process for obtaining a correlation with respect to a peripheral region of a certain size centered on the intersection of the fundus images A and B obtained in S110, and regarding a pair of intersections having a maximum correlation value equal to or greater than a certain value as a corresponding intersection S112 ... on the joint line between the blood vessel of the fundus image A and the blood vessel of the fundus image B, and connecting the corresponding blood vessels at the midpoint on the joint line of the two intersections obtained in S111 between the fundus images A and B, the fundus images A and B In each case, a process of transforming a rectangle having two continuous intersections of the joining line as one side and a certain length from the joining line as another side into a trapezoid having the corresponding two consecutive midpoints as one side. S113: A process of joining the deformed fundus images on a joining line S114: A process of displaying a combined image S115: A process of storing a combined image S116: End 201: A plurality of input images 202 ... Power unit 203 Control unit 204 Feature extraction unit 205 Correlation processing unit 206 Image synthesis unit 207 Image connection unit 208 Image deformation unit 209 Image storage unit 210 Data storage unit 211 Display unit 301 Fundus image A 302 ... fundus image B 303 ... branch / cross point image A 304 ... branch / cross point image B 305 ... branch point 306 ... cross point 307 ... composite image 308 ... peripheral area T _KAi 309 ... cross point KA _i 310 ... peripheral area T _KBj 311 ... intersection KB _j 312 ... neighboring region D _KBj 313 ... joint line L M ... size N ... near region D _KBj peripheral region T _KAI or peripheral region T _KBj size T _P1 ... center P ₁ on the image a, Peripheral area T _Q1 of size M × M: center Q ₁ on image B, peripheral areas P ₁ , P ₂ , P ₃ , P ₄ , P _{5 of} size M × M... Each of the thinned blood vessels Intersection _{Q 1, Q 2, Q 3} , Q 4, Q 5 ... center P ₅ on each intersection T _P5 ... image A and joint line L and each thinned vessels on the image B, the magnitude M × M peripheral area T _Q5 … center Q ₅ on image B, size M × M peripheral area A ₁ , A ₂ , A ₃ , A ₄ , A ₅ … continuation of each blood vessel on the joining line on image A Position of the middle point B ₁ , B ₂ , B ₃ , of the joining line and the opposite side in a rectangular area where the two intersections (P ₁ and P _{i + 1} ) to make B ₄ , B ₅ ... Joined in a rectangular area where two consecutive intersections (Q _i and Q _{i + 1} ) of each blood vessel on the joining line on the image B are defined as one side and a fixed distance w from the joining line L is defined as another side. Positions P ₁ , P ₂ , P ₃ , P ₄ , P ₅ of the midpoint of the line and the opposite side Intersections Q ₁ , Q ₂ , Q ₃ , of the joining line L and the thinned blood vessels on the image A, respectively. Q ₄ , Q ₅ ... joined on image B Each intersection of the line L and each thinned vessels _{R 1, R 2, R 3} , R 4, R 5 ... respectively, the midpoint of P ₁ and Q _1, the midpoint of P ₂ and Q _2, P midpoint of ₃ and Q _3, the midpoint of P ₄ and Q _4, the middle point P ₀ of the P ₅ and _{Q 5, P 6, Q 0} , Q 6 ... each image a, the bonding wire and the rectangular area B intersect Both end points 601: processed image 602: deformed composite image

 ────────────────────────────────────────────────── ─── Continued on the front page F term (reference) 5B057 AA07 CA08 CA12 CA16 CB08 CB12 CB16 CC03 CD01 CE09 DA07 DB02 DB09 DC07 5C076 AA12 AA19 AA23 BA06 CA02

Claims (5)

[Claims] 1. A plurality of fundus images having the same subject's line of sight different from each other are input, and a combining position is determined for two of the fundus images A and B, and the combining positions are matched with each other.
A superimposition process of synthesizing two fundus images; a process of joining the fundus image A and the fundus image B with a joining line passing through the matched combining position; Detecting a blood vessel passing through the blood vessel, and obtaining an intersection between the joint line and the blood vessel from each of the fundus image A and the fundus image B. From the fundus image A before joining, in order to find an intersection pair of the fundus image B corresponding to one intersection of the fundus image A, all of the points extracted on the joining line of the fundus image B,
Alternatively, a correlation between a peripheral area on the fundus image B before joining centered on an intersection of the blood vessel and the blood vessel at a certain distance from the intersection point of the fundus image A or the junction line limited by the number of blood vessels and a point near the blood vessel is calculated. When the correlation value is equal to or more than a certain value and is the maximum, an intersection correlation calculation processing step on a joining line performed to find an intersection pair to be a position of the fundus image B corresponding to one intersection of the blood vessel of the fundus image A, Using the intersection correlation calculation process on the joint line, after finding the intersection pair of the joint line and the blood vessel on the fundus image B to the intersection of all other joint lines and blood vessels on the fundus image A, A corresponding blood vessel determining process for comparing each correlation value and determining each corresponding blood vessel; an intersection of the blood vessel with the joint line of the fundus image A;
At the midpoint of the junction line with the intersection of the blood vessel and the junction line,
In order to connect the corresponding blood vessels, in each of the fundus images A and B, each continuous two intersections on the joining line are defined as one side, and a rectangle having a certain length from the joining line as the other side is defined as a corresponding rectangle. The length of one side parallel to the joining line separated by a certain length from the joining line is the same, and an image deformation processing step of transforming into a trapezoid having two consecutive midpoints on the corresponding joining line as one side, A fundus image transformation comprising: an image transformation / combination process of combining the transformed images; and / or a process of displaying the combined image and / or accumulating the combined image. Synthesis method. 2. In the superimposing process, two fundus images A and B are superimposed at an arbitrary position, and an arbitrary point in an overlapping area is set as a synthesis position. 3. A fundus image deformation synthesis method according to claim 1. 3. The method according to claim 1, wherein in the superimposing process, two fundus images A and B for which an operator has manually input an instruction using a pointing device. Fundus image deformation synthesis method. 4. In the superimposition processing step, a fundus image inputting step of inputting a plurality of fundus images of the same subject having different gazes, a binarization of the input plurality of fundus images by image processing, and a blood vessel image Extracting a blood vessel image, thinning the extracted blood vessel image to obtain a branch point and an intersection point which are intersections of blood vessels, and two fundus images A to be synthesized among the plurality of fundus images. , B, the correlation between the intersection of all the blood vessels extracted on the fundus image B and the surrounding area centered on the vicinity of the intersection of all the blood vessels extracted on the fundus image B with respect to the surrounding area of a certain size centered on one intersection of a certain blood vessel on the fundus image A And an intersection correlation calculation process of finding an intersection pair in which the maximum value of the correlation is the position of the fundus image B corresponding to one intersection of the blood vessels of the fundus image A, and for each other intersection on the fundus image A eye The intersection correlation processing for finding the intersection pair on the image B is performed, and the intersection position having the largest correlation value among the found intersection pairs and a correlation value equal to or more than a certain threshold value is set as the synthesis position of the two fundus images. 4. The method according to claim 1, further comprising: a determination process; and a process of combining the two fundus images by matching positions of the intersection pairs determined in the synthesis position determination process. 5. Fundus image deformation synthesis method. 5. A fundus image deformation synthesizing program, wherein a program for causing a computer to execute the steps in the fundus image deformation synthesizing method according to claim 1 is recorded on a computer-readable medium. Recording medium on which is recorded.