JP6750842B2 - Image generating apparatus, image generating method, and program - Google Patents

Description

The present invention relates to an image generation device that generates a panoramic image by combining a plurality of fundus images, an image generation method by the image generation device, and a program for causing a computer to function as each unit in the image generation device. is there.

BACKGROUND OF THE INVENTION Examination of the fundus of the eye to be inspected is widely performed for the purpose of early diagnosis of lifestyle-related diseases and diseases that are the leading causes of blindness. 2. Description of the Related Art As an ophthalmologic apparatus utilizing the principle of a confocal laser microscope, there is a scanning laser ophthalmoscope (SLO: Scanning Laser Ophthalmoscope). This SLO device is a device that performs a raster scan on a fundus of an eye to be inspected with a laser that is a measurement light, and obtains a fundus image that is a planar image with high resolution and high speed from the intensity of the returned light.

In recent years, an adaptive optics SLO (AO-SLO: Adaptive Optics) having an adaptive optics system in which the aberration of the eye to be inspected is measured in real time by a wavefront sensor and the aberration of the measurement light or its return light generated in the eye to be inspected is corrected by a wavefront correction device. SLO) devices have been developed. This adaptive optics SLO device enables acquisition of a high-resolution planar image.

When acquiring a fundus image which is a high-resolution planar image with the SLO device, there is a problem of optical aberration of the device itself and also a problem of time to image one frame, and an imaging range to be imaged at one time, that is, a view angle is narrow. turn into. On the other hand, in order to obtain information necessary for diagnosis, examination, etc., a fundus image in a wide range to some extent is required, and thus it is necessary to perform imaging at a plurality of positions by the SLO device. Further, even when a local region of the eye to be inspected is imaged, a wide range of the eye to be inspected may be separately imaged with a low resolution in order to grasp the whole. However, even if an area required for diagnosis or inspection is photographed, it is necessary to grasp the relationship between the obtained plural photographed images. In other words, it is necessary to clarify the positional relationship between the plurality of captured images and to bond the images according to the positions, thereby generating one high-definition panoramic image.

In order to solve such a problem, Patent Document 1 proposes a compound device in which an SLO device having a wide angle of view and an SLO device having a small angle of view but high resolution are combined. In addition, Patent Document 2 proposes a method of accurately recognizing the positional relationship between the fundus images and generating a panoramic image even when the fundus images having different angles of view and resolutions are combined.

JP, 2010-259543, A JP, 2014-128366, A

R. Szeliski, Image alignment and stitching: a tutorial, Tech. Rep. MSR-TR-2004-92, Microsoft Research, December 2004.

As various methods are introduced in Non-Patent Document 1, it is a general method to align a plurality of images and generate a panoramic image. However, for example, when features such as a high-definition image with a small angle of view of the fundus are very small structures such as photoreceptor cells and capillaries, it may occur that they cannot be resolved into a wide angle image. .. In the method of Patent Document 1, when a small angle-of-view image and a wide angle-of-view image with few features are combined, a case where no matching may occur may occur. Further, although the method of Patent Document 2 can deal with the case as described above, there may be few features even between small angle-of-view images depending on the imaged site and the image quality of the obtained image, and there is not a little matching. Cases that do not remain.

The generation of the panoramic image can be significantly reduced by automating, for example, the processing of a computer program. However, if the above-mentioned unmatched images remain, the alignment of these images is performed manually, which puts a burden on the user. In addition, as described in Non-Patent Document 1, in addition to alignment, panorama processing includes various processes such as image brightness adjustment and distortion correction, image selection in a superimposing unit of images, and joining processing. The process is executed. Also regarding these processes, if the above-mentioned unmatched images remain, for example, the processes of the computer program may not be completed correctly, and manual correction may occur. Further, it is not possible to obtain such accuracy (rigidity) by hand for a computer program that performs alignment and various correction processes by calculation.

The present invention has been made in view of such a problem, and when a panoramic image obtained by pasting a plurality of fundus images is corrected, the work load of the user who performs the correction is reduced and the correction work is performed. The purpose is to provide a mechanism for improving accuracy.

The image generation device of the present invention is an image generation device that generates a panoramic image by pasting together a plurality of fundus images, and at least one change target of the fundus image forming the panoramic image or the fundus image that can be formed. Selecting means for selecting the fundus image, and the fundus image and the panoramic image selected by the selecting means according to the first change content specified by the user with respect to the fundus image selected by the selecting means. A first calculation means for calculating a second change content for at least one of the two, and a generation means for generating a new panoramic image based on the second change content. The change content of No. 1 is relative position movement between the fundus image and the panoramic image selected by the selecting means, and the relative position movement destination is specified as the first change content until the relative position movement destination is designated. The fundus image selected by the selection means is displayed in a semitransparent state.
Further, the present invention includes an image generation method by the above-described image generation device, and a program for causing a computer to function as each unit in the above-described image generation device.

According to the present invention, when a panoramic image obtained by combining a plurality of fundus images is corrected, it is possible to reduce the work load on the user who performs the correction.

It is a figure which shows an example of a functional structure of the image generation apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the hardware constitutions of the image generation apparatus which concerns on the 1st Embodiment of this invention. 3 is a flowchart showing an example of a processing procedure of panoramaization processing performed in the panoramaization processing unit of FIGS. 1 and 2. FIG. 3 illustrates the first embodiment of the present invention, and is a diagram illustrating an example of a UI screen of utility software displayed on the display unit of FIGS. 1 and 2. 6 is a flowchart illustrating an example of a processing procedure of panoramic image correction processing performed in the image generation apparatus according to the first embodiment of the present invention. FIG. 5 is a diagram showing the first embodiment of the present invention and showing a display example of a parameter setting unit in an image replacement mode of utility software. FIG. 5 is a diagram showing the first embodiment of the present invention and showing a display example of a parameter setting unit in a frame changing mode of utility software. FIG. 6 is a diagram showing a first exemplary embodiment of the present invention and showing a display example of a parameter setting unit in a layer change mode of utility software. FIG. 6 is a diagram showing the first embodiment of the present invention and showing a display example of a parameter setting unit in a contrast adjustment mode of utility software. It is a figure which shows the 1st Embodiment of this invention and shows an example of the panoramic image in the fundus image imaged by the SLO apparatus. It is a figure which shows the 2nd Embodiment of this invention and shows an example of the UI screen of the utility software displayed on the display part of FIG. 1 and FIG. It is a flow chart which shows an example of the processing procedure of the panoramic image modification processing performed in the image generation device concerning a 2nd embodiment of the present invention. It is a figure which shows the 2nd Embodiment of this invention and shows an example of the single image in the middle of being dragged by the mouse. FIG. 12 is a diagram showing an example of an image displayed in a layer display area of the UI screen shown in FIG. 11 according to the second embodiment of the present invention. FIG. 12 is a diagram showing an example of an image displayed in a layer display area of the UI screen shown in FIG. 11 according to the second embodiment of the present invention.

Hereinafter, modes (embodiments) for carrying out the present invention will be described with reference to the drawings.

(First embodiment)
First, a first embodiment of the present invention will be described.

In the present embodiment, the user takes a panorama image of a panoramic image generated by combining and combining a plurality of fundus images (hereinafter, referred to as “single images”) of the retina of the eye to be inspected by the adaptive optics SLO device. A process for correcting a single image forming an image will be described. Here, in the present embodiment, the single image is an image or a moving image obtained by imaging the plane of the retina of the subject's eye, and is obtained when the subject's eye is photographed from the front with a fundus camera, etc. I will say. As the processing, the user selects a single image to be corrected and gives a rough correction instruction, and then the image generation apparatus calculates and corrects the correlation between the single image and its peripheral image based on the designated instruction. Determine the amount. As described above, in the present embodiment, in the correction work by the user, the image generation device performs the high-definition correction processing such as the correlation calculation to simplify the correction work of the user, and as a result, the user who performs the correction work. It is intended to reduce the work load of and improve the accuracy of the correction work.

<Structure of image generation device>
FIG. 1 is a diagram showing an example of a functional configuration of an image generating apparatus 100 according to the first embodiment of the present invention. The image generating apparatus 100 is an apparatus that combines a plurality of single images to generate a panoramic image. The image generating apparatus 100 has functions of a single image storage unit 110, a panorama conversion processing unit 120, a panorama information storage unit 130, a panorama image output unit 140, a display unit 150, a user interface unit 160, and a panorama information correction unit 170. It is configured to have a configuration.

The single image storage unit 110 stores a single image that is a plurality of fundus images that are plane images at various positions in the fundus of the subject's eye captured by the adaptive optics SLO device. Further, in the single image storage unit 110, together with the single image, incidental information of the single image is stored in association with the single image. Here, the incidental information is information on the left and right eyes of the eye to be inspected, angle of view information of the image, reference position information on the fundus of the eye to be inspected, image ID information for distinguishing images having the same imaging condition, and the like. is there. The reference position information is, for example, the presentation position information of the fixation target used by the SLO device at the time of photographing. The image ID information is, for example, information such as serial numbers.

The panoramaization processing unit 120 selects a single image that is a fundus image based on the reference position information, performs correlation calculation between the single images to sequentially determine the relative position, and combines the plurality of single images. To generate a panoramic image. Then, the panorama processing unit 120 stores the generated panoramic image and the panorama information including information related to the processing of the panoramic image in the panorama information storage unit 130. The panorama information includes position information of each single image on the panoramic image, and in addition to parameters such as rotation angle information, brightness correction information, and image deformation information, image processing such as brightness correction and deformation is performed according to those parameters. The applied image itself is also included.

The panorama information storage unit 130 stores the panorama information obtained by the panorama conversion processing unit 120 and the panorama information corrected by the panorama information correction unit 170.

The panorama image output unit 140 outputs the panorama image and the like included in the panorama information stored in the panorama information storage unit 130 to the display unit 150.

The display unit 150 displays the panoramic image output from the panoramic image output unit 140 and presents it to the user.

The user interface unit 160 is an interface for inputting, to the image generation apparatus 100, a user's instruction of correction content for the panoramic image or the like displayed on the display unit 150.

The panorama information correction unit 170 corrects the panorama information stored in the panorama information storage unit 130 by performing recalculation of correlation between single images based on a correction instruction for the panorama image input from the user interface unit 160. To update.
Then, when the panorama information correction unit 170 corrects the panorama information, the panorama conversion processing unit 120 generates a new panorama image based on the corrected panorama information.

FIG. 2 is a diagram showing an example of a hardware configuration of the image generating apparatus 100 according to the first embodiment of the present invention. 2, the same components as those shown in FIG. 1 are designated by the same reference numerals. The image generating apparatus 100 according to this embodiment is realized by, for example, a personal computer (PC).

The image generating apparatus 100 is configured to have each hardware configuration of a calculation unit 201, a storage unit 202, a display unit 150, and a user interface unit 160.

The arithmetic unit 201 is composed of, for example, a CPU (Central Processing Unit). The panorama processing unit 120, the panoramic image output unit 140, and the panorama information correction unit 170 are programs executed by the arithmetic unit 201, for example.

The storage unit 202 is composed of, for example, a hard disk. The single image storage unit 110 and the panorama information storage unit 130 correspond to the storage area of the storage unit 202.

The image generating apparatus 100 according to the present embodiment uses a LAN (Local Area Network) or a USB (Universal Serial Bus) that is a general-purpose interface of a computer, and uses a database server, a storage device, an adaptive optics SLO device, etc. outside the device. The single image is acquired from the single image storage unit 110 and stored in the single image storage unit 110.

The display unit 150 shown in FIG. 2 is composed of, for example, a display. The user interface section 160 shown in FIG. 2 is composed of, for example, a mouse and a keyboard.

<Processing procedure for panorama processing>
FIG. 3 is a flowchart showing an example of the procedure of panoramaization processing performed in the panoramaization processing unit 120 of FIGS. 1 and 2.

<<Step S310>>
Step S310 is a step of selecting an image used for the panoramic processing.
Specifically, in step S310, the panoramaization processing unit 120 refers to the incidental information of the single image that is the fundus image stored in the single image storage unit 110, and refers to the imaging conditions (the eye to be examined, the angle of view, and the fixation). Search for single images with the same target presentation position). Then, when such an image group is found at one position of the eye to be inspected, the panoramaization processing unit 120 selects one image or frame having the highest image quality using any known image quality evaluation index. .. Note that the panoramaization processing unit 120 selects the single image when there is no image having the same shooting conditions and only one single image. That is, in this step, the panoramaization processing unit 120 selects only one representative single image from the plurality of single images having the same shooting condition and sets one corresponding single image for each shooting condition. .. Then, the panoramaization processing unit 120 stores information such as an image ID and a frame number in the panorama information storage unit 130 as information of the single image selected for each shooting condition.

Here, in order to select the image, alignment between the images of each frame in one moving image is performed, and a frame having extremely low brightness due to a blink, a frame in which image distortion is caused by a slight movement of fixation, or an aberration correction failure is caused. A frame having a low S/N ratio may be excluded as an exceptional frame, an averaged image may be generated using the remaining images, and the generated averaged image may be used. Any known registration method can be used for the inter-frame registration and the method of generating the averaged image. If the images of the same fixation on the subject's eye with good fixation are images at relatively the same position, an averaged image may be generated using the method described above between a plurality of images at the same position. Alternatively, the user may select an image or manually generate an averaged image and use it. When the panorama processing unit 120 has generated the averaged image, the panorama information storage unit 130 also stores the generated averaged image itself together with the image information.

<<Step S320>>
Step S320 is a process of adjusting the contrast of the image used for the panorama processing.
Specifically, in step S320, the panoramaization processing unit 120 selects the image with the highest image quality from the images selected for the panoramaization process. The image quality evaluation for this selection may be a simple contrast comparison, or any known image quality evaluation index may be applied. For the selected image, the deviation of the gradation distribution on the histogram is eliminated by applying linear gradation conversion, and this is used as a reference image. Then, the panoramaization processing unit 120 calculates the average of the gradation values of all the pixels of the reference image, and the average of the gradation values of the images is equal to that of the reference image for each of the other images selected in step S310. A uniform offset is added to or subtracted from the gradation value so that After that, the panoramaization processing unit 120 applies linear gradation correction to each image in a range where there is no blown-out highlights or blackouts, while maintaining the average gradation value. Here, although the linear gradation correction is basically used, non-linear gradation conversion using a tone curve may be mixed, and any other contrast adjustment can be applied. The image to be processed here is read from the single image storage unit 110 for a single image and read from the panorama information storage unit 130 for an averaged image according to the image selection information stored in the panorama information storage unit 130. Then, all the gradation-corrected images are stored in the panorama information storage unit 130.

≪Step S330≫
Step S330 is a step of selecting a background image for panoramaization. Here, the background image is a single image or an averaged image of a plurality of frames that is initially selected in the panorama processing, and the panorama is aligned with another single image or an averaged image with respect to this background image. Then, you can start from the place of pasting.
Since the background image is preferably an image with a wide angle of view, in step S330, the panoramaization processing unit 120 selects the image with the widest angle of view as the background image among the images selected in step S310. Here, when there are a plurality of candidate images, the image closest to the average value of the reference position information of each image may be selected, or the highest image quality may be obtained by using any known image quality evaluation index. You may select an image.

<<Step S340>>
Step S340 is a rough alignment process in the panorama processing.
Specifically, in step S340, the panoramaization processing unit 120 selects the next selected position with respect to the background image selected in step S330, which is the original image for positioning, or the panoramic image that has been registered so far. Roughly align the matching partner images. The images to be aligned are selected in order of priority from images with a wide angle of view, and based on the reference position information, there are many regions that overlap the background image and the panoramic images that have been aligned so far. .. This order serves as information for determining the front side/back side of each single image on the panoramic image, and is stored in the panorama information storage unit 130.

Here, with respect to the original image for alignment, the correlation image between the images is calculated by a simple shift such as parallel, rotational movement, and enlargement/reduction of the other image by any known method. Depending on the data, rotation may be omitted if there is at least parallel movement. In the present embodiment, the reference position information stored in the single image storage unit 110 is used as the initial value of the position search, and the cross-correlation is calculated by using the pixel value of the overlapping area of the two images shifted by the parallel movement. Calculate the number of relationships (normalized correlation coefficient). Then, the position having the highest correlation is calculated using the normalized correlation coefficient, and the normalized correlation coefficient and the obtained position information for rough alignment are stored in the panorama information storage unit 130. In addition, using the same pixel value, a general pattern patching method such as SAD (sum of absolute values of difference in brightness value) or SSD (sum of squares of difference in brightness value) is used. May be. The calculation can be performed faster if the search range is limited to some extent, but the entire search may be performed. Here, when the correlation coefficient is so small that no overlapping area (overlap) can be found in the alignment of the background image or the original image that has been aligned so far with the partner image to be merged, Information that the partner image is an image that does not match is stored in the panorama information storage unit 130.

≪Step S350≫
Step S350 is a highly accurate alignment process performed by dividing the image into small areas. Specifically, in step S350, the panoramaization processing unit 120 locates each of the background images selected in step S330 or the panoramic images that have been aligned so far by dividing the alignment partner image into small images. To match. Here, the division of the image can be determined by the characteristics of the image, and in the present embodiment, it is assumed that 12×12 division is performed, but the number is not limited to this. The method of aligning each small image with respect to the background image or the panoramic image that has been aligned so far is any known method with the rough alignment position set in step S340 as the initial position. Align with. In this embodiment, similarly to step S340, the shift due to the parallel movement and the normalized cross-correlation are calculated, and the position with the highest correlation of each small image is acquired.

≪Step S360≫
Step S360 is a step of performing affine deformation of images and actual bonding.
Specifically, the panoramaization processing unit 120 selects an arbitrary position based on the position of each small image set in step S350 with respect to the background image selected in step S330 or the panoramic image that has been aligned so far. The affine coefficient of the combined image is determined by a known method. In the present embodiment, the panoramaization processing unit 120 first creates an image Si in which all small images are arranged on a plane based on the position acquired in step S350. Then, the panoramaization processing unit 120 draws a quadrangular frame line that surrounds these small images, deforms the combined image by affine transformation so as to have the same shape as this frame line, and sets this image as Ai. In the first transformation, all the small images are not transformed and are still in a divided state, so it is sufficient to draw the frame lines using the images at the four corners. Next, the panoramaization processing unit 120 calculates the shift amount between the small images of the image Ai corresponding to the same position as the small images of the image Si by using the correlation coefficient. Then, the panoramaization processing unit 120 repeats the process of changing the affine coefficient so that the sum of the shift amounts becomes the smallest and calculating the same sum of the shift amounts. Then, the panoramaization processing unit 120 acquires the affine coefficient with which the total sum of the shift amounts is the smallest and uses this as the position information. In addition, the panoramaization processing unit 120 attaches the changed image to the reference frame or the panoramic image that has been aligned so far by this affine deformation, and the alignment image and the alignment partner finally determined. The affine coefficient of the image is stored in the panorama information storage unit 130.

The stitched image obtained in step S360 is used for aligning the next image, and is updated each time. Further, the method of combining the pasting may be any known method, for example, the blending may be performed only on the edge portion, or the brightness value may be adjusted. In this embodiment, only simple deformation by an affine is dealt with, but any other known image conversion method or correction method, for example, optical distortion may be corrected, and is limited to the method described above. is not.

≪Step S370≫
In step S370, the panoramaization processing unit 120 determines whether or not all the images selected in step S310 have been aligned (stitched). As a result of this determination, if the alignment (bonding) has not been performed on all the images selected in step S310 (S370/No), the process returns to step S340, and the image to be aligned next. Is selected, and steps S340 to S360 are performed.

On the other hand, as a result of the determination in step S370, if all the images selected in step S310 have been aligned (bonded), the process proceeds to step S380.

≪Step S380≫
Step S380 is a step of re-matching the images that have not been matched in step S340.
Specifically, in step S380, the panoramaization processing unit 120 reads out the information of the unmatched images stored in the panorama information storage unit 130 in step S340, and executes the same process as steps S340 to S360. Align again. If there is an image matched in this step, the panorama processing unit 120 stores the affine coefficient and the final panorama image in the panorama information storage unit 130. In addition, the panoramaization processing unit 120 regards images that do not match even in this step as unsuitable images, and as the information thereof, only the affine value at the closest correlation position is stored in the panorama information storage unit 130 without being attached to the panoramic image. save.

<Panorama image correction procedure>
Next, with reference to the UI screen of FIG. 4 and the flowchart of FIG. 5, a panorama image correction processing procedure and a user operation in the image generating apparatus 100 according to the present embodiment will be described.

The panorama image output unit 140 reads out the panorama image generated by the panorama conversion processing shown in FIG. 3 from the panorama information storage unit 130 and displays it on the display unit 150. The image generation apparatus 100 is a utility that provides the user with a function of correcting the generated panoramic image by inputting the correction content (first modification content) specified by the user from the display unit 150 and the user interface unit 160. It is equipped with software.

FIG. 4 shows the first embodiment of the present invention, and is a diagram showing an example of a UI screen of the utility software displayed on the display unit 150 of FIGS. 1 and 2.
On the UI screen 910 shown in FIG. 4, the generated panoramic image is displayed in the layer display area 920. Further, in the present embodiment, the above-described nonconforming images 940 are displayed side by side on the left side of the application. These non-conforming images 940 may be displayed so that they can be recognized as non-conforming images by, for example, changing the color of the frame after affine-deforming and pasting them to the position having the highest correlation. In the command bar 930 of the utility software, in the example shown in FIG. 4, five commands are prepared corresponding to the types of modification that the user can specify. The modification contents of the five commands are as follows.
・Move... Change the position of single image.
-Change... Replaces a single image with another image.
Frame: Change the frame number used for the panoramic image.
Front/back... Replaces the single image displayed in the front in the area where the single images overlap.
・Contrast... Adjust the contrast.

When any one of these commands on the command bar 930 is selected, detailed parameters necessary for correction can be input from the parameter setting unit 950 according to the selected command. In order to start the correction process with the input parameters, the “Enter” button located below in the parameter setting unit 950 of FIG. 4 is pressed.

Next, a flow of correcting a panoramic image will be described with reference to the flowchart of FIG.
FIG. 5 is a flowchart showing an example of a processing procedure of panoramic image correction processing performed in the image generating apparatus 100 according to the first embodiment of the present invention.

≪Step S510≫
Step S510 is a step of receiving the type of correction for the panoramic image designated by the user.
Specifically, when the user confirms the panorama image displayed in the layer display area 920 and has a content to be modified, when the user selects a desired modification type from the command bar 930 through the user interface unit 160, In S510, for example, the panorama information correction unit 170 receives a command related to the selection.

<<Step S520>>
In step S520, for example, the panorama information correction unit 170 determines whether the single image to be corrected (change target) has already been selected by the user via the user interface unit 160. Here, the user can select the correction target image, for example, by clicking the single image 921 on the panoramic image displayed in the layer display area 920 using the mouse of the user interface unit 160. In the present embodiment, as the single image selected here, at least one of the fundus images forming the panoramic image or the fundus images that can be formed can be applied. In addition, the user interface unit 160 that selects a single image to be corrected (change target) constitutes a selection unit.

If the result of determination in step S520 is that the single image to be modified has already been selected by the user (S520/Yes), the flow proceeds to step S540.
On the other hand, if the result of determination in step S520 is that the single image to be modified has not been selected by the user (S520/No), the flow proceeds to step S530.
Note that this selection process may be performed before the step of selecting the type of correction in step S510, in which case the target image has already been selected, and the process proceeds to step S540.

<<Step S530>>
In step S530, the image generation apparatus 100 (for example, the panorama information correction unit 170) displays a message prompting the user to select the correction target image on the display unit 150, and accepts the selection of the correction target image from the user. Then, when the user selects the correction target image via the user interface unit 160, the process proceeds to step S540.

<<Step S540>>
In step S540, the image generating apparatus 100 (for example, the panorama information correction unit 170) displays the detailed correction parameters corresponding to the selected single image and the type of correction on the parameter setting unit 950, and the user interface unit 160 is used. Accept user input.

Here, the correction content by the user accepted in step S510 and step S540 constitutes the first change content.

<<Step S550>>
In step S550, the panorama information correction unit 170 performs contrast adjustment, correlation calculation, affine transformation, and the like according to the correction parameter received in step S540, performs recalculation of correlation between single images, and the panorama information storage unit 130. Modify and update the panorama information stored in. Specifically, in this step, the panorama information modification unit 170 modifies at least one of the selected modification target image and panorama image according to the modification content (first modification content) by the user. The content (second modification content) is calculated. The panorama information correction unit 170 that calculates the second change content by calculation constitutes a first calculation unit.

After that, the panoramaization processing unit 120 generates a new panoramic image based on the corrected panoramic information, and stores this in the panoramic information storage unit 130. The panorama conversion processing unit 120 that generates this new panoramic image constitutes a generation unit.
After that, the panorama image output unit 140 reads out the panorama image newly generated by the panoramaization processing unit 120 from the panorama information storage unit 130 and outputs the panorama image to the display unit 150 for display.

Then, the image generating apparatus 100 repeats the processing of steps S510 to S540 for each item that the user determines to be corrected.

Next, these processing procedures will be described in detail for each type of modification.

[Move; move selected image]
First, the processing when the type of modification is “move” will be described. Specifically, when the type of correction is "move", the process of relative position movement between the selected fundus image and panoramic image is performed.
When “move” on the command bar 930 is selected in step S510 of FIG. 5, the user clicks the panorama image displayed in the layer display area 920 or the nonconforming image 940 with the mouse to move the target image. select. Next, the user specifies the destination by dragging and dropping the image while clicking the image. Here, for example, the image generation apparatus 100 uses the single image being dragged so that the match between the selected single image and the background panoramic image can be easily understood until the relative position movement is specified by the user. Is displayed on the display unit 150 so that the background is transparent. In addition, as a method of designating the moving destination, a grid is displayed over the layer display area 920 to define the X coordinate and the Y coordinate, and then the moving destination coordinate (X, Y) of the image selected by the user from the parameter setting unit 950. ) May be used.

When the position is determined, the panorama information correction unit 170 performs a correlation calculation between the background panoramic image and the moved single image at that position to determine an accurate position and affine coefficient. These processes are similar to steps S340 to S360 of FIG. 3 performed by the panoramaization processing unit 120, but the search range in step S340 is limited to a narrow range because the user roughly specifies the position. It is possible to increase the speed, and in this embodiment, the range is half the angle of view of the single image to be moved. The start of the correlation calculation process may be triggered by a drop of an image by a mouse, or a trigger of pressing the “Enter” button in the parameter setting unit 950 after the drop.

Next, regarding the image moving process, a function of moving a plurality of images at the same time will be described. In a panoramic image in which images with different angles of view are mixed, the image with a small angle of view basically has a higher pixel density and higher resolution, so a small angle of view image is pasted on top of a wide angle of view image. To be combined. Therefore, when the movement of the wide-angle image is instructed, it is desirable that the small-angle image attached thereon can be moved at the same time. Here, when selecting images to be moved, multiple images can be selected by a mouse click or the like while holding down the Control key of the keyboard, which is a general means of multiple selection, and all the selected images can be moved in parallel. It is prepared. In addition, when a wide-angle-of-view image is selected and moved, a setting may be provided in which the small-angle-of-view image pasted on the wide-angle-of-view image to be moved is also moved. In this case, the correlation calculation after movement can be limited between the background panoramic image and the wide-angle image, and the calculation processing can be omitted by omitting the calculation process between the wide-angle image and the small-angle image. It is possible to speed up. Information necessary for correcting the positional relationship between the wide-angle image and the small-angle image is read from the panorama information storage unit 130, and the affine coefficient changed by the correction process of the panorama information correction unit 170 is the panorama information. It is overwritten in the storage unit 130. Further, the order of determining the front surface/rear surface of the single images determined in step S340 of FIG. 3 is also read from the panorama information storage unit 130. Then, when a new overlapping portion of the single images occurs when the images are moved, which image is displayed on the front is determined according to this order.

In this way, the image generation apparatus 100 can perform the process of recalculating the correlation and determining the affine coefficient in accordance with the correction content based on the rough movement instruction represented by the user's mouse operation.

[Change; replacement of selected image]
Next, the processing when the modification type is “change” will be described.
When “change” on the command bar 930 is selected in step S510 of FIG. 5, the user clicks the single image on the panoramic image to be replaced displayed on the layer display area 920 with the mouse to select it. Then, the panorama information correction unit 170 reads out the reference position information (presentation position information of the fixation target) of the selected single image from the single image storage unit 110, and at the same time, the same reference position information and image as the selected single image. Search for other single images with corners. Then, if there is no such image, the image generating apparatus 100 displays a message indicating that there is no image to be replaced on the display unit 150, presents it to the user, and ends the process.

On the other hand, if there is a replacement target image, the panorama information correction unit 170 performs contrast adjustment on the replacement target image in the same manner as in step S320, and saves it in the panorama information storage unit 130. After that, the parameter setting unit 950 of the utility software changes to the display of FIG. 6, and the panorama information correction unit 170 performs image selection, frame selection, and averaged image generation by the user. Here, it is assumed that there are three types of replaceable images. Each image is assigned to the buttons of Image1 to Image3 of the parameter setting unit 950 shown in FIG. 6, and the image on the panoramic image is replaced by pressing any button. Further, the frame No. is directly operated in the wheel operation of the mouse or in the “Frame No.” box of the parameter setting unit 950 shown in FIG. By inputting the frame number to be displayed. Can be changed. Here, even if the images at the same reference position, that is, the same fixation target presenting position, the positions may differ due to involuntary eye movements, etc., the process of moving the selected image described above in parallel with frame selection is performed. You may also be able to.

Further, when the user presses the “ave” button of the parameter setting unit 950 shown in FIG. 6, the panorama information correction unit 170 generates an averaged image by selecting all frames or only good images, and the panorama image is generated. Try to replace it with the image above. Once the image to be replaced is determined, the user presses the “Enter” button of the parameter setting unit 950 shown in FIG. 6, and the panorama information correction unit 170 causes the background panorama image, which is an automatic process in the subsequent stage, to be replaced with the replaced single image. Correlation calculation is performed to determine the exact position and affine coefficient. These processes are the same as steps S340 to S360 of FIG. 3 performed by the panoramaization processing unit 120.

If there is an image with the same angle of view or an image with a smaller angle of view in front of the replaced single image on the panoramic image, the front/back of the single images stored in the panorama information storage unit 130 is determined. With reference to the order, the single images located in front of the replaced image are sequentially reprocessed in steps S340 to S360 of FIG. 3 in this order. Then, the affine coefficient changed by the series of correction processes and the selected image ID and frame number are overwritten and saved in the panorama information saving unit 130. In this processing, for example, when the number of background single images is limited to one like the single image 921 with a small angle of view in FIG. 4, based on the accurate position adjustment of the background single image and the affine coefficient. The calculation processing may be reduced by adjusting the image 921.

In this way, the image generation apparatus 100 performs complicated processing such as re-calculation of the correlation again caused by the user's image replacement instruction, affine coefficient determination, and change of the affine coefficient of another single image affected by the replacement. I can carry it.

[Frame; frame number. Change]
Next, the processing when the modification type is “frame” will be described.
When “frame” on the command bar 930 is selected in step S510 in FIG. 5, the user selects the frame number to be used. Select a single image from the panorama image. Then, the panorama information correction unit 170 reads another frame image of the selected image from the single image storage unit 110, performs the same contrast adjustment as in step S320, and stores it in the panorama information storage unit 130. Next, the parameter setting section 950 of the utility software changes to the display of FIG. 7, and the frame number is directly displayed in the wheel operation of the mouse or in the “Frame No.” box of the parameter setting section 950 shown in FIG. By inputting the frame number to be displayed. Can be changed. Subsequent processing is the same as the above-described processing of replacing (changing) the selected image.

[Front/back; swap front/back]
Next, a process when the type of modification is “front/back” will be described. Specifically, when the type of modification is “front/back”, a process of changing the order in which at least one fundus image forming the panoramic image is pasted is performed in the process of generating the panoramic image.
When “front/back” on the command bar 930 is selected in step S510 of FIG. 5, the user selects a single image to be moved front/back from the panoramic image. When the number of selected single images is two, the parameter setting unit 950 of the utility software changes to the display of FIG. 8A, and every time the “front⇔back” button is pressed, the order of the two selected layers in the panoramic image is changed. Are replaced. Press the button again to return to the original state.

Further, when only one single image is selected, the parameter setting unit 950 of the utility software changes to the display of FIG. 8B. The four operation buttons shown in FIG. 8B represent processing for each selected image. For example, when the “Send Backward” button is selected, one layer is moved to the back.

These processes may be assigned to the mouse wheel so that the layer of the selected image can be moved by the wheel. In both cases shown in FIGS. 8A and 8B, when the layer movement (exchange) is determined, the “Enter” button is pressed to cause the panorama information correction unit 170 to perform the subsequent automatic processing. Correlation calculation is performed between the background panoramic image and the layer-shifted image to determine the accurate position and affine coefficient. At this time, the panorama information correction unit 170 rewrites the order of determining the front surface/back surface of the single images stored in the panorama information storage unit 130 according to the layer movement performed. Further, the panorama information correction unit 170 performs the re-processing of S340 to S360 illustrated in FIG. 3 in order of the single image in front of the images involved in the layer movement in this order. Then, the panorama information correction unit 170 overwrites and saves the affine coefficient changed by the series of correction processes in the panorama information storage unit 130. Also in this case, when the background image is limited to one single image, the background image may be modified to reduce the processing weight.

In this way, the image generation apparatus 100 performs complicated processing such as re-calculation of correlations that occurs again when the layer of the image selected by the user is changed, affine coefficient determination, and change of affine coefficients of other single images that are affected. It is possible to generate a new panoramic image. In the present embodiment, the panorama information correction unit 170 takes into consideration not only the order in which the fundus images are pasted when the panoramic image is generated, but also at least one of the information indicating whether or not the fundus images overlap each other. , At least one of the order of adding the second change content and the selection of the fundus image affected by the first change content may be determined.

[Contrast; contrast adjustment]
Next, the processing when the modification type is "contrast" will be described.
When “contrast” on the command bar 930 is selected in step S510 of FIG. 5, the user selects an image for which contrast adjustment is to be performed from the panoramic image. Then, the parameter setting section 950 of the utility software changes to the display of FIG. When the user manually adjusts the brightness, contrast, and gamma value of the selected single image, a correction value such as "+10" or "-15" is input to each input box of the parameter setting unit 950 shown in FIG. input. The correction is immediately reflected in the panoramic image, and the user adjusts the correction value while observing the state of the panoramic image, and when the correction value is determined, the user can press the “Enter” button shown in FIG. 9 to change the contrast. Determine. By this operation, the panorama information correction unit 170 overwrites and saves new single image data reflecting the contrast adjustment in the panorama information storage unit 130. In addition to the manual adjustment, a semi-automatic contrast adjustment function is also provided, and by pressing the "Auto" button shown in FIG. 9, the mode is switched to the semi-automatic mode.

In the adaptive optics SLO device, particularly in the case of a diseased eye or the like, a part of the image captured due to the state of the pupil is darkened at the upper part or the right end, and an image with a gradationally distributed luminance distribution is obtained. May be acquired. Then, for example, as shown in FIG. 10A, the boundary of the panoramic image may be conspicuous due to the luminance distribution with gradation. The semi-automatic (contrast) adjustment function here is an example of a correction means for correcting these brightness gradations. However, the present embodiment is not limited to this.

In the semi-automatic mode, the user can mark, on the panoramic image, the portion where the boundary is conspicuous in the stitched portion due to the difference in brightness between the single images, using the mouse. Further, for an area of a single image where gradation is applied to the brightness, drag and drop an arrow 960 indicating the gradation direction arranged in the parameter setting unit 950 shown in FIG. Adjust the rotation so that it is parallel to the direction. It is assumed that these operations are repeated to mark a noticeable boundary and an area with gradation due to the brightness difference, and finally, for example, a white broken line 1010 and an arrow 1020 in FIG. In this state, pressing the "Enter" button of the parameter setting unit 950 shown in FIG. 9 starts automatic correction. The panorama information correction unit 170 performs automatic correction for each single image in which the arrow 1020 is arranged, sets the brightness difference of the boundary portion indicated by the white broken line 1010 as the boundary condition, and sets the direction in the direction indicated by the arrow 1020. Gradation correction is performed by monotonically increasing or decreasing the luminance. Then, the panorama information correction unit 170 performs boundary blending processing after gradation correction, and then overwrites and saves image data of a new single image reflecting the contrast adjustment in the panorama information storage unit 130.

In this way, the image generating apparatus 100 can perform complicated contrast adjustment in consideration of gradation in accordance with a simple correction instruction from the user.

In the first embodiment, intricate operations such as high-definition additional corrections and adjustments to the entire panoramic image that are necessary due to the correction of a single image are made according to the rough corrections that the user makes to the panoramic image and the like. The image generating apparatus 100 is responsible for the processing of various correction contents.
With this configuration, when a panoramic image obtained by combining a plurality of fundus images (single images) is corrected, it is possible to reduce the work load on the user who performs the correction and improve the accuracy of the correction work. You can

(Second embodiment)
Next, a second embodiment of the present invention will be described.

In the second embodiment, the image generation apparatus of the first embodiment that corrects the panoramic image obtained by combining the fundus images captured by the adaptive optics SLO apparatus is further presented to the user. In this mode, the user's correction work load is further reduced. Here, as the correction candidate in the present embodiment, for example, the candidate of the relative position between the selected fundus image and the panoramic image, the first change content or the second change described in the first embodiment, Candidates for the order in which content is added can be applied. The configuration of the image generation apparatus according to the second embodiment and the processing procedure of the panoramaization processing are the same as those in the first embodiment. In the following, in particular, a function of presenting a correction candidate added to the panoramic image correction procedure will be described.

As the processing of the second embodiment, when the user selects a single image to be modified and selects the type of modification (command), the image generation apparatus 100 narrows down the modification candidates based on rough conditions and calculations, and then provides the modification to the user. Present. Then, when the user selects a correction candidate determined to be correct, the image generation apparatus 100 further performs high-definition processing to determine the correction content for the panoramic image. In this way, in the correction by the user, the image generation apparatus 100 narrows down the correction candidates in advance and presents them to the user, so that the correction work by the user can be further simplified.

<Panorama image correction procedure>
Hereinafter, with reference to the UI screen of FIG. 11 and the flowchart of FIG. 12, in the image generating apparatus 100 according to the present embodiment, a panorama image correction processing procedure and a user operation will be described.

FIG. 11 shows the second embodiment of the present invention, and is a diagram showing an example of the UI screen of the utility software displayed on the display unit 150 of FIGS. 1 and 2. 11, the same components as those shown in FIG. 4 are designated by the same reference numerals.

The UI screen 910 shown in FIG. 11 is the same as the UI screen 910 according to the first embodiment shown in FIG. 4 except that an “Auto Advance” button 951 is further arranged in the parameter setting unit 950.

When the “Auto Advance” button 951 shown in FIG. 11 is pressed, the utility software switches to a mode for presenting correction candidates, and the button 951 changes color to display that mode. When the button 951 is pressed again, the normal mode is restored, and the utility software performs the same operation as in the first embodiment. In this embodiment, of the five commands shown in the command bar 930, the following three commands are effective in the mode of presenting correction candidates. Note that the remaining two commands can be used with the same operation specifications as in the first embodiment, regardless of whether the "Auto Advance" button 951 is pressed.
・Move... Change the position of single image.
-Change... Replaces a single image with another image.
Frame: Change the frame number used for the panoramic image.

When any one of these three commands is selected when the “Auto Advance” button 951 is valid, correction candidates are presented according to the single image to be corrected selected by the user. In order to determine the candidate selected by the user and start the panorama image correction process, the “Enter” button located below in the parameter setting unit 950 of FIG. 11 is pressed.

Next, a flow of correcting a panoramic image will be described with reference to the flowchart of FIG.
FIG. 12 is a flowchart showing an example of the processing procedure of the panoramic image correction processing performed in the image generating apparatus 100 according to the second embodiment of the present invention. In FIG. 12, processing steps similar to those shown in FIG. 5 are designated by the same step numbers. That is, the processing from steps S510 to S530 is the same as the processing in the first embodiment shown in FIG. 5, and therefore the description thereof is omitted.

<<Step S1210>>
In step S1210, the panorama information correction unit 170 calculates the correction candidates for the selected single image and the correction content (first modification content) related to the type of correction specified in step S510. At this time, the panorama information correction unit 170 may obtain a correction candidate by calculation based on the photographing conditions of the fundus image forming the panoramic image or at least one fundus image of the fundus images that can be formed. The panorama information correction unit 170 that calculates this correction candidate by calculation constitutes a second calculation means.
After that, the display unit 150 displays the correction candidates calculated by the panorama information correction unit 170 in the layer display area 920 and presents them to the user. The display unit 150 that presents the correction candidates to the user constitutes a presentation unit. Here, the correlation calculation between the single image to be corrected, the image of another frame that is a correction candidate, the other unused image stored in the single image storage unit 110 and the background image, and the incidental information of the image are used. Then, the correction candidate is presented or selected from a plurality of candidates and presented.

<<Step S1220>>
Step S1220 is a step in which the image generation apparatus 100 accepts an input in the process of trying and searching various conditions when the user decides a correction candidate.
Specifically, in step S1220, the image generation device 100 (for example, the panorama information correction unit 170) determines whether the user has given an instruction for additional input. For example, the input here is rough current position information of a single image which is being moved by mouse dragging, frame selection, image selection, and the like. Since the selection condition of the correction candidate is changed by these inputs, if the user gives an instruction for additional input (S1220/Yes), the process returns to step S1210 to redo the selection of the correction candidate and present it to the user.
On the other hand, as a result of the determination in step S1220, if the user has not instructed additional input (S1220/No), the process proceeds to step S1230.

<<Step S1230>>
In step S<b>1230, the panorama information correction unit 170 determines detailed correction parameters corresponding to the single image to be corrected and the type of correction when the user presses the “Enter” button of the parameter setting unit 950. At this time, when any one of the presented correction candidates is selected by the user, the panorama information correction unit 170 selects the original correction content (first change content) based on the selected correction candidate. (First correction content) The panorama information correction unit 170 that performs this replacement constitutes a replacement unit.

<<Step S1240>>
Subsequently, in step S1240, the panorama information correction unit 170 performs contrast adjustment, correlation calculation, affine transformation, etc. according to the correction parameter received and determined in step S1230, and performs recalculation of correlation between single images. The panorama information stored in the panorama information storage unit 130 is corrected and updated. After that, the panorama conversion processing unit 120 generates a new panoramic image based on the corrected panoramic information, and stores the new panoramic image in the panoramic information storage unit 130. After that, the panorama image output unit 140 reads out the panorama image newly generated by the panoramaization processing unit 120 from the panorama information storage unit 130 and outputs the panorama image to the display unit 150 for display.

Then, the image generating apparatus 100 repeatedly performs the processing of steps S510 to S530 and S1210 to S1240 for each item that the user determines to be corrected.

Next, these processing procedures will be described in detail for each type of correction. The description of the same parts as those in the first embodiment will be omitted.

[Move; move selected image]
First, the processing when the type of modification is “move” will be described.
In the present embodiment, two functions are provided as the function of presenting the candidates of the moving destination in moving the image pasting position.

The first function is a function of presenting candidates for the destination.
The panorama information correction unit 170 selects the panorama image at the position where the selected single image was originally pasted or the temporary position determined by dragging the mouse or specifying the coordinates through the parameter setting unit 950. Calculate the correlation with a single image. Correlation calculation here requires high-speed processing in order to respond to the instruction that changes every moment by dragging the mouse, so the area for correlation calculation in the panorama image is centered on the selected single image. The range of view angle is three times the view angle of the selected image. Also, the correlation calculation is performed by performing only the same rough calculation as in the method in step S340 of FIG. 3, omitting the calculation for highly accurate alignment in step S350 of FIG. 3 and S360 of FIG. 3 to reduce the calculation amount. There is. That is, in the present embodiment, the amount of calculation performed for one-time image registration is determined as a correction candidate in the present embodiment rather than the calculation for determining the second change content described in the first embodiment. There are less calculations. Then, as a result of the correlation calculation, a position having a high correlation is displayed as a pasting candidate position at a few places and displayed to the user.

FIG. 13 shows the second embodiment of the present invention, and is a diagram showing an example of the single image 200 being dragged by the mouse. In FIG. 13A, two small areas 210-1 and 210-2 surrounded by broken lines are positions presented as moving destination candidates with high correlation, and a large range 220 surrounded by broken lines is for correlation calculation. At this time, the range of the panoramic image (correlation calculation range) whose area is limited is shown. The latter frame (220) does not need to be presented to the user. Further, in order to make it easy to understand the display of the image being dragged and the frame of the movement destination candidate, the frame line of each single image in the layer display area 920 shown in FIG. FIG. 13B shows a diagram when the single image 200 is further moved as compared with the case of FIG. 13A by advancing the mouse drag. The correlation calculation range 220 also moves with the movement of the single image 200, whereby the area 210-1 of the candidate frame in FIG. 13A is removed from the candidates, and the area 210-3 of the candidate frame is newly replaced. It is presented as a candidate. The user is configured so that he/she can select the one that seems to be correct from the presented candidate frames, drag and overlap the selected image thereon, and drop it if they match. In this embodiment, only the broken line frame is displayed as the candidate position, but a copy of the selected image may be displayed so as to overlap each candidate position. After the position is determined, the same processing as that of the first embodiment is performed thereafter, and accurate alignment is performed by the highly accurate correlation calculation in step S350 in FIG. 3 and the affine coefficient determination in step S360 in FIG. Regeneration of the panoramic image is executed.

The second function is a function of presenting the correction order.
When position adjustment is performed on multiple single images in sequence, every time one image is moved, the single image (layer) pasted above the moved image is affected by the correction of the lower image and Correlation calculation and affine coefficient recalculation are required. Therefore, by using the order of combining determined when the panoramic image is generated, that is, the order of determining the front side/back side of the single images stored in the panorama information storage unit 130, the user can perform the processing according to this order. prompt. When “move” in the command bar 930 shown in FIG. 11 is selected with a plurality of single images selected, the image generating apparatus 100 according to the present embodiment first determines whether or not the same moving process is performed on all the images. The user is asked to select by displaying a dialog. If it is a batch move, move to that process, but if not, display the single image that is the backmost layer of the selected multiple images in the foreground, and only the images behind it. Is displayed as a panorama.

FIG. 14 shows the second embodiment of the present invention and is a diagram showing an example of an image displayed in the layer display area 920 of the UI screen 910 shown in FIG. 11. In FIG. 14A, it is assumed that the medium angle-of-view image 310 and the small angle-of-view image 320 near the center are selected as the single images to be moved. Then, since the middle-angle-of-view image 310 is the rearmost image, a display prompting the movement of the middle-angle-of-view image 310 shown in FIG. Disappear. When the position of the medium-angle-of-view image 310 is determined in this state, the medium-angle-of-view image 310 and the small-angle-of-view image 320 are reprocessed in S340 to S360 of FIG. Other images in the intervening layers are also reprocessed in S340 to S360 of FIG. 3 in the order of the layers. Next, the display of FIG. 14C is displayed, and the small angle of view image 320 is aligned. In the subsequent processing, after the position of the small angle of view image 320 is similarly determined, the small angle of view image 320 and the image in front of it are reprocessed to regenerate the panoramic image.

In this way, by presenting the destination of the image to simplify the user's selection, or by defining the order of position correction to minimize the calculation waiting time by reprocessing, the workload of editing the panoramic image is reduced. And time can be reduced.

[Frame; frame number. Change]
Next, the processing when the modification type is “frame” will be described.
As described in the first embodiment, even in the continuous frame images captured at the same fixation target presenting position, the frame No. The position may differ depending on the location. For example, by operating the wheel of the mouse or directly in the “Frame No.” box of the parameter setting unit 950 shown in FIG. By inputting the frame number to be displayed. If it is changed, it becomes necessary to correct the position. In the first embodiment, the position can be manually corrected, but in the present embodiment, the correlation calculation between the background image and the new frame is performed every time the frame is changed, and the background image is displayed at the position with high correlation. Even in this case, since high-speed processing is required in order to respond to an instruction that changes every moment by operating the wheel of the mouse or the like, the area for which the correlation calculation is performed is limited in the panoramic image, and the correlation calculation is performed in the steps of FIG. It is desirable to use only the same rough calculation as the method in S340.

As a result, it is possible to select a frame while automatically correcting the positional deviation due to the frame change, and it is possible to reduce the workload of frame selection.

<change; replacement of selected image>
Next, the processing when the modification type is “change” will be described.
Even in the replacement of the selected image, even if the images have the same reference position information (fixation target presentation position), the imaging position may differ due to the influence of the fixational eye movement as described above. The imaging position may change every time. Therefore, it is useful to visually know at what position each of the candidate images to be replaced is an image captured, when visually selecting the replacement candidate.

In the parameter setting unit 950 shown in FIG. 6, it is assumed that “Image2” is selected as the candidate image to be replaced. Then, the panorama information correction unit 170 calculates a rough position by performing rough correlation calculation with the background image for all the frames, and then presents a range in which the image is at least one frame or more. If the number of frames is large, the processing here is expected to take some time even for area limitation or rough correlation calculation. It is desirable to store it in the storage unit 130. Further, since it is troublesome to display all the frames in the layer display area 920, only the “Frame No.” of the parameter setting unit 950 shown in FIG. 6 or the frame selected by the mouse wheel is displayed at the position obtained by the correlation calculation. FIG. 15 shows the layer display area 920 in which they are displayed.

FIG. 15 shows the second embodiment of the present invention and is a diagram showing an example of an image displayed in the layer display area 920 of the UI screen 910 shown in FIG.
In FIG. 15, a single image 400 surrounded by a thick frame line is an image candidate after replacement, and a single frame No. with "Image2" is displayed. Is an image of. A dashed frame line 410 indicates a range in which at least one image is present in all the frames of “Image2”. By displaying in this way, it is possible to immediately determine whether or not the imaging frame at the position desired by the user is included.

When there are a large number of images with the same reference position information, the user's work of selecting candidates for replacement becomes complicated. In such a case, the parameter associated with the focus position is extracted from the information attached to the single image, and the average value of all the single images forming the panoramic image is calculated. By presenting the image captured at the focus position close to the calculated average value as a candidate limitedly or preferentially, the user can quickly select an appropriate replacement target image from many image candidates. be able to. Since the fundus image (retinal image) captured by the adaptive optics SLO device has a shallow depth of focus, the captured image differs depending on the focus position, and when generating a panoramic image, it is necessary to select an image with a close focus position.

According to the second embodiment, when the user makes a correction on the panoramic image, the image generation device 100 narrows and presents the correction candidates in advance, so in addition to the effect of the first embodiment. It is possible to further reduce the work load of the user who makes the correction.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
The program and a computer-readable storage medium storing the program are included in the present invention.

It should be noted that the above-described embodiments of the present invention are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be limitedly interpreted by these. Is. That is, the present invention can be implemented in various forms without departing from its technical idea or its main features.

100 image generation device, 110 single image storage unit, 120 panorama processing unit, 130 panorama information storage unit, 140 panorama image output unit, 150 display unit, 160 user interface unit, 170 panorama information correction unit

Claims (13)

An image generation device for generating a panoramic image by combining a plurality of fundus images,
Selecting means for selecting at least one fundus image of the fundus image that constitutes the panoramic image or a fundus image that can constitute the panoramic image;
A first performed on at least one of the fundus image and the panoramic image selected by the selecting unit according to the first change content designated by the user for the fundus image selected by the selecting unit; A first calculation means for calculating the change content of 2 by calculation;
Generating means for generating a new panoramic image based on the second change content,
The first change content is relative position movement of the fundus image and the panoramic image selected by the selection means,
An image generating apparatus, characterized in that the fundus image selected by the selecting means is displayed in a semi-transparent state until the relative position movement destination is designated as the first change content. The first change content further includes a change in which at least one fundus image forming the panoramic image is replaced with another fundus image that can form the panoramic image. The image generation device described. The first change content further includes a change in the order in which at least one fundus image forming the panoramic image is pasted in the process of generating the panoramic image. The image generation device described. The image generation apparatus according to claim 1, wherein the first change content further includes a change in contrast of at least one fundus image forming the panoramic image. When there is another fundus image pasted on the fundus image selected by the selecting unit when the second change content is calculated, the first computing unit determines the other pasted fundus image. The image generating apparatus according to claim 1, wherein the fundus image is modified by the same modification as that of the fundus image selected by the selecting unit. The first calculation means selects a fundus image affected by the first change content from among fundus images forming the panoramic image,
The image according to any one of claims 1 to 5, wherein the generation unit generates the new panoramic image by adding the second modification content to the affected fundus image. Generator. The first calculation means uses the information of at least one of the order of stitching the fundus images when the panoramic image is generated or the presence/absence of overlapping of the fundus images, and the order of adding the second change content. The image generation apparatus according to claim 6, wherein at least one of selection of the affected fundus image is determined. The image according to any one of claims 1 to 7, wherein the first calculation unit includes, as the first change content, a change for aligning at least two or more images. Generator. 9. The image generation apparatus according to claim 8, wherein the method of aligning the images calculates a position having the highest correlation by using a normalized correlation coefficient. Second calculation means for calculating a correction candidate for the first change content by calculation;
Presentation means for presenting a plurality of the correction candidates,
Replacing means for causing the user to select any one of the plurality of correction candidates presented by the presenting means and replacing the first modification content with the first modification content;
Further has,
10. The image generating apparatus according to claim 1, wherein the first calculation unit calculates the second change content according to the first modification content. The image generation apparatus according to claim 10, wherein the correction candidate is a candidate for a relative position between the fundus image selected by the selection unit and the panoramic image. An image generation method by an image generation device that generates a panoramic image by combining a plurality of fundus images,
A selection step of selecting at least one fundus image of the fundus image that constitutes the panoramic image or a fundus image that can be configured;
A first performed on at least one of the fundus image selected at the selecting step and the panoramic image according to the first change content specified by the user for the fundus image selected at the selecting step; A first calculation step of calculating the change content of 2 by calculation;
A generation step of generating a new panoramic image based on the second change content,
The first change content is relative position movement of the fundus image and the panoramic image selected in the selecting step,
The image generation method, wherein the fundus image selected in the selecting step is displayed in a semitransparent state until the relative position movement destination is designated as the first change content. A program for causing a computer to function as each unit in the image generating apparatus according to claim 1.