JP2017055916A - Image generation apparatus, image generation method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mechanism which, when correcting a panoramic image obtained by splicing a plurality of fundus images together, realizes reduction of the workload of a user performing the correction and improvement of the accuracy of the correction work.SOLUTION: In an image generation apparatus 100 which generates a panoramic image by splicing a plurality of fundus images together, provided are a user interface unit 160 for selecting at least one change target fundus image from among fundus images which constitute or may constitute a panoramic image, a panoramic information correction unit 170 for finding, by calculation, the content of a second change to be performed for at least one side among the selected fundus image and the panoramic image according to the content of a first change specified by a user for the selected fundus image, and panoramic processing unit 120 for generating a new panoramic image based on the content of the second change.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image generation apparatus that generates a panoramic image by pasting together a plurality of fundus images, an image generation method using the image generation apparatus, and a program for causing a computer to function as each unit in the image generation apparatus. is there.

  For the purpose of early diagnosis of lifestyle-related diseases and diseases that occupy the top causes of blindness, examinations of the fundus of the eye to be examined are widely performed. As an ophthalmologic apparatus using the principle of a confocal laser microscope, there is a scanning laser opthalmoscope (SLO). This SLO device is a device that performs a raster scan on the fundus of the subject's eye with laser as measurement light, and obtains a fundus image that is a planar image with high resolution and high speed from the intensity of the return light.

  2. Description of the Related Art In recent years, compensation optics SLO (AO-SLO: Adaptive Optics) has a compensation optical system that measures aberrations of a subject's eye in real time with a wavefront sensor and corrects aberrations of measurement light generated in the subject's eye and its return light with a wavefront correction device. SLO) devices have been developed. In this adaptive optics SLO apparatus, it is possible to acquire a high-resolution planar image.

  When acquiring a fundus image that is a high-resolution planar image with an SLO device, there is a problem of time to image one frame in addition to the problem of optical aberration of the device itself, and the imaging range for imaging at one time, that is, the angle of view is narrow. turn into. On the other hand, in order to obtain information necessary for diagnosis, examination, etc., a fundus image of a certain wide range is required, so that it is necessary to perform imaging at a plurality of positions by the SLO device. Further, even when photographing a local part of the eye to be examined, a wide range of the eye to be examined may be photographed separately at a low resolution in order to grasp the whole. However, even if a region required for diagnosis or examination is photographed, it is necessary to grasp the relevance between a plurality of photographed images obtained. That is, it is required to generate a single high-definition panoramic image by clarifying the positional relationship between the captured images and combining them according to the positions.

  In order to solve such a problem, Patent Document 1 proposes a composite apparatus that combines an SLO apparatus with a wide angle of view and an SLO apparatus with a small angle of view but a high resolution. Japanese Patent Application Laid-Open No. 2004-228561 proposes a method of generating a panoramic image by accurately recognizing the positional relationship between the respective fundus images even when the fundus images having different angles of view and resolution are bonded together.

JP 2010-259543 A JP 2014-128366 A

R. Szeliski, Image alignment and stitching: a tutorial, Tech. Rep. MSR-TR-2004-92, Microsoft Research, December 2004.

  As various methods are introduced in Non-Patent Document 1, it is a common method to perform panorama image alignment by aligning a plurality of images. However, for example, when features such as a high-definition image with a small angle of view of the fundus are very small structures such as photoreceptor cells and capillaries, they may not be resolved into a wide-angle image. . In the method of Patent Literature 1, when two small-angle images and wide-angle images with few features are combined, there may be a case where matching is not performed. Further, the method of Patent Document 2 can deal with the above-described case, but there may be few features depending on the imaging region and the image quality of the obtained image even if the images are small angle of view. The case which does not do will remain.

  The generation of a panoramic image can be greatly reduced, for example, by automating the processing of a computer program as a user (examiner). However, if the above-mentioned unmatched images remain, their alignment is performed manually, which places a burden on the user. Further, as described in Non-Patent Document 1, the panoramic processing includes various processes such as image brightness adjustment and distortion correction, image selection in the image superimposing unit, and joining processing in addition to the alignment. Processing is executed. Regarding these processes as well, if the above-mentioned unmatched images remain, for example, the process of the computer program may not be completed correctly, and manual correction occurs. Furthermore, it is difficult to obtain such accuracy (rigidity) by hand for a computer program that performs alignment and various correction processes by calculation.

  The present invention has been made in view of such problems, and when correcting a panoramic image obtained by pasting together a plurality of fundus images, the workload of the user who performs the correction is reduced and the correction work is performed. The purpose is to provide a mechanism for improving accuracy.

The image generation apparatus of the present invention is an image generation apparatus that generates a panoramic image by pasting together a plurality of fundus images, and at least one change target of the fundus image constituting the panoramic image or the configurable fundus image. A fundus image selected by the selection unit according to a first change specified by the user with respect to the fundus image selected by the selection unit, and a panoramic image, First calculating means for calculating a second change content to be performed on at least one of the two, and generating means for generating a new panoramic image based on the second change content.
The present invention also includes an image generation method by the above-described image generation apparatus and a program for causing a computer to function as each unit in the above-described image generation apparatus.

  According to the present invention, when correcting a panoramic image obtained by pasting together a plurality of fundus images, it is possible to reduce the work burden on the user who performs the correction.

It is a figure which shows an example of a function structure of the image generation apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the hardware constitutions of the image generation apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows an example of the process sequence of the panorama process performed in the panorama process part of FIG.1 and FIG.2. FIG. 3 is a diagram illustrating an example of a UI screen of utility software displayed on the display unit in FIGS. 1 and 2 according to the first embodiment of this invention. It is a flowchart which shows an example of the process sequence of the panorama image correction process performed in the image generation apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the 1st Embodiment of this invention and shows the example of a display of the parameter setting part in the image replacement mode of utility software. It is a figure which shows the 1st Embodiment of this invention and shows the example of a display of the parameter setting part in the frame change mode of utility software. It is a figure which shows the 1st Embodiment of this invention and shows the example of a display of the parameter setting part in the layer change mode of utility software. It is a figure which shows the 1st Embodiment of this invention and shows the example of a display of the parameter setting part in the contrast adjustment mode of utility software. It is a figure which shows the 1st Embodiment of this invention and shows an example of the panoramic image in the fundus image image | photographed with the SLO apparatus. It is a figure which shows the 2nd Embodiment of this invention and shows an example of UI screen of the utility software displayed on the display part of FIG.1 and FIG.2. It is a flowchart which shows an example of the process sequence of the panorama image correction process performed in the image generation apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows the 2nd Embodiment of this invention and shows an example of the single image in the middle of being dragged with the mouse | mouth. It is a figure which shows the 2nd Embodiment of this invention and shows an example of the image displayed on the layer display area of UI screen shown in FIG. It is a figure which shows the 2nd Embodiment of this invention and shows an example of the image displayed on the layer display area of UI screen shown in FIG.

  Hereinafter, embodiments (embodiments) for carrying out the present invention will be described with reference to the drawings.

(First embodiment)
First, a first embodiment of the present invention will be described.

  In the present embodiment, the panorama image is generated by the user with respect to a panorama image generated by combining a plurality of fundus images (hereinafter referred to as “single images”) obtained by photographing the retina of the eye to be examined with the adaptive optics SLO device. A process for correcting a single image constituting an image will be described. Here, in the present embodiment, the single image is an image or moving image obtained by imaging the plane of the retina of the eye to be examined, and is a front image obtained when the eye to be examined is photographed from the front with a fundus camera or the like. It shall be said. As processing, after the user selects a single image to be corrected and gives a rough correction instruction, the image generation apparatus calculates a correlation between the single image and the surrounding image based on the specified instruction and corrects the correction. Determine the amount. As described above, in the present embodiment, in the correction work by the user, the image generation apparatus is responsible for high-definition correction processing such as correlation calculation, thereby simplifying the user's correction work and, as a result, the user performing the correction It is intended to reduce the work load and improve the accuracy of the correction work.

<Configuration of Image Generation Device>
FIG. 1 is a diagram illustrating an example of a functional configuration of an image generation apparatus 100 according to the first embodiment of the present invention. The image generation apparatus 100 is an apparatus that generates a panoramic image by combining a plurality of single images. The image generation apparatus 100 includes functions of a single image storage unit 110, a panorama processing unit 120, a panorama information storage unit 130, a panorama image output unit 140, a display unit 150, a user interface unit 160, and a panorama information correction unit 170. It has a configuration.

  The single image storage unit 110 stores a single image that is a plurality of fundus images that are planar images at various positions in the fundus of the eye to be examined, which are captured by the adaptive optics SLO device. Further, in the single image storage unit 110, accompanying information of the single image is stored in association with the single image together with the single image. Here, the supplementary information includes left and right eye information of the eye to be examined, image angle of view information, reference position information on the fundus of the photographed eye to be examined, image ID information for distinguishing images having the same photographing conditions, and the like. is there. The reference position information is, for example, fixation target presentation position information used by the SLO device when photographing. The image ID information is information such as serial numbers.

  The panorama processing unit 120 selects a single image that is a fundus image based on the reference position information, calculates a correlation between the single images, sequentially determines a relative position, and combines a plurality of single images. To generate a panoramic image. Then, the panorama processing unit 120 stores panorama information including the generated panorama image and information related to the processing of the panorama image in the panorama information storage unit 130. The panorama information includes position information of each single image on the panorama image, and in addition to parameters such as rotation angle information, luminance correction information, and image deformation information, image processing such as luminance correction and deformation is performed according to these parameters. Including the applied image itself.

  The panorama information storage unit 130 stores the panorama information obtained by the panorama processing unit 120 and the panorama information corrected by the panorama information correction unit 170.

  The panorama image output unit 140 outputs a panorama image included in the panorama information stored in the panorama information storage unit 130 to the display unit 150.

  The display unit 150 displays the panorama image output from the panorama image output unit 140 and presents it to the user.

  The user interface unit 160 is an interface for inputting, to the image generation apparatus 100, an instruction of correction content from the user for the panoramic image displayed on the display unit 150.

The panorama information correction unit 170 corrects the panorama information stored in the panorama information storage unit 130 by performing correlation recalculation of single images based on a correction instruction for the panorama image input from the user interface unit 160. Update.
When the panorama information is corrected by the panorama information correcting unit 170, the panorama processing unit 120 generates a new panorama image based on the corrected panorama information.

  FIG. 2 is a diagram illustrating an example of a hardware configuration of the image generation apparatus 100 according to the first embodiment of the present invention. In FIG. 2, the same components as those shown in FIG. The image generation apparatus 100 according to the present embodiment is realized by, for example, a personal computer (PC).

  The image generation apparatus 100 is configured to have each hardware configuration of an arithmetic unit 201, a storage unit 202, a display unit 150, and a user interface unit 160.

  The arithmetic unit 201 is composed of, for example, a CPU (Central Processing Unit). The panorama processing unit 120, the panorama image output unit 140, and the panorama information correction unit 170 are programs executed by the arithmetic unit 201, for example.

  The storage unit 202 is composed of, for example, a hard disk. The single image storage unit 110 and the panorama information storage unit 130 correspond to the storage area of the storage unit 202.

  Note that the image generation apparatus 100 according to the present embodiment uses a general-purpose interface of a computer such as a LAN (Local Area Network) or a USB (Universal Serial Bus), a database server outside the apparatus, a storage device, an adaptive optics SLO device, or the like. A single image is obtained from the image data and stored in the single image storage unit 110.

  The display unit 150 illustrated in FIG. 2 includes, for example, a display. Also, the user interface unit 160 shown in FIG. 2 includes, for example, a mouse and a keyboard.

<Procedure for panorama processing>
FIG. 3 is a flowchart illustrating an example of a processing procedure of the panorama processing performed in the panorama processing unit 120 of FIGS. 1 and 2.

<< Step S310 >>
Step S310 is an image selection process used for the panorama processing.
Specifically, in step S <b> 310, the panorama processing unit 120 refers to the supplementary information of the single image that is the fundus image stored in the single image storage unit 110, and captures the imaging conditions (the eye to be examined, the angle of view, and the fixation). A search is made as to whether there are single images having the same target presentation position. Then, when such a group of images is found at one position of the eye to be examined, the panorama processing unit 120 selects one image or frame with the highest image quality using any known image quality evaluation index. . The panorama processing unit 120 selects the single image when there is no image with the same shooting condition and there is only one single image. That is, in this step, the panorama processing unit 120 selects only one representative single image from a plurality of single images having the same shooting conditions, and sets one corresponding single image for each shooting condition. . The panorama processing unit 120 stores information such as an image ID and a frame number in the panorama information storage unit 130 as information on a single image selected for each shooting condition.

  Here, the image is selected by aligning the images of each frame in one moving image, a frame whose brightness is extremely low due to blinking, a frame in which image distortion is caused by fixation fixation, or an aberration correction failure. A frame having a low S / N ratio may be excluded as an exceptional frame, an averaged image may be generated using the remaining images, and the generated averaged image may be used. Note that any known alignment method can be used as the inter-frame alignment and the averaged image generation method. If the same fixation image is an image at a relatively same position with the subject eye having good fixation, an averaged image may be generated using the above-described method between a plurality of images at the same position. Further, the user may select an image, or an averaged image may be manually generated and used. When the panorama processing unit 120 generates an averaged image, the panorama information storage unit 130 stores the averaged image itself generated together with the image information.

<< Step S320 >>
Step S320 is a step of adjusting the contrast of the image used for the panorama processing.
Specifically, in step S320, the panorama processing unit 120 selects an image with the best image quality among the images selected for panorama processing. The image quality evaluation for selection may be simple contrast comparison, or any known image quality evaluation index may be applied. For the selected image, the gradation distribution bias on the histogram is eliminated by applying linear gradation conversion, and this is used as a reference image. Then, the panorama processing unit 120 calculates the average of the gradation values of all the pixels of the reference image, and the average of the gradation values of the image is equal to that of the reference image for each of the other images selected in step S310. A uniform offset is added to or subtracted from the gradation value. Thereafter, the panorama processing unit 120 applies linear gradation correction to each image within a range in which there is no whiteout or blackout while maintaining the average gradation value. Here, linear gradation correction is basically used, but non-linear gradation conversion using a tone curve may be used, or any other contrast adjustment can be applied. As for the image processed here, according to the image selection information stored in the panorama information storage unit 130, the single image is read from the single image storage unit 110, and the averaged image is read from the panorama information storage unit 130. All the gradation-corrected images are stored in the panorama information storage unit 130.

<< Step S330 >>
Step S330 is a step of selecting a background image for panorama. Here, the background image is a single image or an averaged image of a plurality of frames that is first selected in the panorama processing, and the panorama is obtained by aligning other single images or averaged images with the background image. And you can start from where you put them together.
Since the background image is preferably an image with a wide angle of view, in step S330, the panorama processing unit 120 selects an image with the widest angle of view among the images selected in step S310 as the background image. Here, when there are a plurality of candidate images, the image closest to the average value of the reference position information of each image may be selected, or the image quality is highest using any known image quality evaluation index. An image may be selected.

<< Step S340 >>
Step S340 is a rough alignment process in the panorama processing.
Specifically, in step S340, the panorama processing unit 120 selects the next position selected for the background image selected in step S330, which is the original image for alignment, or the panorama image that has been aligned so far. Roughly align the matching partner image. The selection order of the images to be aligned is given priority to images with a wide angle of view, and based on the reference position information, the order of the regions that overlap the background image and the panorama image that has been aligned so far is the order. . This order is information for determining the front / back of each single image on the panorama image, and is stored in the panorama information storage unit 130.

  Here, the correlation coefficient between the images is calculated by a simple shift such as parallel, rotational movement, and enlargement / reduction with respect to the original image for alignment, which is an arbitrary known method. Depending on the data, rotation or the like may be omitted if there is at least parallel movement. In the present embodiment, the reference position information stored in the single image storage unit 110 is used as the initial value of the position search, and the cross-correlation is calculated by using the pixel value of the region where two images shifted by translation are overlapped. The number of relations (normalized correlation coefficient) is obtained. Then, the position with the highest correlation is calculated using the normalized correlation coefficient, and the normalized correlation coefficient and the obtained coarse alignment position information are stored in the panorama information storage unit 130. In addition, a general pattern patching method such as SAD (sum of absolute values of differences in luminance values) or SSD (sum of squares of differences in luminance values) is used by using similar pixel values. May be. The calculation can be performed faster if the search range is limited to some extent, but a full search may be performed. Here, when the correlation coefficient is so small that the overlap area (overlap) is not found in the alignment of the background image or the original image that has been aligned so far and the partner image to be combined, Information that the partner image is an unmatched image is stored in the panorama information storage unit 130.

<< Step S350 >>
Step S350 is a highly accurate alignment process performed by dividing an image into small regions. Specifically, in step S350, the panorama processing unit 120 positions each individual image obtained by dividing the alignment partner image into small images with respect to the background image selected in step S330 or the panorama image that has been aligned so far. Match. Here, the division of the image can be determined by the feature of the image. In the present embodiment, it is assumed that the 12 × 12 division is performed, but the number is not limited to this. The method for aligning each small image with respect to the background image or the panorama image that has been aligned so far is any known method with the coarse alignment position set in step S340 as the initial position. Align with. In the present embodiment, as in step S340, the shift by translation and the normalized cross-correlation are calculated, and the position with the highest correlation of each small image is acquired.

<< Step S360 >>
Step S360 is a step of performing affine deformation of the image and actual pasting.
Specifically, the panorama processing unit 120 selects an arbitrary image based on the position of each small image set in step S350 with respect to the background image selected in step S330 or the panorama image that has been aligned so far. The affine coefficient of the combined image is determined by a known method. In the present embodiment, the panorama processing unit 120 first creates an image Si in which all small images are arranged on a plane based on the position acquired in step S350. Then, the panorama processing unit 120 draws a rectangular frame line surrounding those small images, transforms the combined image by affine transformation so as to have the same shape as this frame line, and sets this image as Ai. In the first modification, all the small images are not deformed and remain divided. Therefore, it is only necessary to draw frame lines using the four corner images. Next, the panorama processing unit 120 calculates a shift amount between the small images of the image Ai corresponding to the same position as the small images of the image Si by using a correlation coefficient. Then, the panorama processing unit 120 changes the affine coefficient so as to minimize the total shift amount, and repeats the process of calculating the same total shift amount. Then, the panorama processing unit 120 acquires an affine coefficient that minimizes the sum of the shift amounts, and uses this as position information. The panorama processing unit 120 also pastes the changed image by the affine deformation to the reference frame or the panorama image that has been aligned so far, and finally determines the alignment image and the alignment partner that has been finally determined. The affine coefficient of the image is stored in the panorama information storage unit 130.

  The combined image obtained in step S360 is used for alignment of the next image and is updated each time. Furthermore, the method of combining the bonding may be any known method. For example, blending may be performed only on the edge portion, or the luminance value may be adjusted. In this embodiment, only simple deformation due to affine is dealt with, but any other known image conversion method or correction method, for example, optical distortion or the like may be corrected, which is limited to the method described above. is not.

<< Step S370 >>
In step S370, the panorama processing unit 120 determines whether or not alignment (pasting) has been performed on all the images selected in step S310. As a result of this determination, if all the images selected in step S310 have not yet been aligned (bonded) (S370 / No), the process returns to step S340, and the next image to be aligned And Steps S340 to S360 are performed.

  On the other hand, as a result of the determination in step S370, if all the images selected in step S310 have been aligned (bonded), the process proceeds to step S380.

<< Step S380 >>
Step S380 is a step of performing rematching on the image that has not been matched in step S340.
Specifically, in step S380, the panorama processing unit 120 reads out the unmatched image information stored in the panorama information storage unit 130 in step S340, executes the same processing as the steps S340 to S360, Perform alignment again. If there is an image matched in this step, the panorama processing unit 120 stores the affine coefficient and the final panorama image in the panorama information storage unit 130. Further, the panorama processing unit 120 regards an image that does not match in this process as a non-conforming image, and does not paste the panorama image into the panorama image, and only the affine value at the closest correlation is stored in the panorama information storage unit 130. save.

<Panorama image correction procedure>
Next, with reference to the UI screen of FIG. 4 and the flowchart of FIG. 5, the panorama image correction processing procedure and the user operation in the image generation apparatus 100 according to the present embodiment will be described.

  The panorama image output unit 140 reads the panorama image generated by the panorama processing shown in FIG. 3 from the panorama information storage unit 130 and displays it on the display unit 150. The image generation apparatus 100 is a utility that provides a user with a function that can correct a generated panoramic image by inputting correction content (first change content) designated by the user from the display unit 150 and the user interface unit 160. Software is provided.

FIG. 4 is a diagram illustrating an example of a UI screen of utility software displayed on the display unit 150 of FIGS. 1 and 2 according to the first embodiment of this invention.
The generated panoramic image is displayed in the layer display area 920 on the UI screen 910 shown in FIG. In the present embodiment, the above-mentioned nonconforming image 940 is displayed side by side on the left side of the application. These non-conforming images 940 may be displayed so that they can be recognized as non-conforming images by changing the color of the frame after affine deformation and pasting to the position with the highest correlation. In the example shown in FIG. 4, the command bar 930 of the utility software has five commands corresponding to the types of correction that can be specified by the user. The correction contents of the five commands are as follows.
・ Move: Change the position of a single image.
Change: Replaces a single image with another image.
Frame: Change the frame number used for the panorama image.
Front / back: In a region where single images overlap each other, the single image displayed on the front surface is replaced.
Contrast ... Adjusts contrast.

  When any of these commands in the command bar 930 is selected, detailed parameters necessary for correction can be input from the parameter setting unit 950 in accordance with the selected command. In order to start the correction process with the input parameters, an “Enter” button located below in the parameter setting unit 950 in FIG. 4 is pressed.

Next, the flow of correcting a panoramic image will be described using the flowchart of FIG.
FIG. 5 is a flowchart illustrating an example of a processing procedure of panorama image correction processing performed in the image generation apparatus 100 according to the first embodiment of the present invention.

<< Step S510 >>
Step S510 is a step of accepting the type of correction for the panoramic image designated by the user.
Specifically, when there is content that the user wants to check and correct the panoramic image displayed in the layer display area 920, when the user selects a desired correction type from the command bar 930 via the user interface unit 160, the step In S510, for example, the panorama information correction unit 170 receives a command related to the selection.

<< Step S520 >>
In step S520, for example, the panorama information correction unit 170 determines whether a single image to be corrected (change target) has already been selected by the user via the user interface unit 160. Here, for example, the user can select the correction target image by clicking the single image 921 using the mouse of the user interface unit 160 from the panorama image displayed in the layer display area 920. In the present embodiment, as the single image selected here, at least one of a fundus image constituting a panoramic image or a fundus image that can be constituted can be applied. In addition, the user interface unit 160 that selects a single image to be corrected (changed) constitutes a selection unit.

As a result of the determination in step S520, if the single image to be corrected has already been selected by the user (S520 / Yes), the process proceeds to step S540.
On the other hand, as a result of the determination in step S520, if the single image to be corrected has not yet been selected by the user (S520 / No), the process proceeds to step S530.
This selection process may be performed before the step of selecting the type of correction in step S510. In this case, the target image has been selected and the process proceeds to step S540.

<< Step S530 >>
In step S530, the image generation apparatus 100 (for example, the panorama information correction unit 170) displays a message prompting the user to select a correction target image on the display unit 150, and accepts the selection of the correction target image from the user. When the user selects a correction target image via the user interface unit 160, the process proceeds to step S540.

<< Step S540 >>
In step S540, the image generation apparatus 100 (for example, the panorama information correction unit 170) displays a detailed correction parameter corresponding to the selected single image and the type of correction on the parameter setting unit 950, and through the user interface unit 160. Accept user input.

  Here, the correction content by the user received in step S510 and step S540 constitutes the first change content.

<< Step S550 >>
In step S550, the panorama information correction unit 170 performs contrast adjustment, correlation calculation, affine deformation, and the like according to the correction parameter received in step S540, performs correlation recalculation of single images, and the panorama information storage unit 130. Correct and update the panorama information stored in. Specifically, in this step, the panorama information correction unit 170 performs correction on at least one of the selected correction target image and the panorama image according to the correction content (first change content) by the user. The contents (second change contents) are obtained by calculation. The panorama information correction unit 170 that calculates the second change content by calculation constitutes a first calculation means.

Thereafter, the panorama processing unit 120 generates a new panorama image based on the corrected panorama information, and stores it in the panorama information storage unit 130. The panorama processing unit 120 that generates a new panorama image constitutes a generation unit.
Thereafter, the panorama image output unit 140 reads the panorama image newly generated by the panorama processing unit 120 from the panorama information storage unit 130, and outputs the panorama image to the display unit 150 for display.

  Then, the image generating apparatus 100 repeatedly performs the processing of steps S510 to S540 for each item that the user determines to be corrected.

  Next, the details of these processing procedures will be described for each type of correction.

[Move; move selected image]
First, a process when the type of correction is “move” will be described. Specifically, when the type of correction is “move”, a process of relative position movement between the selected fundus image and the panoramic image is performed.
When “move” in the command bar 930 is selected in step S510 in FIG. 5, the user clicks on the panoramic image displayed in the layer display area 920 or the image to be moved by clicking the non-conforming image 940 with the mouse. select. Next, the user designates a movement destination by dragging and dropping an image with a click selected. Here, for example, the image generating apparatus 100 may be configured to drag the single image being dragged so that the match between the selected single image and the panoramic image of the background can be easily understood until the relative position movement is designated by the user. Is displayed on the display unit 150 so that the background is transparent. Also, the destination designation method is such that the grid coordinates are displayed on the layer display area 920 to define the X and Y coordinates, and then the destination coordinates (X, Y) of the image selected by the user from the parameter setting unit 950 are displayed. ) May be used.

  When the position is determined, the panorama information correcting unit 170 performs correlation calculation between the panoramic image of the background and the moved single image at the position, and determines an accurate position and an affine coefficient. These processes are the same as steps S340 to S360 of FIG. 3 performed by the panorama processing unit 120, but the search range in step S340 is limited to a narrow range because there is a rough position designation by the user. It is possible to increase the speed, and in this embodiment, the range is a half of the angle of view of the single image to be moved. The start of the correlation calculation process may be triggered by a drop of an image with a mouse, or may be triggered by pressing an “Enter” button in the parameter setting unit 950 after the drop.

  Next, a function for simultaneously moving a plurality of images will be described with respect to the image moving process. In a panoramic image with images with different angles of view, a small angle of view image is basically a higher resolution image with a higher pixel density, so a small angle of view image is pasted on a wide angle of view image. Combined. Therefore, when the movement of the wide angle image is instructed, it is desirable that the small angle image pasted thereon can be moved simultaneously. Here, when selecting an image to be moved, it is possible to select a plurality of images by clicking the mouse while holding down the Control key of the keyboard, which is a general means for selecting a plurality of images. It is prepared. In addition, when a wide-angle image is selected and moved, a setting may be provided in which the small-angle image pasted on the movement-target wide-angle image is also moved at the same time. In this case, correlation calculation after movement can be limited between the panoramic image of the background and the wide-angle image, and the calculation process between the wide-angle image and the small-angle image is omitted. Can be speeded up. Information necessary for correcting the positional relationship between the wide angle image and the small angle image is read from the panorama information storage unit 130, and the affine coefficient changed by the correction process of the panorama information correction unit 170 is the panorama information. The storage unit 130 is overwritten. Further, the order of determining the front / back of single images determined in step S340 in FIG. 3 is also read from the panorama information storage unit 130. When a new overlapping portion of single images occurs during image movement, it is determined which image is displayed on the front according to this order.

  As described above, the image generation apparatus 100 can perform processing for recalculating the correlation and determining the affine coefficient according to the correction content based on the rough movement instruction typified by the user's mouse operation.

[Change: Replace selected image]
Next, processing when the type of correction is “change” will be described.
When “change” in the command bar 930 is selected in step S510 in FIG. 5, the user selects a single image on the panoramic image to be replaced displayed in the layer display area 920 by clicking with the mouse. Then, the panorama information correcting unit 170 reads the reference position information (presentation position information of the fixation target) of the selected single image from the single image storage unit 110, and at the same time, the same reference position information and image as the selected single image. Search for other single images with corners. If there is no such image, the image generating apparatus 100 displays a message to the effect that there is no replacement target image on the display unit 150 and presents it to the user, and ends the process.

  On the other hand, if there is a replacement target image, the panorama information correction unit 170 performs the same contrast adjustment as in step S320 on the replacement target image and stores the same in the panorama information storage unit 130. Thereafter, the parameter setting unit 950 of the utility software changes to the display of FIG. 6, and the panorama information correction unit 170 performs image selection, frame selection, and averaged image generation by the user. Here, it is assumed that there are three types of replaceable images. Each image is assigned to the buttons Image1 to Image3 of the parameter setting unit 950 shown in FIG. 6, and the image on the panoramic image is replaced by pressing any button. Further, the frame number is directly set in the “Frame No.” box of the mouse wheel operation or the parameter setting unit 950 shown in FIG. To display the frame number to be displayed. Can be changed. Here, since the position of the image at the same reference position, that is, the same fixation target presenting position may be different due to fixation fixation fine movement or the like, the processing of moving the selected image described above in parallel with the frame selection May also be made possible.

  Further, when the user presses the “ave” button of the parameter setting unit 950 shown in FIG. 6, the panorama information correcting unit 170 generates an averaged image by selecting all the frames or only good images, and generates a panoramic image. Replace with the image above. When the image to be replaced is determined, when the user presses the “Enter” button of the parameter setting unit 950 shown in FIG. 6, the panorama information correcting unit 170 replaces the background panorama image with the replaced single image. Perform correlation calculations to determine exact position and affine coefficients. These processes are the same as steps S340 to S360 of FIG.

  If there is an image with the same angle of view or an image with a smaller angle of view in front of the single image replaced on the panorama image, the front / back of the single images stored in the panorama information storage unit 130 is determined. With reference to the order, the single image in front of the replaced image is sequentially reprocessed in steps S340 to S360 in FIG. 3 according to this order. Then, the affine coefficient changed by the series of correction processes, the selected image ID, and the frame number are overwritten and saved in the panorama information saving unit 130. In this process, for example, when the background single image is limited to one, such as the single image 921 with a small angle of view in FIG. 4, based on the accurate position adjustment of the background single image and the affine coefficient. The calculation process may be reduced by adjusting the image 921.

  In this manner, the image generation apparatus 100 performs complicated processing such as re-correlation re-calculation that occurs in response to a user's image replacement instruction, affine coefficient determination, and change of affine coefficients of other single images affected by replacement. Can bear.

[Frame; Frame No. Change]
Next, processing when the type of correction is “frame” will be described.
When “frame” in the command bar 930 is selected in step S510 in FIG. Select a single image to change the panorama image. Then, the panorama information correction unit 170 reads other frame images of the selected image from the single image storage unit 110, performs the same contrast adjustment as in step S320, and stores it in the panorama information storage unit 130. Then, the parameter setting section 950 of the utility software changes to the display of FIG. 7, and the frame number is directly displayed in the “Frame No.” box of the parameter setting section 950 shown in FIG. To display the frame number to be displayed. Can be changed. The subsequent processing is the same as the processing for replacing the selected image described above.

[Front / back; front / back replacement]
Next, processing when the type of correction is “front / back” will be described. Specifically, when the type of correction is “front / back”, processing for changing the order in which at least one fundus image constituting the panoramic image is pasted is performed in the process of generating the panoramic image.
When “front / back” in the command bar 930 is selected in step S510 in FIG. 5, the user selects a single image for front / back movement from the panoramic image. When there are two selected single images, the parameter setting section 950 of the utility software changes to the display of FIG. 8A, and each time the “front⇔back” button is pressed, the order of the two selected layers in the panoramic image Will be replaced. Press the button again to return.

  If only one single image is selected, the parameter setting unit 950 of the utility software is changed to the display shown in FIG. Each of the four operation buttons shown in FIG. 8B represents processing for the selected image. For example, when the “Send Backward” button is selected, the operation button is moved one layer back.

  These processes may be assigned to the mouse wheel to enable the layer movement of the selected image by the wheel. In both cases shown in FIGS. 8A and 8B, when the layer movement (exchange) is determined, the “Enter” button is pressed, so that the panorama information correcting unit 170 is the automatic processing in the subsequent stage. The correlation between the panoramic image of the background and the image whose layer has been moved is calculated, and an accurate position and affine coefficient are determined. At this time, the panorama information correction unit 170 rewrites the order of determining the front / back of the single images stored in the panorama information storage unit 130 according to the layer movement performed. Furthermore, the panorama information correcting unit 170 performs the reprocessing of S340 to S360 shown in FIG. 3 in order of the single images in front of the images related to the layer movement in this order. Then, the panorama information correction unit 170 overwrites and saves the affine coefficient changed by the series of correction processes in the panorama information storage unit 130. Again, if the background image is limited to a single image, the processing may be reduced by using a modification of the background image.

  As described above, the image generation apparatus 100 performs complicated processing such as re-correlation re-calculation, affine coefficient determination, and change of affine coefficients of other single images that are affected by moving the layer of the image selected by the user. It is possible to generate a new panoramic image. In the present embodiment, the panorama information correction unit 170 takes into account not only the order of the fundus images to be combined when the panorama image is generated, but also information on at least one of the presence or absence of overlapping of the fundus images. In addition, at least one of the order of adding the second change content or the selection of the fundus image affected by the first change content may be determined.

[Contrast; Contrast adjustment]
Next, a process when the type of correction is “contrast” will be described.
When “contrast” in the command bar 930 is selected in step S510 in FIG. 5, the user selects an image for contrast adjustment from the panoramic image. Then, the parameter setting unit 950 of the utility software is changed to the display of FIG. When the user manually adjusts the brightness, contrast, and gamma value of the selected single image, correction values such as “+10” and “−15” are set in each input BOX of the parameter setting unit 950 shown in FIG. input. The correction is immediately reflected in the panoramic image, and the user adjusts the correction value while viewing the state of the panoramic image. When the correction value is determined, the user presses the “Enter” button shown in FIG. 9 to change the contrast. Determine. With this operation, the panorama information correction unit 170 overwrites and saves the new single image data reflecting the contrast adjustment in the panorama information storage unit 130. In addition to manual adjustment, a semi-automatic contrast adjustment function is also provided. By pressing an “Auto” button shown in FIG. 9, the mode is switched to the semi-automatic mode.

  In the adaptive optics SLO device, particularly in the case of a diseased eye or the like, a part of the upper part or the right end of the image taken due to the state of the pupil becomes dark, and an image of a luminance distribution with gradation as a whole is obtained. May be acquired. Then, for example, as illustrated in FIG. 10A, the boundary of the panoramic image may be noticeable due to the gradation distribution with gradation. The semi-automatic (contrast) adjustment function here shows an example of correction means for the purpose of correcting these luminance gradations. However, the present embodiment is not limited to this.

  In the semi-automatic mode, the user can mark a portion where the boundary is conspicuous in the pasted portion due to the luminance difference for each single image using the mouse. Further, with respect to the region of the single image that appears to have gradation on the luminance, an arrow 960 indicating the gradation direction arranged in the parameter setting unit 950 shown in FIG. Adjust the rotation so that it is parallel to the direction. These operations are repeated to mark a noticeable boundary and a gradation region due to a luminance difference, and finally, for example, a white broken line 1010 and an arrow 1020 in FIG. In this state, the automatic correction is started by pressing the “Enter” button of the parameter setting unit 950 shown in FIG. The panorama information correcting unit 170 performs automatic correction for each single image on which the arrow 1020 is arranged, and uses the luminance difference at the boundary indicated by the white broken line 1010 as a boundary condition, in the direction indicated by the arrow 1020. Tone correction is performed by increasing or decreasing luminance monotonously. Then, the panorama information correction unit 170 performs boundary blending processing after tone correction, and then overwrites and saves the new single image data reflecting the contrast adjustment in the panorama information storage unit 130.

  As described above, the image generation apparatus 100 can perform complicated contrast adjustment in consideration of gradation according to a simple correction instruction from the user.

In the first embodiment, in accordance with the rough correction content that the user performs on the panorama image or the like, complicated adjustment such as high-definition additional correction or adjustment to the entire panorama image required due to the correction of the single image. The image generation apparatus 100 is responsible for processing the correct correction contents.
According to such a configuration, when correcting a panoramic image obtained by pasting together a plurality of fundus images (single images), it is possible to reduce the workload of the user who performs the correction and improve the accuracy of the correction work. Can do.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.

  In the second embodiment, correction candidates are further presented to the user with respect to the image generation apparatus according to the first embodiment that corrects a panoramic image obtained by synthesizing the fundus image captured by the adaptive optics SLO device. In this mode, the user's correction work load is further reduced. Here, as the correction candidates in the present embodiment, for example, the relative position candidates of the selected fundus image and the panoramic image, the first change content described in the first embodiment, or the second change. Candidates for order to add content can be applied. The configuration of the image generation apparatus according to the second embodiment and the processing procedure of the panorama processing are the same as those in the first embodiment. In the following, a correction candidate presentation function added to the panorama image correction procedure will be described in particular.

  As processing of the second embodiment, when the user selects a single image to be corrected and selects a correction type (command), the image generation apparatus 100 narrows down correction candidates by rough conditions or calculation, and then prompts the user. Present. When the correction candidate determined to be correct by the user is selected, the image generation apparatus 100 further performs high-definition processing to determine the correction content for the panoramic image. Thus, in the correction by the user, the image generation apparatus 100 narrows down the correction candidates in advance and presents them to the user, so that the user's correction work can be further simplified.

<Panorama image correction procedure>
Hereinafter, with reference to the UI screen in FIG. 11 and the flowchart in FIG. 12, the panorama image correction processing procedure and the user operation in the image generation apparatus 100 according to the present embodiment will be described.

  FIG. 11 is a diagram illustrating an example of a utility software UI screen displayed on the display unit 150 of FIGS. 1 and 2 according to the second embodiment of this invention. In FIG. 11, the same components as those shown in FIG.

  The UI screen 910 shown in FIG. 11 is the same as the UI screen 910 according to the first embodiment shown in FIG. 4 except that an “Auto Advance” button 951 is further arranged in the parameter setting unit 950.

When the “Auto Advance” button 951 shown in FIG. 11 is pressed, the utility software switches to a mode for presenting correction candidates, and the button 951 changes color to display that mode. When the button 951 is pressed again, the mode returns to the normal mode, and the utility software performs the same operation as in the first embodiment. In the present embodiment, a mode in which correction candidates are presented in the following three commands among the five commands shown in the command bar 930 is effective. The remaining two commands can be used with the same operation specifications as in the first embodiment regardless of whether the “Auto Advance” button 951 is pressed.
・ Move: Change the position of a single image.
Change: Replaces a single image with another image.
Frame: Change the frame number used for the panorama image.

  When one of these three commands is selected when the “Auto Advance” button 951 is valid, a correction candidate is presented according to the single image to be corrected selected by the user. In order to determine the candidate selected by the user and start the correction process of the panoramic image, the user presses the “Enter” button located below in the parameter setting unit 950 of FIG.

Next, the flow of correcting a panoramic image will be described using the flowchart of FIG.
FIG. 12 is a flowchart illustrating an example of a processing procedure of panorama image correction processing performed in the image generation device 100 according to the second embodiment of the present invention. In FIG. 12, process steps similar to the process steps shown in FIG. 5 are given the same step numbers. That is, the processing from step S510 to S530 is the same as the processing in the first embodiment shown in FIG.

<< Step S1210 >>
In step S1210, the panorama information correction unit 170 calculates a correction candidate for the selected single image and the correction content (first change content) related to the type of correction specified in step S510. At this time, the panorama information correcting unit 170 may obtain correction candidates by calculation based on imaging conditions of at least one fundus image of the fundus image constituting the panoramic image or the fundus image that can be formed. The panorama information correction unit 170 that calculates the correction candidates by calculation constitutes a second calculation means.
Thereafter, the display unit 150 displays the correction candidates obtained by the calculation in the panorama information correction unit 170 in the layer display area 920 and presents them to the user. The display unit 150 that presents correction candidates to the user constitutes a presentation unit. Here, use is made of correlation calculation between background images and single images to be corrected, images of other frames that are candidates for correction, other unused images stored in the single image storage unit 110, and the like. Then, a correction candidate is presented or selected from a plurality of candidates.

<< Step S1220 >>
Step S1220 is a step in which the image generating apparatus 100 receives an input in the process of searching for various conditions when the user determines a correction candidate.
Specifically, in step S1220, image generation apparatus 100 (for example, panorama information correction unit 170) determines whether or not there has been an instruction for additional input from the user. For example, the input here includes rough current position information of a single image being moved by dragging the mouse, input of frame selection, image selection, and the like. Since the selection conditions for the correction candidates change depending on these inputs, if there is an instruction for additional input from the user (S1220 / Yes), the process returns to step S1210, and the correction candidates are selected again and presented to the user.
On the other hand, as a result of the determination in step S1220, when there is no instruction for additional input from the user (S1220 / No), the process proceeds to step S1230.

<< Step S1230 >>
In step S1230, the panorama information correcting unit 170 determines a single correction target single image and detailed correction parameters corresponding to the type of correction when the user presses the “Enter” button of the parameter setting unit 950. At this time, when any one of the presented correction candidates is selected by the user, the panorama information correction unit 170 corrects the original correction content (first change content) based on the selected correction candidate. Replace with (first modification). The panorama information correction unit 170 that performs the replacement constitutes a replacement unit.

<< Step S1240 >>
Subsequently, in step S1240, the panorama information correction unit 170 performs contrast adjustment, correlation calculation, affine deformation, and the like according to the correction parameter received and determined in step S1230, and performs recalculation of correlation between single images. The panorama information stored in the panorama information storage unit 130 is corrected and updated. Thereafter, the panorama processing unit 120 generates a new panorama image based on the corrected panorama information, and stores it in the panorama information storage unit 130. Thereafter, the panorama image output unit 140 reads the panorama image newly generated by the panorama processing unit 120 from the panorama information storage unit 130, and outputs the panorama image to the display unit 150 for display.

  Then, the image generating apparatus 100 repeatedly performs the processes of steps S510 to S530 and S1210 to S1240 for each item that the user determines to be corrected.

  Next, details of these processing procedures will be described for each type of correction. Note that description of the same parts as those of the first embodiment is omitted.

[Move; move selected image]
First, a process when the type of correction is “move” will be described.
In the present embodiment, there are two functions as a destination presentation function for moving the image pasting position.

The first function is a destination candidate presentation function.
The panorama information correcting unit 170 selects the panoramic image for the position where the selected single image was originally pasted, or the temporary position determined by dragging the mouse or specifying coordinates through the parameter setting unit 950. Calculate the correlation with a single image. Since the correlation calculation here requires high-speed processing in order to respond to the instruction that changes every moment by dragging the mouse, the area to be subjected to correlation calculation in the panoramic image is centered on the selected single image. This is limited to a field angle range that is three times the field angle of the selected image. Further, the correlation calculation is performed only with the same rough calculation as the method in step S340 of FIG. 3, and the calculation for high-precision alignment in step S350 of FIG. 3 and S360 of FIG. Yes. That is, in the present embodiment, the amount of calculation performed for one registration between images is a correction candidate in the present embodiment rather than the calculation for determining the second change content described in the first embodiment. Less computation. Then, as a result of the correlation calculation, two or three positions with high correlation as candidate positions for pasting are displayed and presented to the user.

  FIG. 13 is a diagram illustrating an example of the single image 200 being dragged by the mouse according to the second embodiment of this invention. In FIG. 13 (a), two small areas 210-1 and 210-2 surrounded by broken lines are positions presented as destination candidates with high correlation, and a large range 220 surrounded by broken lines is a correlation calculation. In this case, the range of the panoramic image (correlation calculation range) limited to the area is shown. The latter frame (220) need not be presented to the user. In addition, during the processing, the single image frame lines in the layer display area 920 shown in FIG. 11 are erased and displayed so that the dragged image and the frame of the movement destination candidate can be easily understood. FIG. 13B shows a diagram when the mouse is dragged and the single image 200 is moved further than in the case of FIG. As the single image 200 moves, the correlation calculation range 220 also moves. As a result, the candidate frame region 210-1 in FIG. 13A is excluded from the candidates, and instead the candidate frame region 210-3 is newly added. Is presented as a good candidate. The user can select what seems to be correct from the presented candidate frames, drag the selected images over there, and drop them if they match. In the present embodiment, only the dotted frame is displayed as the candidate position. However, a copy of the selected image may be displayed overlapping each candidate position. When the position is determined, the subsequent processing is the same as in the first embodiment, and accurate alignment is performed by high-precision correlation calculation in step S350 in FIG. 3 and affine coefficient determination in step S360 in FIG. Panorama image regeneration is performed.

The second function is a correction order presentation function.
When correcting the position of multiple single images in sequence, each time an image is moved, the single image (layer) that is pasted above the moved image is affected by the correction of the lower image. Correlation calculation and affine coefficient recalculation are required. Therefore, the order of pasting determined when the panoramic image is generated, that is, the order of determining the front / back of the single images stored in the panorama information storage unit 130 is used, and the user can perform processing according to this order. Prompt. When “move” in the command bar 930 shown in FIG. 11 is selected in a state where a plurality of single images are selected, the image generation apparatus 100 according to the present embodiment first determines whether or not to perform the same movement process on all images. The user is prompted to select a dialog or the like. If it is a batch move, the process proceeds, but if not, the single image that is the backmost layer among the selected images is displayed in the foreground, and only the image behind it is displayed. Is displayed as a panorama.

  FIG. 14 is a diagram illustrating an example of an image displayed in the layer display area 920 of the UI screen 910 illustrated in FIG. 11 according to the second embodiment of this invention. In FIG. 14A, it is assumed that the medium angle image 310 and the small angle image 320 near the center are selected as the single images to be moved. Then, since the middle angle image 310 is the backmost image, the display prompts the user to move the middle angle image 310 shown in FIG. 14B, and the single image that is in front of the middle angle image 310 is temporarily displayed. Disappear. When the position of the medium angle image 310 is determined in this state, the intermediate angle image 310 and the small angle image 320 are added to the intermediate angle image 310 and the background panorama image in addition to the reprocessing of S340 to S360 in FIG. Reprocessing of S340 to S360 in FIG. 3 is executed for the other images in the intermediate layer in the order of the layers. Next, the display of FIG. 14C is performed, and the position of the small angle-of-view image 320 is aligned. In the subsequent processing, after the position of the small angle image 320 is determined in the same manner, the small angle image 320 and the image in front of the small angle image 320 are reprocessed to regenerate the panorama image.

  In this way, the workload of editing panoramic images can be simplified by presenting the destination of the image and simplifying the user's selection, or by specifying the order of position correction and minimizing the calculation wait time due to reprocessing. And can save time.

[Frame; Frame No. Change]
Next, processing when the type of correction is “frame” will be described.
As described in the first embodiment, even if the continuous frame images are captured at the same fixation target presentation position, the frame No. Depending on the location, it may be different. For example, the frame number is directly entered in the “Frame No.” box of the mouse wheel operation or the parameter setting unit 950 shown in FIG. To display the frame number to be displayed. If this is done, there is a need for position correction. In the first embodiment, manual position correction is possible. In this embodiment, however, the correlation between the background image and a new frame is calculated every time the frame is changed, and the correlation is displayed at a position with high correlation. Even in this case, since high-speed processing is required in order to respond to an instruction that changes every moment due to a mouse wheel operation or the like, the region to be subjected to correlation calculation in the panoramic image is limited, and the correlation calculation is also performed in the step of FIG. It is desirable to use only the same rough calculation as the method in S340.

  As a result, it becomes possible to select a frame while automatically correcting misalignment caused by the frame change, and the work load of frame selection can be reduced.

<Change; replacement of selected image>
Next, processing when the type of correction is “change” will be described.
Even in the replacement of the selected image, there is a possibility that the imaging position may be different even if the images have the same reference position information (fixation target presenting position) due to the influence of the fixation fixation as described above. The imaging position can change every time. Therefore, it is useful to visually know at which position each candidate image to be replaced is an image captured at the time of selecting a replacement candidate.

  In the parameter setting unit 950 illustrated in FIG. 6, for example, “Image2” is selected as a candidate image to be replaced. Then, the panorama information correcting unit 170 performs a rough correlation calculation with the background image for all the frames to calculate an approximate position, and then presents a range where there is an image of at least one frame. If there are a large number of frames, it is expected that this processing will take some time even for area limitation or coarse correlation calculation. It is desirable to store in the storage unit 130. Further, since it is troublesome to display all the frames in the layer display area 920, only the frame selected by the “Frame No.” of the parameter setting unit 950 shown in FIG. 6 or the mouse wheel is displayed at the position obtained by the correlation calculation. FIG. 15 shows a layer display area 920 in which they are displayed.

FIG. 15 is a diagram illustrating an example of an image displayed in the layer display area 920 of the UI screen 910 illustrated in FIG. 11 according to the second embodiment of this invention.
In FIG. 15, a single image 400 surrounded by a thick frame line is an image candidate after replacement, and one frame No. with “Image2” is displayed. It is an image. A broken line 410 indicates a range in which at least one image of all the frames of “Image2” is present. By performing such display, it is possible to immediately determine whether or not an imaging frame at a position desired by the user is included.

  Note that when there are a large number of images of the same reference position information, the operation of the user selecting a replacement candidate becomes complicated. In such a case, a parameter related to the focus position is extracted from information incidental to the single image, and an average value of all the single images constituting the panoramic image is calculated. By selectively or preferentially presenting images picked up at a focus position close to the calculated average value, the user quickly selects an appropriate replacement target image from among many image candidates. be able to. Since the fundus image (retinal image) captured by the adaptive optics SLO device has a shallow depth of focus, the image captured differs depending on the focus position, and it is required to select an image with a close focus position when generating a panoramic image.

  According to the second embodiment, when the user makes corrections to the panoramic image, the image generation apparatus 100 narrows down and presents correction candidates in advance. In addition to the effects of the first embodiment, The work burden on the user who makes the correction can be further reduced.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
This program and a computer-readable storage medium storing the program are included in the present invention.

  Note that the above-described embodiments of the present invention are merely examples of implementation in practicing the present invention, and the technical scope of the present invention should not be construed as being limited thereto. It is. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 100 Image production | generation apparatus, 110 Single image memory | storage part, 120 Panorama processing part, 130 Panorama information storage part, 140 Panorama image output part, 150 Display part, 160 User interface part, 170 Panorama information correction part

Claims (18)

An image generation device that generates a panoramic image by pasting together a plurality of fundus images,
Selection means for selecting a fundus image constituting the panoramic image or at least one fundus image to be changed among the configurable fundus images;
A first operation performed on at least one of the fundus image selected by the selection unit and the panoramic image according to the first change content designated by the user with respect to the fundus image selected by the selection unit. First calculating means for calculating the change contents of 2 by calculation;
An image generation apparatus comprising: generation means for generating a new panoramic image based on the second change content.   The image generation apparatus according to claim 1, wherein the first change content is a relative position shift between the fundus image selected by the selection unit and the panorama image.   The image generating apparatus according to claim 2, wherein the fundus image selected by the selection unit is displayed in a translucent manner until the relative position movement is designated as the first change content.   2. The first change content is a change in which at least one fundus image constituting the panoramic image is replaced with another fundus image capable of constituting the panorama image. Image generation device.   2. The image according to claim 1, wherein the first change content is a change in an order in which at least one fundus image constituting the panorama image is pasted in the process of generating the panorama image. Generator.   The image generation apparatus according to claim 1, wherein the first change content is a change in a contrast of at least one fundus image constituting the panoramic image.   When the first calculation unit obtains the second change content by calculation, if there is another fundus image pasted on the fundus image selected by the selection unit, the other calculation unit 7. The image generation apparatus according to claim 1, further comprising: a content to which the same change as the change to the fundus image selected by the selection unit is added to the fundus image. The first calculation means selects a fundus image affected by the first change contents from the fundus images constituting the panoramic image,
The image according to any one of claims 1 to 7, wherein the generation unit generates the new panoramic image by adding the second change content to the affected fundus image. Generator.   The first calculation means uses the information on at least one of the order of pasting of the fundus images when the panoramic image is generated or the presence / absence of overlapping of the fundus images, and the order of adding the second change contents The image generating apparatus according to claim 8, wherein at least one of the selection of the affected fundus image is determined.   The image according to any one of claims 1 to 9, wherein the first calculation means includes a change for performing alignment between at least two images as the first change content. Generator.   The image generation apparatus according to claim 10, wherein the registration method between images calculates a position having the highest correlation by using a normalized correlation coefficient. Second calculating means for calculating correction candidates for the first change content by calculation;
Presenting means for presenting the correction candidates;
Replacement means for causing the user to select any one of the correction candidates presented by the presenting means and replacing the first change content with the first correction content;
Further comprising
The image generation apparatus according to claim 1, wherein the first calculation unit calculates the second change content by calculation according to the first correction content.   The second calculation means includes, as the first correction content, an amount of calculation performed for one registration between images, including a correction that performs alignment between at least two images. 13. The image generation apparatus according to claim 12, wherein the number of calculations performed by the second calculation unit is less than the calculation performed by the first calculation unit.   The image generation apparatus according to claim 12 or 13, wherein the correction candidate is a candidate for a relative position between the fundus image selected by the selection unit and the panoramic image.   The second calculation means calculates the correction candidates by calculation based on photographing conditions of at least one fundus image of the fundus image constituting the panoramic image or the fundus image that can be formed. Item 15. The image generation device according to any one of Items 12 to 14.   The image generation apparatus according to claim 12, wherein the correction candidate is a candidate for an order of adding the first change content or the second change content. An image generation method by an image generation device that generates a panoramic image by pasting together a plurality of fundus images,
A selection step of selecting a fundus image constituting the panoramic image or at least one fundus image to be modified of the configurable fundus image;
A first operation performed on at least one of the fundus image selected in the selection step and the panoramic image according to the first change content designated by the user with respect to the fundus image selected in the selection step. A first calculation step for calculating the change contents of 2 by calculation;
And a generation step of generating a new panoramic image based on the second change content.   The program for functioning a computer as each means in the image generation apparatus of any one of Claims 1 thru | or 16.