JP2003087550A - Device and method for compositing image and computer readable recording medium having image composite processing program recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image compositing device by which work efficiency for arranging the positions of adjacent images is improved in the case a user arranges the positions of the images manually so as to overlap the overlapping parts of the images in each pair of adjacent images. SOLUTION: This image compositing device is provided with a first means for arranging the positions of the adjacent images manually so as to overlap the overlapping parts of the images in each pair of adjacent images selected out of a plurality of images arranged two-dimensionally on a monitor screen, and a second means for combining the whole images on the basis of results of arranging the positions of the images on the first means. The first means is provided with a means for displaying two images constituting a pair of adjacent images on the monitor screen, a means for displaying the relative positional relation of the two images on the monitor screen in a pattern, and a means for moving one of the two images so as to overlap the overlapping parts of the two images displayed on the monitor screen.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to, for example, an image synthesizing apparatus, an image synthesizing method for automatically synthesizing images from a plurality of microscope images to obtain a high-resolution and wide-field synthetic image. The present invention relates to a computer-readable recording medium recording an image composition program.

[0002]

2. Description of the Related Art [1] Description of conventional optical flow calculation method

A technique is known in which an optical flow is calculated from two images and the alignment between the two images is performed based on the obtained optical flow. A conventional optical flow calculation method will be described.

(1) Lucas-Kanade method A number of methods have conventionally been proposed for calculating the apparent velocity field (optical flow) of a moving object in a moving image. Among them, the Lucas-Kanade method, which is a local gradient method,
This is one of the best methods. The reason is that processing is fast,
It is easy to implement and the result is reliable.

For details of the Lucas-Kanade method, see the literature:
B. Lucas and T. Kanade, "An Iterative Image Registra
tion Technique with an Application to Stereo Visio
n, "In Seventh International Joint Conference on Ar
tificial Intelligence (IJCAI-81), pp. 674-979, 1981
checking ...

The outline of the Lucas-Kanade method will be described below.
Grayscale pattern I at image coordinate p = (x, y) at a certain time t
(X, y, t) has a coordinate (x
+ Δx, y + δy), the following optical flow constraint equation 1 holds when the grayscale distribution is moved while being kept constant.

[0007]

[Equation 1]

Optical flow {v =
In order to calculate (δx / δt, δy / δt) = (u, v)}, since there are two unknown parameters, another constraint equation is necessary. Lucas and Kanade hypothesized that local regions of the same object would have the same optical flow.

For example, if the optical flow is constant in the local area ω on the image, the square error E of the grayscale pattern to be minimized is I ₀ (p) = I (x, y, t), I _If rewritten as ₁ (p + v) = I (x + u, y + v, t + δt), it can be defined by the following expression 2.

[0010]

[Equation 2]

Here, when v is very small, terms of the second or higher order of the Taylor expansion can be ignored, so that the relation of the following expression 3 is established.

[0012]

[Equation 3]

Here, g (p) is the first derivative of I ₁ (p).

The error E is minimized when the differential value of E with respect to v is 0, and therefore the relation of the following equation 4 is established.

[0015]

[Equation 4]

Therefore, the optical flow v is calculated by the following equation 5.

[0017]

[Equation 5]

Further, as shown in the following equation 6, Newton ·
It can be obtained accurately by Rafson-like iterative calculation.

[0019]

[Equation 6]

(2) Hierarchical estimation method The biggest problem of the gradient method including the Lucas-Kanade method is that it cannot be applied to a large motion because a good initial value is required. Therefore, conventionally, there has been proposed a method in which images having different resolutions are created several times in a pyramid hierarchical structure type.

First, from two consecutive images,
Images having different resolutions of several levels are created in advance. Next, a rough optical flow is calculated between the images with the lowest resolution. Then, referring to this result, a more accurate optical flow is calculated between the images with higher resolution. This process is sequentially repeated until the images with the highest resolution.

4 is an original image, FIG. 3 is an image having a lower resolution than the original image of FIG. 4, FIG. 2 is an image having a lower resolution than the low resolution image of FIG. 3, and FIG. 1 is a low resolution image of FIG. The lower resolution images are shown respectively. 1 to 4, S indicates one patch.

The image of FIG. 1 (image of layer 1), the image of FIG. 2 (image of layer 2), the image of FIG. 3 (image of layer 3) and the image of FIG. 4 (image of layer 4) are staged in this order. Optical flow is required. 1 to 4, the arrow indicates
The optical flow vector obtained for each patch is shown.

However, the problem here is that in the actual image, there are few areas including a sufficient pattern (texture),
There is no reliable optical flow.

[2] Description of optical flow calculation method developed by the applicant.

The optical flow calculation method developed by the present applicant is premised on hierarchical estimation in which an image having different resolutions of several stages is created in a pyramid hierarchical form and the optical flow is calculated stepwise. The optical flow calculation method follows a gradient method such as the Lucas-Kanade method. In other words, it is premised on the optical flow estimation method by the hierarchical gradient method. Here, Lucas- is used as the gradient method.
The Kanade method is used.

A feature of the optical flow estimation method developed by the present applicant is that the optical flow obtained at each stage of the optical flow estimation method by the hierarchically structured Lucas-Kanade method is complemented by dilation processing. Hereinafter, this will be described in detail.

One of the advantages of the Lucas-Kanade method is that the tracking result is reliable. Tomasi and Kanade showed that the traceability of a region can be calculated from differential images as follows (C. Tomasi and T. Kanade, "Shape an
d Motion from Image Streams: a Factorization metho
d-Part 3 Detection and Tracking of Point Features
, "CMU-CS-91-132, Carnegie Mellon University, 199
1.).

The traceability of a certain area image ω is determined by calculating its eigenvalue from the 2 × 2 coefficient matrix G of the following equation 7 having squared differentials in the vertical and horizontal directions as elements. can do.

[0030]

[Equation 7]

When the eigenvalues of the matrix G are both large, the region has a change in the orthogonal direction and unique positioning is possible. Therefore, the smaller eigenvalue λ _min and
From the grayscale residual E between regions after tracking, the reliability γ of the tracking result
Can be obtained by the following equation 8.

[0032]

[Equation 8]

The present inventors have developed a method of interpolating a region of low reliability by using the result of high reliability within the same hierarchy of optical flow. This uses the result of the one-level coarse layer only for the initial value of the tracking, and does not use anything for the result of the current stage layer of interest. Instead, it is assumed that the optical flow in the region with few textures has a value close to the optical flow around it, and complements the flow field by morphological processing.

FIG. 5 shows the flow vector expansion processing.

The left figure shows the map of the reliability of the flow vector in shades. Here, the darker the black, the higher the reliability.

First, the obtained flow is thresholded. The white areas are thresholded because the results are less reliable.

Next, the expansion processing of the result is performed in the flow field as follows, simulating the filling processing by the morphological operation on the binary image. The flow vector u (i, j) of a certain region i, j can be calculated as in the following Expression 9 by weighting the flow vectors of the four neighborhoods according to the reliability γ.

[0038]

[Equation 9]

This process is repeated until all thresholded low reliability regions are filled. This complementing process is performed in each layer. Note that the flow vector u (i, j) of a certain region i, j may be calculated by weighting the flow vectors in the eight neighborhoods according to the reliability γ.

FIG. 6A shows an optical flow thresholded for an image of a certain layer.
(B) shows the optical flow after the complement. In FIG. 6A, an arrow indicates an optical flow vector whose reliability is determined to be high by the threshold processing, and a cross indicates a portion whose reliability is low.

[3] Description of Image Synthesis Technology

The applicant of the present invention has developed a technique (image synthesis software) for calculating a geometric conversion coefficient between images for a plurality of picked-up images and synthesizing them based on the calculated geometric conversion coefficient.

By the way, in the image synthesizing software developed by the present applicant, when a plurality of images photographed in a two-dimensional array are concatenated to generate one synthetic image, first, a plurality of photographed images are Two-dimensional arrangement is performed on the monitor screen so that the arrangement position matches the actual relative positional relationship. By using this arrangement position information (relative positional relationship between images), the adjacent image pair from which the overlapping portion of the image should be extracted is determined, and the overlapping portion of the image is extracted for each adjacent image. The feature points are associated with each adjacent image using the overlapping portion of the images extracted for each adjacent image. Using the feature points associated with each adjacent image, the geometric conversion coefficient is calculated for each adjacent image, and all the images are combined using the geometric conversion coefficient calculated for each adjacent image.

As a mode for extracting the overlapping portion of the images, an automatic mode and a manual mode are prepared. When manually extracting the overlapping portion of the images, the two images forming the adjacent image pair are displayed semitransparently on the monitor screen so that the overlapping portions of the two images displayed on the monitor screen overlap. One of the images is moved by the user. That is, the two images are manually aligned by the user so that the overlapping portions overlap.

However, the relative positional relationship between the adjacent image pairs displayed on the monitor screen is as follows.
Although there are various directions such as the diagonal direction, it is difficult for the user to grasp the relative positional relationship between the adjacent image pairs, so that there is a problem that the work efficiency of alignment is poor.

Also, as modes for associating feature points, there are prepared an automatic mode and a manual mode. When the feature points are associated manually, two images forming an adjacent image pair are displayed on the monitor screen, and the feature points corresponding to the two images are displayed on the two images displayed on the monitor screen. Is set by the user.

In this case as well, it is difficult for the user to grasp the relative positional relationship between adjacent image pairs, and there is the problem that the work efficiency for associating the feature points is poor.

By the way, when determining an adjacent image pair from which an overlapping portion of images is to be extracted from a plurality of two-dimensionally arranged images on the monitor screen, the plurality of two-dimensionally arranged images are adjacent to each other. It is necessary to make a one-dimensional array so that there is overlap between the images.

A method for one-dimensionally arranging a plurality of two-dimensionally arranged images is as follows:
It may be possible to predetermine the order for arranging a plurality of two-dimensionally arranged images in a one-dimensional arrangement. However, as a result of two-dimensional arrangement of images on the monitor screen, there is no image in the center. However, this method has a problem in that it is not possible to perform a one-dimensional array so that there is an overlap between adjacent images.

[0050]

SUMMARY OF THE INVENTION According to the present invention, in the case where the user manually positions the adjacent image pairs so that the overlapping portions of the images overlap each other, the work efficiency for the positioning is improved. An object of the present invention is to provide an image synthesizing device, an image synthesizing method, and a computer-readable recording medium in which an image synthesizing processing program is recorded, which can be improved.

According to the present invention, an image synthesizing apparatus, an image synthesizing method, and an image synthesizing method capable of improving the work efficiency for associating feature points when the user manually associates the feature points for each pair of adjacent images. An object of the present invention is to provide a computer-readable recording medium recording an image composition processing program.

According to the present invention, a plurality of two-dimensionally arranged images may overlap each other even when no images exist in the central portion as a result of the two-dimensional arrangement of the images on the monitor screen. Another object of the present invention is to provide a computer-readable recording medium in which an image synthesizing device, an image synthesizing method, and an image synthesizing processing program capable of one-dimensional arrangement are recorded.

[0053]

A first image synthesizing apparatus according to the present invention displays a plurality of images to be synthesized on a monitor screen so that the arrangement positions of the images match the actual relative positional relationship. In an image synthesizing device for synthesizing all images by arranging the images in two dimensions and using the arrangement position information of each image, for each adjacent image pair selected from a plurality of two-dimensionally arranged images on the monitor screen, A first means for allowing the user to manually perform the alignment so that the overlapping portions of the images overlap, and a second means for synthesizing all the images based on the alignment result by the first means are provided. One means is a means for displaying two images forming an adjacent image pair on a monitor screen, a means for displaying a relative positional relationship between the two images on the monitor screen, and two means displayed on the monitor screen. Picture As the overlapping portion of the overlap, characterized in that it comprises means for moving one of the images by the user.

The second image synthesizing apparatus according to the present invention arranges a plurality of images to be synthesized two-dimensionally on the monitor screen so that their arrangement positions match the actual relative positional relationship, and the respective images are arranged. In the image synthesizing device for synthesizing all the images by using the arrangement position information of the above, the feature points are associated with each other for each adjacent image pair selected from the plurality of images two-dimensionally arranged on the monitor screen. The first means for allowing the user to manually perform the operation and the second means for synthesizing all the images by using the feature points associated with each adjacent image pair by the first means are provided. Is a means for displaying two images forming a pair of adjacent images on a monitor screen, a means for displaying a relative positional relationship between the two images on the monitor screen, and two images displayed on the monitor screen. At both Characterized in that it comprises means for setting by the user of the feature points corresponding between images.

In the above-mentioned first or second image synthesizing device, as a means for displaying the relative positional relationship between two images on the monitor screen, for example, a symbol representing the vertical direction,
From the four types of symbols representing the left-right direction, symbols representing the diagonal direction connecting the lower left and upper right, and symbols representing the diagonal direction connecting the upper left and lower right, the symbol corresponding to the relative positional relationship between the two images is displayed on the monitor screen. What is displayed is used.

According to the first image synthesizing method of the present invention, a plurality of images to be synthesized are two-dimensionally arranged on the monitor screen so that their arrangement positions match the actual relative positional relationship, and each image is arranged. In the image synthesizing method for synthesizing all the images by using the arrangement position information of the above, the overlapping portions of the images are overlapped for each adjacent image pair selected from the plurality of images two-dimensionally arranged on the monitor screen. As described above, the first step for allowing the user to manually perform the alignment, and the second step for synthesizing all the images based on the alignment result by the first step are provided.
A step of displaying two images forming an adjacent image pair on a monitor screen, a step of displaying a relative positional relationship between the two images on the monitor screen as a pattern, and an overlapping portion of the two images displayed on the monitor screen. And a step of moving one image by the user so that the two overlap.

In the second image synthesizing method according to the present invention, a plurality of images to be synthesized are two-dimensionally arranged on the monitor screen so that their arrangement positions match the actual relative positional relationship. In the image synthesizing method of synthesizing all the images by using the arrangement position information of, the feature points are associated with each other for each adjacent image pair selected from a plurality of images two-dimensionally arranged on the monitor screen. The method includes a first step to be manually performed by a user, and a second step of synthesizing all images by using feature points associated with each adjacent image pair by the first step. The step of displaying two images forming the adjacent image pair on the monitor screen, the step of displaying the relative positional relationship of the two images on the monitor screen as a pattern, and the two images displayed on the monitor screen. Characterized in that it comprises the step of setting the corresponding feature points between the two images on the image by the user.

In the first or second image synthesizing method, the step of displaying the relative positional relationship between the two images on the monitor screen is, for example, a symbol indicating the vertical direction, a symbol indicating the horizontal direction, and a lower left corner. From the four types of symbols representing the diagonal direction connecting the upper right and the symbols representing the diagonal direction connecting the upper left and the lower right, the one that displays the symbol corresponding to the relative positional relationship between the two images on the monitor screen is used.

The first computer-readable recording medium according to the present invention makes it possible to arrange a plurality of images to be combined so that their arrangement positions match the actual relative positional relationship.
A computer-readable recording medium in which an image composition processing program for arranging two-dimensionally on a monitor screen and synthesizing all images using the arrangement position information of each image is recorded, and is arranged two-dimensionally on the monitor screen. For each adjacent image pair selected from the plurality of images, based on the first step of allowing the user to manually perform the alignment so that the overlapping portions of the images overlap, and the alignment result of the first step. , A program for causing a computer to execute the second step of combining all images, the first step is a step of displaying two images forming an adjacent image pair on a monitor screen, and the above two steps. The step of displaying the relative positional relationship of the images on the monitor screen as a pattern, and the overlapping portion of the two images displayed on the monitor screen so as to overlap each other. Characterized in that it comprises a step of moving one of the images by the user.

The second computer-readable recording medium according to the present invention makes it possible to arrange a plurality of images to be combined so that their arrangement positions match the actual relative positional relationship.
A computer-readable recording medium in which an image composition processing program for arranging two-dimensionally on a monitor screen and synthesizing all images using the arrangement position information of each image is recorded, and is arranged two-dimensionally on the monitor screen. For each adjacent image pair selected from the plurality of images, the first step of allowing the user to manually associate the feature points with each other, and the feature points associated with each adjacent image pair by the first step. The second step of combining all images using
A program to be executed by a computer is recorded, and the first step is a step of displaying two images forming an adjacent image pair on a monitor screen, and a relative positional relationship between the two images on the monitor screen. The method is characterized by including a step of displaying and a step of allowing a user to set corresponding feature points between the two images on the two images displayed on the monitor screen.

In the first or second computer-readable recording medium, the step of displaying the relative positional relationship between the two images as a symbol on the monitor screen is, for example, a symbol indicating the vertical direction and a symbol indicating the horizontal direction. , 4 types of a symbol representing the diagonal direction connecting the lower left and the upper right and a symbol representing the diagonal direction connecting the upper left and the lower right are used to display the symbol corresponding to the relative positional relationship between the two images on the monitor screen. To be

The third image synthesizing device of the present invention arranges a plurality of images to be synthesized two-dimensionally on the monitor screen so that their arrangement positions match the actual relative positional relationship, and the respective images are arranged. In the image synthesizing apparatus for synthesizing all the images by using the arrangement position information of 1), the plurality of images two-dimensionally arranged on the monitor screen may be overlapped between the adjacent images.
A first means for dimensionally arranging, and a second means for synthesizing all images based on the overlapped portion extracted for each adjacent image pair for each adjacent image pair of the one-dimensionally arranged images. The first means includes the image selected as the leading image of the one-dimensional array from the plurality of two-dimensionally arranged images on the monitor screen as the target image, and it is assumed that each image is arranged without gaps. In this case, among the images that have not been selected yet, the image with the shortest center-point distance to the target image is selected as the image to be arranged next to the target image,
All the images are arranged one-dimensionally by repeating the same process with the selected image as the image of interest.

According to the third image synthesizing method of the present invention, a plurality of images to be synthesized are two-dimensionally arranged on the monitor screen so that their arrangement positions match the actual relative positional relationship. In the image synthesizing method of synthesizing all images by using the arrangement position information of 1), it is possible that a plurality of images two-dimensionally arranged on the monitor screen are overlapped between adjacent images.
A first step of dimensionally arranging, and a second step of extracting an overlapping part for each adjacent image pair of the one-dimensionally arranged images and synthesizing all images based on the overlapping part extracted for each adjacent image pair. In the first step, the image selected as the first image of the one-dimensional array from the plurality of two-dimensionally arranged images on the monitor screen is set as the image of interest, and it is assumed that each image is arranged without gaps. In case of
By selecting the image that has the shortest center-to-center distance from the target image among the images that have not been selected yet as the image to be arranged next to the target image, and repeating the same processing using the selected image as the target image, It is characterized in that the images are arranged one-dimensionally.

The third computer-readable recording medium of the present invention arranges a plurality of images to be combined two-dimensionally on the monitor screen so that their arrangement positions match the actual relative positional relationship. A computer-readable recording medium in which an image combining processing program for combining all images is recorded by using arrangement position information of each image,
The first step of one-dimensionally arranging a plurality of images two-dimensionally arranged on the monitor screen so that adjacent images overlap each other, and the overlapping portion is extracted for each adjacent image pair of the one-dimensionally arranged images. , A program for causing a computer to execute the second step of synthesizing all the images based on the overlapping portion extracted for each adjacent image pair is recorded, and the first step is a two-dimensional arrangement on the monitor screen. Assuming that the image selected as the first image of the one-dimensional array from the plurality of selected images is the target image and that each image is arranged without gaps, the target image among the images not yet selected is The image with the shortest distance between the center points is selected as the image to be arranged next to the target image, and the same process is repeated using the selected image as the target image so that all the images are one-dimensionally arranged. And characterized in that.

[0065]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention applied to a microscope image synthesizing apparatus will be described below.

[1] Description of Overall Structure of Microscope Image Synthesizer

FIG. 7 shows the overall structure of the microscope image synthesizing apparatus.

The microscope image synthesizing device is realized by a personal computer. Personal computer 1
A display (monitor) 21, a mouse 22, and a keyboard 23 are connected to 0. The personal computer 10 includes a CPU 11, a memory 12, a hard disk 13, and a drive (disk drive) 14 for a removable disk such as a CD-ROM.

The hard disk 13 stores a microscope image composition processing program in addition to the OS (operating system) and the like. The microscope image composition processing program is installed in the hard disk 13 using the CD-ROM 20 in which the program is stored. Further, it is assumed that the hard disk 13 stores in advance a plurality of microscope images to be combined.

[2] Description of the microscope image composition processing procedure performed by the CPU 11 when the microscope image composition processing program is activated.

The user uses a camera equipped with a microscope lens to take a plurality of microscope images having overlap between the images while moving the table or moving the sample. Then, the plurality of obtained microscope images are stored in the hard disk 13.
Stored in.

FIG. 8 shows the overall processing procedure performed by the CPU 11.

First, a plurality of microscope images (hereinafter referred to as images) to be combined are input (step 1). That is,
A plurality of images to be combined are selected, and the plurality of selected images are arranged on the monitor screen so that their arrangement positions match the actual relative positional relationship.

Next, preprocessing is performed on each input image (step 2). That is, an image serving as a reference plane for the combining process is selected from the input images, a lens distortion correction process is performed on the input image, and a shading correction process is performed.

Next, a process for joining a plurality of images subjected to the lens distortion correction and the shading correction is performed. There are three image combining modes.

First of all, the first is the first mode in which all are performed automatically, the alignment between images (extraction of the overlapping portion between images).
There is a second mode in which is manually set, and a third mode in which feature points between images are manually associated.

First, in the first mode, after the preprocessing, automatic alignment is performed (step 3). That is, the rough alignment (only parallel movement) of all input images is automatically performed, and the result of the alignment is displayed.

Next, the user is allowed to select a synthesis parameter (step 4). For example, the user is allowed to select whether to combine (translation) only by parallel movement or combine (translation + rotation) in consideration of rotation around the optical axis in addition to parallel movement.

After that, the feature point extraction process (step 5) and the feature point tracking process (step 6) are performed, and then the geometric transformation matrix is calculated (step 7). Then, the weight of the pixel value mixture is calculated (step 8), and finally image combination is performed (step 9).

The second mode is prepared in case the automatic alignment in the first mode fails.
In the second mode, manual alignment is performed after pretreatment (step 10). That is, for each adjacent image pair selected from all the input images, the alignment between the images (only parallel movement) is manually performed. After this, the processes of steps 4 to 9 are performed.

The third mode is prepared in case the synthesis in the first mode or the second mode fails. In the third mode, the user is made to set the feature points after the preprocessing (step 11). That is, for each adjacent image pair selected from all the input images, the portion (feature point) to be matched between both images is manually set. Next, the user is made to select the synthesis parameter (step 12). After this,
The above steps 7 to 9 are performed.

Each step will be described below.

[2] Description of Image Input (Step 1)

FIG. 9 shows an image input screen.

The image input screen includes a folder tree display section 101 for selecting an image folder in which images to be combined are stored, and an image file display for displaying image files in the selected image folder. Part 1
02, work space 103, matrix setting section 104, reference image selection button 105, lens distortion correction button 10
6, shading correction button 107, selected image input button 108, all image input button 109, selected image input cancel button 110, all image input cancel button 111, automatic alignment button 112, manual alignment button 113,
A feature point setting button 114 and the like are provided.

FIG. 10 shows an image input procedure.

First, an image to be combined is selected (step 21). When an image folder storing images to be combined is designated from the folder tree displayed in the folder tree display unit 101, the images stored in the designated image folder are displayed vertically in the image file display window 102. Is displayed in. Here, it is assumed that nine images are stored in the designated image folder, and those nine images are displayed in the folder order in the image file display window 102. Note that these images may be displayed in order of the date and time when the images were taken.

Next, the number of rows and columns of image input frames displayed in the work space 103 is set according to the number of selected images and the like (step 22). The initial value of the number of rows and columns is 3
It is × 3, and a 3 × 3 image input frame is displayed in the work space 103.

The number of rows and columns can be set by, for example, pressing the Enter key after inputting the number of rows and the number of columns in the number-of-rows setting section 104, or the adjustment knob at the lower right of the entire image input area S surrounding the entire image input frame. This is done by dragging 115. Here, it is assumed that the initial values “3” are maintained for both the number of rows and the number of columns. Therefore, the 3 × 3 image input frame remains displayed in the work space 103.

Next, when the all image input button 109 is clicked (step 23), an image layout pattern selection screen as shown in FIG. 11 is displayed on the image input screen (step 24).

On the image layout pattern selection screen, a plurality of types of image layout patterns indicating how the images vertically displayed on the image file display unit 102 are to be input in the two-dimensionally arranged image input frame A to D, a radio button for selecting an image arrangement pattern, an OK button,
The cancel button and are displayed.

One of the plurality of radio buttons is selected. Immediately after the image synthesizing program is installed, a radio button for a predetermined default image layout pattern is displayed as selected, but after the user has selected the image layout pattern even once, The radio button for the image layout pattern most recently selected by the user is displayed as selected.

The user selects an image layout pattern so that the plurality of images vertically displayed on the image file display unit 102 are two-dimensionally arranged in a state where the actual relative positional relationship is maintained, and then OK. Click the button (step 25). As a result, each image vertically displayed on the image file display unit 102 is automatically input (displayed) in the two-dimensionally arranged image input frame (step 26). That is, the respective images are two-dimensionally arranged while maintaining their actual relative positional relationship.

The nine images are A1-A in the order of file names.
If represented by 9, the image arrangement patterns A to D and the images A1 to
The relationship between A9 and the two-dimensional array position is as shown in FIG.

In this way, when a plurality of images to be combined are two-dimensionally arranged, the file name of each image and the two-dimensional arrangement position of each image (the position of the image input frame into which the image is input) ( x, y) are stored (step 27). The two-dimensional arrangement position (x, y) of each image is represented by an XY coordinate system in which the upper left of the entire image input area S is the origin and the x direction is in the right direction and the y direction is in the down direction.

The user can input one image in each image input frame by dragging and dropping or selecting image input button 108.

The user can also cancel the image input in the image input frame. When the all image input cancel button 111 is pressed, all input images are canceled, and when the selected image input cancel button 110 is pressed after selecting an input image, the selected input image is canceled. The input image can also be released by dragging and dropping the input image outside the entire image input area S.

As the image layout pattern, "line feed return / replacement return pattern shown in (a) to (c) of FIG.
The "folding pattern" shown in (d) to (f) of FIG. 13 and the spiral pattern shown in (g) to (i) of FIG. 13 can be used.

[3] Description of preprocessing (step 2)

[3-1] Setting of Reference Image To select the reference image, the reference image selection button 105 may be pressed after selecting the input image to be used as the reference.

[3-2] Lens Distortion Correction When the user presses the lens distortion correction button 106 on the image input screen (FIG. 9), lens distortion correction processing is performed on the input image.

The microscope lens has distortion in its peripheral portion. A correction vector (du, d for removing this distortion
v) is calculated using the following equation 10.

[0103]

[Equation 10]

However, u and v are the coordinates when the center of the image is the origin and the orthogonal coordinate axes are U and V. Also,
k is a lens distortion correction coefficient.

Using the correction vector (du, dv), the coordinates (u, v) of the input image are calculated by the following equation 11.
Lens distortion is corrected by converting to coordinates (u ′, v ′).

[0106]

[Equation 11]

When the user presses the lens distortion correction button 106 on the image input screen (FIG. 9), a lens distortion coefficient setting screen as shown in FIG. 14 is displayed. The user sets the lens distortion correction coefficient k by operating the slider bar 31 on the lens distortion coefficient setting screen.

The lens distortion coefficient setting screen is provided with an area 32 in which an image before lens distortion correction is displayed and an area 33 in which an image after lens distortion correction is displayed. Therefore, the user can view the image before the lens distortion correction and the image after the lens distortion correction at the same time. Further, the user can set the coefficient intuitively by a simple operation because the converted image (image after lens distortion correction) is displayed in real time when the coefficient k is set by the slider bar 31.

[3-3] Shading correction When the user presses the shading correction button 107 on the image input screen (FIG. 9), the shading correction process is performed on the input image.

In the microscope image pickup system, depending on the setting of the optical system,
Shading occurs where the periphery of the image is dark. In order to correct this, the corrected pixel value e (u, v) according to the distance from the image center is calculated for each pixel by the following expression 12.

[0111]

[Equation 12]

However, u and v are coordinates when the center of the image is the origin and the orthogonal coordinate axes are U and V. Also,
s is the shading correction coefficient, and R is the distance from the center of the pixel farthest from the image center.

Shading is performed by converting the pixel value I (u, v) of the input image into the pixel value I '(u, v) by using the corrected pixel value e (u, v) according to the following expression 13. To correct.

[0114]

[Equation 13]

When the user presses the shading correction button 107 on the image input screen (FIG. 9), the shading correction coefficient setting screen as shown in FIG. 15 is displayed on the display 21.
Is displayed in. The user selects the shading correction factor s
Is set by operating the slider bar 41 on the shading correction coefficient setting screen.

The shading correction coefficient setting screen is provided with a region 42 in which an image before shading correction is displayed and a region 43 in which an image after shading correction is displayed. Therefore, the user can simultaneously view the image before shading correction and the image after shading correction. In addition, the user can move the slider bar 41
When the coefficient s is set in, the converted image (image after shading correction) is displayed in real time, so that the coefficient can be set intuitively by a simple operation.

[4] Description of automatic alignment (step 3)

When the user presses the automatic alignment button 112 on the image input screen (FIG. 9), automatic alignment processing is performed, and as shown in FIG. 16, an automatic alignment result display screen containing alignment results is displayed. Is displayed. On the automatic alignment result display screen, an automatic combination button 121, a cancel button 122, etc. are displayed in addition to the alignment result.

In the automatic alignment process, first, the adjacent image pair from which the overlapping portion is to be extracted is determined, and then the overlapping portion is extracted for each adjacent image pair. After that, based on the overlapped portion extracted for each adjacent image pair, all the images are aligned using only the parallel movement. Then, the alignment result is displayed.

A method of determining the adjacent image pair for extracting the overlapping portion will be described in the manual alignment (step 10). The process of extracting the overlapping portion is performed as follows. For convenience of description, here, a case will be described in which the overlapping portion of one pair of adjacent images (first image A1 and second image A2) is extracted.

The process of extracting the overlapping portion is performed by, for example, SSD.
(Sum of Squared Difference) or the normalized cross-correlation method.

(A) Description of SSD Method In the SSD method, first, for each of the two images A1 and A2 from which the overlapping portion is to be extracted, images I ₁ and I ₂ having a lower resolution than the original image are generated. The overlapping portion ω (size: M × N) of the _two low resolution images I ₁ and I ₂ is obtained using the squared error E per pixel as shown in the following Expression 14. The movement amount (d) between the images is changed within a possible range, and the overlapping portion is extracted from the movement amount (d) where E is the smallest.

[0123]

[Equation 14]

(B) Description of Normalized Cross-Correlation Method In the normalized cross-correlation method, first, the overlapping portion should be extracted 2
For each of the images A1 and A2, images I ₁ and I ₂ having a lower resolution than the original image are generated. The overlapping portion ω (size: M × N) of the _two low-resolution images I ₁ and I ₂ is
It is obtained using the normalized cross-correlation coefficient C as shown in the following Expression 15. The moving amount (d) between the images is changed within a possible range, and the overlapping portion is extracted from the moving amount (d) having the largest C.

[0125]

[Equation 15]

In the equation 15, I _{1 −} and I _{2 −} are the first
When the image is fixed and the second image is moved by d,
It is the average of the gray value of each image in the overlapping portion of both images. Further, σ ₁ and σ ₂ are variances of the grayscale values of the respective images in the overlapping portion of the two images when the first image I ₁ is fixed and the second image I ₂ is moved by d.

When the user presses the automatic composition button 121 on the automatic alignment result display screen (FIG. 16), a composition parameter selection screen as shown in FIG. 17 is displayed.
The user selects synthesis parameters (step 4 in FIG. 8).
Processing). A “translation” button 131, a “translation + rotation” button 132, a cancel button 133, etc. are displayed on the synthesis parameter selection screen. User has "translation" button 1
When either 31 or the "translation + rotation" button 132 is pressed, the processing of steps 5-9 (see FIG. 8) is performed,
All images are combined.

[5] Description of Manual Positioning (Step 10)

FIG. 18 shows a manual alignment processing procedure performed when the user presses the manual alignment button 113 on the image input screen (FIG. 9).

When the user presses the manual alignment button 113 on the image input screen (FIG. 9), the adjacent image pair from which the overlapping portion should be extracted is determined (step 31).

FIG. 19 shows the procedure for determining adjacent image pairs.

The adjacent image pair is determined by arranging the two-dimensionally arranged images so that there is overlap between the adjacent images.

The image number of the one-dimensional array is represented by i. First, i = 1 is set (step 41), and the first (first) image B _i of the one-dimensional array is selected from the two-dimensional array of images (step 42). For example, position (0,0)
Image is selected as the first image B _i of the one-dimensional array.

Next, assuming that the two-dimensionally arranged images are arranged without gaps, among the unselected images, the image having the shortest center-to-center distance from the image B _i is The image B _{i + 1} to be arranged is selected (step 43).

Then, it is judged whether or not there are still unselected images (step 44). If there are still unselected images, i = i + 1 is set (step 45) and the process returns to step 43.

In this way, by repeating the processing of steps 43, 44 and 45 until all the images are selected, two-dimensionally arranged images are one-dimensional so that adjacent images overlap each other. Arranged. Adjacent ones of the images arranged in a one-dimensional manner form an adjacent image pair from which an overlapping portion is extracted.

In this example, the two-dimensionally arranged images are
Since the long length of the horizontal than vertical, it arranged laterally than the distance between the center points of the two images, since the person between the center point distance in the longitudinal direction into two aligned images is short, for example, the image B _i If the unselected image adjacent in the vertical direction for present, unselected image image adjacent in the vertical direction with respect to the image B _i B _{i + 1}
Is selected as. Further, for example, when there is no unselected image vertically adjacent to the image B _i but an unselected image horizontally adjacent to the image B _i , there is an unselected image horizontally adjacent to the image B _i . The selected image is selected as the image B _{i + 1} .

Therefore, as shown in FIG. 20, for example, when 20 images are arranged in 5 rows and 4 columns, these images are arranged one-dimensionally in the order shown by the arrows in FIG.

However, as shown in FIG. 21, of the image input frames arranged in 5 rows and 4 columns, the two image input frames at the center (image input at positions (2, 3) and (3, 3)) are input. frame)
If there is no image in the image, these images are arranged one-dimensionally in the order shown by the arrow in FIG.

When the two-dimensionally arranged images are one-dimensionally arranged so that adjacent images overlap each other, a manual alignment screen as shown in FIG. 22 is displayed (step 32, FIG. 18).

The manual alignment screen is provided with a one-dimensional array image display section 141, a work space 142, an automatic alignment button 143, an automatic composition button 144, a cancel button 145 and the like.

The one-dimensional array image display unit 141 is provided in the lower stage, and the work space 142 is the one-dimensional array image display unit 14.
It is provided above 1. One-dimensional array image display unit 14
1 and the work space 142, two pair selection keys 146 and 147 for selecting the adjacent image pair to be aligned are displayed, and the relative positional relationship of the selected adjacent image pair (composite image) is displayed. A composite direction indicator mark 148 indicating the direction is displayed.

The one-dimensionally arrayed image display section 141 displays each one-dimensionally arrayed image. In the work space 142, two images selected as an adjacent image pair among the images displayed in the one-dimensional array on the one-dimensional array image display unit 141 are displayed semitransparently.

On the one-dimensional array image display section 141, a rectangular frame 149 surrounding the currently selected adjacent image pair is displayed in order to clearly indicate the currently selected adjacent image pair. When the pair selection key 146 is clicked, the adjacent image pair is shifted to the left by one image. When the pair selection key 147 is clicked, the adjacent image pair is shifted to the right by one image.

As the composite direction mark 148, as shown in FIG. 23, the vertical direction (a), the horizontal direction (b), the oblique direction connecting the lower left and the upper right (c), and the oblique direction connecting the upper left and the lower right (d). There are four types.

The combination direction of the selected adjacent image pair is determined based on the two-dimensional arrangement position of the two images.
The two-dimensional arrangement position of one image of the adjacent image pair is (x1,
y1), and the two-dimensional arrangement position of the other image is (x2, y
2).

If x1 = x2, the combination direction is the vertical direction. If y1 = y2, the combination direction is the left-right direction. In the case where x1 ≠ x2 and y1 ≠ y2,
If (x1−x2) × (y1−y2) <0, the combining direction is an oblique direction connecting the lower left and upper right. When x1 ≠ x2 and y1 ≠ y2, (x1−x2) ×
If (y1-y2)> 0, the combining direction is an oblique direction connecting the upper left and lower right.

When the manual alignment screen (FIG. 22) is displayed, the user selects the adjacent image pair, and then manually visually overlaps the overlapping portions of the two images semitransparently displayed in the working space 142 ( Step 33, FIG.
8). This operation is performed, for example, by dragging one of the two semi-transparently displayed images in the work space 142 with the mouse. It is also possible to perform this operation by operating the keys of the keyboard. The manual registration is completed by performing the work of step 33 on all the adjacent image pairs.

When the automatic alignment button 143 is pressed in a state where the two images are displayed in the work space 142 in a semi-transparent state, the two images in the work space 142 which are displayed in a semi-transparent state are described above. Positioning is automatically performed by the same processing as the automatic extraction processing of the overlapping portion.

When the user presses the automatic synthesizing button 144 after aligning all the adjacent image pairs, a synthesizing parameter selection screen as shown in FIG. 17 is displayed, so that the user selects a synthesizing parameter. (Processing of step 4 in FIG. 8). The user selects "Translation" button 131 or "Translation + Rotation" button 1 on the synthesis parameter selection screen.
When any one of 32 is pressed, the processing of steps 5 to 9 (see FIG. 8) is performed and all the images are combined.

[6] Description of setting of feature points (step 11)

When the user presses the feature point setting button 114 on the image input screen (FIG. 9), the feature point setting screen as shown in FIG. 24 is displayed after the adjacent image pair from which the overlapping portion is to be extracted is determined. To be done.

Each adjacent image pair is determined by a process similar to the process of step 31 of the manual alignment process (FIG. 18).

On the feature point setting screen, two image display sections 161, 162 for respectively magnifying and displaying two images forming an adjacent image pair, two pair selection keys 163 for selecting an adjacent image pair, 164, feature point delete button 165, automatic composition button 166, cancel button 1
It is equipped with 67 etc. Further, in the lower right portion of the feature point setting screen, a combination direction instruction mark 168 indicating the relative positional relationship (composition direction) of the selected adjacent image pairs is displayed.

The pair selection keys 163 and 164 and the combination direction instruction mark 168 are displayed on the manual alignment screen (see FIG. 2).
It has the same function as the pair selection keys 146 and 147 and the combination direction instruction mark 148 of 2).

The user has two image display units 161, 16
Based on the two images displayed in No. 2, the feature points are manually associated. Specifically, if you double-click any feature point position on the right image, the feature point setting frame (rectangular frame)
169 is displayed on both images. By dragging the feature point setting frame 169 displayed on the left image with the mouse, the feature point setting frame 169 is moved to the position corresponding to the feature point on the right image. When the feature point delete button 165 is pressed, the selected feature point setting frame is deleted.

When "translation" is selected as the synthesis method, one or more feature points between the two images are set. When "translation + rotation" is selected as the synthesis method, the feature points between the two images are set. Set at two or more locations. The larger the number of set feature points, the higher the synthesis accuracy.

When the user presses the automatic composition button 166 after setting the feature points for all the adjacent image pairs, a composition parameter selection screen as shown in FIG. 17 is displayed.
The user selects synthesis parameters (step 1 in FIG. 8).
Process 2). On the synthesis parameter selection screen, the user
"Translation" button 131 or "Translation + rotation" button 132
When any of the above is pressed, the processes of steps 7 to 9 (see FIG. 8) are performed and all the images are combined.

[7] Description of Feature Point Extraction Process (Step 5)

When the process of step 4 is completed, that is, the user selects the "translation" button 131 on the synthesis parameter selection screen (see FIG. 17) displayed in step 4.
Or press one of the "Translation + Rotation" buttons 132
Feature point extraction processing is performed.

For convenience of explanation, one adjacent image pair A1,
Only the process for A2 will be described. First image A1
A plurality of partial images (rectangular regions) effective for tracking are extracted as feature points from the overlapping portion with the second image A2 in. However, the respective feature points are extracted so as not to overlap each other. Specifically, the above-mentioned eigenvalue λ _min (see equation 8)
Is extracted as a feature point.

[8] Description of Feature Point Tracking Process (Step 6)

When the feature point extraction process for each adjacent image pair is completed, the feature point tracking process is performed. That is, the positions on the second image A2 with respect to the extracted feature points on the first image A1 are tracked.

Specifically, first, an appropriate size (for example, 13 ×) is determined by the optical flow estimation method developed by the present applicant (see [2] in the section of the description of the prior art).
The optical flow vector for each patch in 13) is obtained. The second image A corresponding to the feature point on the first image A1
The position on 2 is obtained in pixel units or less by linear interpolation from the flow vectors of the patches in the vicinity of the feature points on the first image A1. Thereby, the first image A1 and the second image A2
At the overlapping portion of and, the coordinates of the corresponding points of both images are obtained.

[0165]

[9] Description of geometric transformation matrix calculation processing (step 7)

The case where the second image A2 is joined to the first image A1 will be described. In the first mode or the second mode, the first value automatically obtained in step 6 above.
Based on the coordinates of the corresponding points between the image A1 and the second image A2,
In the third mode, the coordinates of each pixel in the second image A2 are set on the first image A1 based on the coordinates of the corresponding points between the first image A1 and the second image A2 manually calculated in step 11. A geometric transformation matrix (geometric transformation coefficient) for transforming into coordinates is calculated.

When the first image A1 is joined to the second image A2, a geometric transformation matrix (geometric transformation) for transforming the coordinates of each pixel in the first image A1 into the coordinates on the second image A2. The coefficient) may be calculated.

When "translation" is selected by the synthesis parameter selection process (step 4 or step 12), a two-dimensional translation matrix is used as the geometric transformation matrix, and "translation + rotation" is used. When is selected, a two-dimensional rigid body transformation matrix is used as the geometric transformation matrix.

At this stage, the coordinates of each pixel in the second image A2 are not converted into the coordinates on the first image A1, but the coordinates (x, y, 1) of each pixel in the second image A2 are converted into the first coordinates. An equation for converting the coordinates (x ′, y ′, 1) of the image A1 is represented by the following equation 16 or equation 17. The coordinates (x,
y), (x ', y') are the coordinates of the pixels in the image,
It is different from the coordinates (x, y) indicating the two-dimensional arrangement position of the image used in the description of step 1 in FIG.

[0170]

[Equation 16]

[0171]

[Equation 17]

The above expression 16 shows a conversion expression when a two-dimensional translation matrix is used as the geometric conversion matrix, and the above expression 17 is a conversion when a two-dimensional rigid body conversion matrix is used as the geometric conversion matrix. Shows the formula.

In the above expression 16, (t _x , t _y ) represents the amount of parallel movement, and k represents any non-zero constant. Further, in the above Expression 17, (t _x , t _y ) represents the amount of parallel movement, θ represents the rotation angle, and k represents any non-zero constant.

[10] Description of Calculation Process of Pixel Value Mixing Weight (Pixel Value Mixing Coefficient) (Step 8)

With a camera equipped with a microscope lens, while moving the sample table or moving the sample on the sample table,
When a plurality of images are captured, the brightness between the images may differ due to the operation of the automatic exposure adjustment function.

Therefore, in a region where the first image and the second image to be joined overlap each other, the process of blending the pixel values of the images is performed as a post-process. The pixel value mixing process is a process in which the difference in brightness is made inconspicuous by smoothly changing the pixel values of the overlapping regions of both images between the images. In addition, this processing can also make the synthesis error due to the geometric conversion inconspicuous.

In this embodiment, the pixel values of the overlapping areas of both images are mixed based on the distance from the center of gravity of each image.

First, the center of gravity position G ₁ of the first image is obtained.
In addition, on the composite image in which the second image is joined to the first image, the center of gravity position G ₂ of the second image is calculated based on the conversion equation (Equation 16 or 17) used for joining. Ask.

Next, the positions of the four vertices (four corners) of the first image are obtained. Further, the positions of the four vertices of the second image are obtained on the composite image in which the second image is joined to the first image, based on the conversion equation (Equation 16 or 17) used for the joining. As a result, the positions of the four vertices of the first image and the positions of the four vertices of the second image on the composite image are known, and therefore the overlapping region between the first image and the second image is extracted on the composite image.

From each pixel (pixel value blending target pixel) in the overlapping area of the first image and the second image on the composite image, the center of gravity positions G ₁ and G ₂ of the respective images of the first image and the second image The weight of the pixel value of the pixel value blending target pixel is determined according to the distance to. This weight is determined so that the pixel value of the image closer to the center of gravity from the pixel value blending target pixel is larger.

That is, from the coordinates of the pixel value blending target pixel in the overlapping area of the first image and the second image on the composite image,
The Euclidean distance to the center of gravity position G ₁ of the _first image on the composite image is d ₁ , and the center of gravity position G of the second image on the composite image G
_When the Euclidean distance to ₂ is d ₂ , the weight w ₁ for the pixel value of the first image and the weight w ₂ for the pixel value of the second image in the pixel value blending target pixel are represented by the following Expression 18.

[0182]

[Equation 18]

Weights w ₁ and w ₂ are calculated for each pixel value blending target pixel based on Expression 21, and a table (weighting table) of weights w ₁ and w ₂ for the pixel value blending target pixel is created.

[12] Description of Pixel Combining Process (Step 9)

In the pixel synthesizing process, the second image is drawn in the coordinate system of the first image by using the above equation 16 or equation 17.

At this time, the pixel values P ₁ and P ₂ in the overlapping area of the first image and the second image are pixel-mixed based on the weight table obtained in step 8. That is, the pixel value P _i of each pixel value blending target pixel is calculated by the following equation 19.

[0187]

[Formula 19]

In this way, when the combining process for all the adjacent image pairs is completed, the combined image display screen containing the combined image which is the result of combining is displayed.

[0189]

According to the present invention, in the case where the user manually performs the alignment so that the overlapping portions of the adjacent image pairs are overlapped, the work efficiency for the alignment is improved. Can be achieved.

According to the present invention, when the user manually associates the feature points for each adjacent image pair, the work efficiency for associating the feature points can be improved.

According to the present invention, even if the image does not exist in the central portion as a result of the two-dimensional arrangement of the images on the monitor screen, a plurality of two-dimensionally arranged images are not overlapped between adjacent images. As is possible, it becomes possible to arrange one-dimensionally.

[Brief description of drawings]

FIG. 1 is a diagram for explaining a hierarchical estimation method and is a schematic diagram showing an image of layer 1.

FIG. 2 is a diagram for explaining a hierarchical estimation method, and is a schematic diagram showing an image of layer 2;

FIG. 3 is a diagram for explaining a hierarchical estimation method, and is a schematic diagram showing an image of layer 3;

FIG. 4 is a diagram for explaining a hierarchical estimation method and is a schematic diagram showing an image of layer 4;

FIG. 5 is a schematic diagram for explaining dilation processing performed in the optical flow estimation method adopted in the embodiment.

FIG. 6A is a schematic diagram showing an example of an optical flow that has been threshold-processed for an image of a certain layer, and FIG. 6B is a schematic diagram showing an optical flow after complementation. It is a figure.

FIG. 7 is a block diagram showing a configuration of a microscope image composition device.

FIG. 8 is a flowchart showing a microscope image composition processing procedure performed by the CPU 11 when a microscope image composition processing program is activated.

FIG. 9 is a schematic diagram showing an image input screen.

FIG. 10 is a flowchart showing a procedure of image input.

FIG. 11 is a schematic diagram showing an example of an image arrangement pattern selection screen.

FIG. 12 is a schematic diagram showing the relationship between the image arrangement patterns A to D and the two-dimensional array positions of the images A1 to A9.

FIG. 13 is a schematic diagram showing an example of an image arrangement pattern.

FIG. 14 is a schematic diagram showing an example of a lens distortion correction coefficient setting screen.

FIG. 15 is a schematic diagram showing an example of a shading correction coefficient setting screen.

FIG. 16 is a schematic diagram showing an example of an automatic alignment result display screen.

FIG. 17 is a schematic diagram showing an example of a synthesis parameter selection screen.

FIG. 18 is a flowchart showing a manual alignment processing procedure.

FIG. 19 is a flowchart illustrating a procedure for determining an adjacent image pair.

FIG. 20 is a schematic diagram showing the order of the one-dimensional array determined by the adjacent image pair determination processing.

FIG. 21 is a schematic diagram showing the order of the one-dimensional array determined by the adjacent image pair determination processing.

FIG. 22 is a schematic diagram showing an example of a manual alignment screen.

FIG. 23 is a schematic diagram showing types of composite direction marks.

FIG. 24 is a schematic diagram showing an example of a feature point setting screen.

[Explanation of symbols]

10 personal computer 11 CPU 12 memories 13 hard disk 14 disk drive

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Takashi Iida             2-5-3 Keihan Hondori, Moriguchi City, Osaka Prefecture             Within Yo Denki Co., Ltd. F-term (reference) 5B050 BA06 BA18 CA07 DA02 EA05                       EA12 EA13 EA19 EA30 FA02                       FA13                 5B057 CA08 CA12 CA16 CB08 CB12                       CB16 CD02 CE08 CE10 CE11                       DA07 DB02                 5C076 AA12 AA17 AA19 AA24 BA06                       CA02 CA08                 5C082 AA00 BA12 BB15 BB42 CA52                       CA54 CA55 CB05 DA53 DA86                       MM09 MM10

Claims (12)

[Claims] 1. A plurality of images to be combined are two-dimensionally arranged on a monitor screen so that their arrangement positions match the actual relative positional relationship, and using the arrangement position information of each image, In an image synthesizing device for synthesizing all images, a user manually aligns the overlapping portions of adjacent images selected from a plurality of two-dimensionally arranged images on the monitor screen so that the overlapping portions of those images overlap each other. And a second means for synthesizing all the images based on the alignment result by the first means. The first means monitors two images forming an adjacent image pair. One of the images is displayed to the user so that the means for displaying on the screen, the means for displaying the relative positional relationship between the two images on the monitor screen, and the overlapping portion of the two images displayed on the monitor screen overlap each other. Yo Image synthesizing apparatus characterized by comprising means for moving Te. 2. A plurality of images to be combined are two-dimensionally arranged on a monitor screen so that the arrangement positions thereof match the actual relative positional relationship, and the arrangement position information of each image is used, In an image synthesizing device for synthesizing all images, a first method for allowing a user to manually associate the feature points with each other for each adjacent image pair selected from a plurality of images two-dimensionally arranged on the monitor screen 1 means, and 1
A second means for synthesizing all images by using the feature points associated with each adjacent image pair by the means is provided, and the first means includes two images forming the adjacent image pair on the monitor screen. To display the relative positional relationship between the two images on the monitor screen, and to allow the user to set the corresponding feature points between the two images on the two images displayed on the monitor screen. An image synthesizing apparatus comprising: 3. A means for displaying a relative positional relationship between two images on a monitor screen as means for displaying a symbol indicating the vertical direction, a symbol indicating the horizontal direction, a symbol indicating an oblique direction connecting the lower left and upper right and the upper left and lower right. The image synthesizing device according to claim 1, wherein a pattern corresponding to the relative positional relationship between the two images is displayed on the monitor screen from four types of symbols representing the diagonal direction connecting the two. 4. A plurality of images to be combined are two-dimensionally arranged on a monitor screen so that the arrangement positions thereof match the actual relative positional relationship, and the arrangement position information of each image is used, In an image compositing method that composes all images, the user manually aligns the overlapping parts of adjacent images selected from a plurality of two-dimensionally arranged images on the monitor screen so that they overlap each other. And a second step of synthesizing all images based on the alignment result of the first step. The first step is to display two images forming an adjacent image pair on a monitor screen. On the monitor screen, the step of displaying the relative positional relationship between the two images on the monitor screen, and one of the two images displayed on the monitor screen so that the overlapping portions overlap each other. Image synthesis method characterized in that comprises a step of moving the image by the user. 5. A plurality of images to be combined are two-dimensionally arranged on a monitor screen so that the arrangement positions thereof match the actual relative positional relationship, and the arrangement position information of each image is used, In an image synthesizing method for synthesizing all images, a user manually associates those feature points with each other for each adjacent image pair selected from a plurality of images two-dimensionally arranged on the monitor screen. , And a second step of synthesizing all the images by using the feature points associated with each adjacent image pair by the first step, the first step including two images forming the adjacent image pair. On the monitor screen, the step of displaying the relative positional relationship between the two images on the monitor screen as a pattern, and the correspondence between the two images on the two images displayed on the monitor screen. Image synthesis method characterized in that comprises a step of setting the feature points by the user. 6. The step of displaying the relative positional relationship between the two images on the monitor screen is a symbol indicating the vertical direction, a symbol indicating the horizontal direction, a symbol indicating the diagonal direction connecting the lower left and upper right and the upper left and lower right. The image synthesizing method according to claim 4, wherein a symbol corresponding to a relative positional relationship between the two images is displayed on the monitor screen from four types of symbols representing an oblique direction connecting the two. 7. A plurality of images to be combined are two-dimensionally arranged on a monitor screen so that the arrangement positions thereof match the actual relative positional relationship, and the arrangement position information of each image is used, A computer-readable recording medium that records an image combination processing program that combines all images, and overlaps the images for each adjacent image pair selected from a plurality of two-dimensionally arranged images on the monitor screen. Recording a program for causing a computer to execute a first step in which the user manually performs alignment so that the parts overlap, and a second step in which all images are combined based on the alignment result of the first step The first step is to display two images forming an adjacent image pair on the monitor screen, and monitor the relative positional relationship between the two images. A computer-readable record comprising a step of displaying a pattern on a surface and a step of moving one image by a user so that an overlapping portion of two images displayed on the monitor screen overlaps. Medium. 8. A plurality of images to be combined are two-dimensionally arranged on a monitor screen so that their arrangement positions match the actual relative positional relationship, and using the arrangement position information of each image, A computer-readable recording medium in which an image combining processing program for combining all images is recorded, and for each adjacent image pair selected from a plurality of images two-dimensionally arranged on a monitor screen, those feature points To cause the computer to perform the first step of manually associating the user and the second step of synthesizing all the images using the feature points associated with each adjacent image pair by the first step The first step is to display two images forming an adjacent image pair on a monitor screen, and monitor the relative positional relationship between the two images. A computer-readable recording medium comprising: a step of displaying a symbol on a screen; and a step of allowing a user to set a corresponding feature point between the two images displayed on the monitor screen. . 9. The step of displaying the relative positional relationship of two images on a monitor screen is a symbol representing the vertical direction, a symbol representing the horizontal direction, a symbol representing an oblique direction connecting the lower left and upper right and the upper left and lower right. 9. The computer-readable record according to claim 7, wherein a pattern corresponding to the relative positional relationship between the two images is displayed on the monitor screen from four types of symbols representing the diagonal direction connecting the two. Medium. 10. A plurality of images to be combined are two-dimensionally arranged on a monitor screen so that their arrangement positions match the actual relative positional relationship, and using the arrangement position information of each image, In an image synthesizing device for synthesizing all images, a first means for arranging a plurality of images two-dimensionally arranged on a monitor screen in a one-dimensional manner so that adjacent images overlap each other, and each of the one-dimensionally arranged images The second means for extracting the overlapping part for each adjacent image pair and synthesizing all the images based on the overlapping part extracted for each adjacent image pair is provided, and the first means is arranged two-dimensionally on the monitor screen. Assuming that the image selected as the first image of the one-dimensional array from the plurality of selected images is the target image and that each image is arranged without gaps, the target image among the images not yet selected is The distance between the center points is Also select an image to be arranged next short image target image, by repeating the same processing the selected image as the target image, the entire image 1
An image synthesizing device characterized by being arranged in a three-dimensional manner. 11. A plurality of images to be combined are two-dimensionally arranged on a monitor screen so that their arrangement positions match the actual relative positional relationship, and using the arrangement position information of each image, In an image synthesizing method for synthesizing all images, a first step of arranging a plurality of images two-dimensionally arranged on a monitor screen so that adjacent images overlap each other, and each of the images arranged in one dimensional A second step of extracting an overlapping portion for each adjacent image pair and synthesizing all images based on the overlapping portion extracted for each adjacent image pair is provided, and the first step is a two-dimensional arrangement on the monitor screen. Assuming that the image selected as the first image of the one-dimensional array from the plurality of selected images is the target image and that each image is arranged without gaps, the target image among the images not yet selected is During ~ The image having the shortest point-to-point distance is selected as an image to be arranged next to the target image, and the same process is repeated using the selected image as the target image, whereby all the images are arranged one-dimensionally. Image composition method. 12. A plurality of images to be combined are two-dimensionally arranged on a monitor screen so that the arrangement positions thereof match the actual relative positional relationship, and the arrangement position information of each image is used, A computer-readable recording medium recording an image combination processing program for combining all images, in which a plurality of images two-dimensionally arranged on a monitor screen are arranged one-dimensionally so that adjacent images overlap each other. The computer executes the one step and the second step of extracting the overlapping part for each adjacent image pair of the one-dimensionally arranged images and synthesizing all the images based on the overlapping part extracted for each adjacent image pair. The first step is to select the image selected as the first image of the one-dimensional array from the plurality of two-dimensionally arranged images on the monitor screen as the target image. Assuming that the images are arranged without gaps, the image with the shortest center-to-center distance from the image of interest among the images that have not been selected is selected as the image to be arranged next to the image of interest, and selected. A computer-readable recording medium, wherein all the images are arranged in a one-dimensional manner by repeating the same processing using the image as the noted image.