JP2000227958A - Method for extracting corresponding area of image, method and device for compositing image, image compositing program, and computer-readable recording medium recording image compositing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately extract a corresponding area from two images without being influenced by a change in a luminance level. SOLUTION: Secondary differential images are found out by secondarily differentiating two original images, and while changing an overlapped area between the two secondary differential images, the number of edges of at least one of the secondary differential images on the overlapped area is calculated. Then, an error map is found out based on an error between the secondary differential images on the overlapped area and a corresponding area between the two original images is extracted based on the number of edges calculated and the error map found out.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a method for extracting a corresponding area of an image, an image synthesizing method, an image synthesizing apparatus, an image synthesizing program, and a computer-readable recording medium on which the image synthesizing program is recorded.

[0002]

2. Description of the Related Art Hitherto, a technique (image mosaicing) of pasting a plurality of images and synthesizing an image having a wide field of view and a high resolution so that seams are not conspicuous has been actively studied. Classic applications include the synthesis of aerial and satellite photographs. Recently, a method of synthesizing a seamless panoramic image from a plurality of digital images to construct a virtual reality environment with a high sense of reality has attracted attention.

As a technique for synthesizing a panoramic image, a plurality of images are captured in advance by moving a camera, and two images having regions (corresponding regions) corresponding to each other are obtained from the obtained plurality of images. Various methods have been proposed for synthesizing two images so that the common area overlaps after the user makes a selection.

[0004] As a method of automatically extracting a common area between the two images using a computer or the like, a technique called an SSD method (Sum of Square Difference) is generally used.

In the SSD method, first, two images specified by a user are read into a memory in a personal computer, and then, for each of the two images, an original image or images J1 and J2 having a lower resolution than the original image are generated. I do. An overlap area ω of these two images J1 and J2 (size: M × N)
Is the square error per pixel (E) as shown in the following equation 1.
Is determined using That is, the amount of movement between images (d)
Is changed as much as possible, the movement amount (d) at which E becomes the smallest is obtained, and the overlapping area ω at that time is extracted as the corresponding area.

[0006]

(Equation 1)

[0007]

However, SSDs
The method uses the original image or an image with a lower resolution than the original image.
If the overall brightness (luminance level) changes due to changes in the weather, lighting, or the like at the time of shooting, the corresponding area may not be accurately extracted due to the influence of the change in the luminance level. .

Further, in a region having few edges such as the sea and the sky, even if the corresponding regions do not coincide with each other, the SSD calculation result becomes low, and there is a possibility that the corresponding region is erroneously extracted.

Therefore, the present invention provides a method for extracting a corresponding area of an image which can accurately extract a corresponding area of two images without being affected by a change in luminance level or the presence of an area having few edges. It is an object of the present invention to provide an image synthesizing method for synthesizing an image using an area extracting method, an image synthesizing device, an image synthesizing program, and a recording medium storing the program.

[0010]

According to the present invention, there is provided a method for extracting a corresponding region of an image, wherein a second differential image is obtained by secondarily differentiating each of the two original images, and a difference between the original images is obtained based on the second differential image. The corresponding area is extracted. The secondary differential image obtained by secondarily differentiating the original image is one in which the change in the pixel value in each original image is emphasized, and the influence of the luminance level is removed.

Further, according to the method for extracting a corresponding region of an image of the present invention, the number of edges of at least one of the secondary differential images in the overlapping region is calculated while changing the overlapping region between the secondary differential images. An error map is obtained based on an error between the secondary differential images, and a corresponding region between the original images is extracted based on the number of edges and the error map.

More specifically, an overlap area where the number of edges is equal to or more than a predetermined value and an error map takes a minimum value is extracted, and a region of each original image corresponding to the overlap region is defined as a corresponding region between the original images. Is extracted as This allows
An overlapping area having a smaller number of edges than the predetermined value is not extracted as a corresponding area, and an overlapping area having the smallest error is extracted as a corresponding area from among the overlapping areas having a larger number of edges than the predetermined value.

[0013] More specifically, the number of edges is equal to or more than a predetermined value, the error map takes a minimum value, and the secondary differential value at the minimum value of the error map is maximum. The overlapping area is extracted, and the area of each original image corresponding to the overlapping area is extracted as a corresponding area between the original images. As a result, an overlapping area where the error changes most before and after the minimum value of the plurality of error maps is extracted as a corresponding area.

Specifically, the number of edges is equal to or more than a predetermined ratio with respect to the total number of edges in the original image;
The overlap area where the error map takes the minimum value is extracted, and the area of each original image corresponding to the overlap area is extracted as a corresponding area between the original images. As a result, an overlapping region having a smaller number of edges than the predetermined ratio is not extracted as a corresponding region from the original image, and an overlapping region having the least error is extracted as a corresponding region from among the overlapping regions having a larger number of edges than the predetermined ratio. Is done.

More specifically, the number of edges is equal to or more than a predetermined ratio with respect to the total number of edges in the original image, and
2 where the second derivative value at the minimum value of the error map is the maximum
An overlapping area of the next differential image is extracted, and an area of each original image corresponding to the overlapping area is extracted as a corresponding area between the original images. As a result, an overlapping area where the error changes most before and after the minimum value of the plurality of error maps is extracted as a corresponding area.

In the image synthesizing method according to the present invention, a second differential image is obtained by secondarily differentiating each of the first original image and the second original image.
A first step of extracting a corresponding area between the original images based on the secondary differential image, a second step of calculating a relational expression between the two original images in the corresponding area, and a first original image or It comprises a third step of converting the second original image to combine the two original images.

The first step is a step of calculating a second derivative image obtained by secondarily differentiating each of the original images. While changing an overlap area between the two second derivative images, at least one of the second derivative images in the overlap area is changed. Calculating the number of edges, obtaining an error map based on an error between the two secondary differential images in the overlapping area, and extracting a corresponding area between the original images based on the number of edges and the error map. Things.

More specifically, an overlap area where the number of edges is equal to or more than a predetermined value and an error map takes a minimum value is extracted, and an area of the original image corresponding to the overlap area is defined as a corresponding area between the original images. It is to extract.

More specifically, the number of edges is equal to or more than a predetermined value, the error map takes a minimum value, and the second derivative value at the minimum value of the error map becomes maximum. The overlapping area is extracted, and the area of each original image corresponding to the overlapping area is extracted as a corresponding area between the original images.

More specifically, the number of edges is equal to or more than a predetermined ratio with respect to the total number of edges in the original image, and
The overlap area where the error map takes the minimum value is extracted, and the area of the original image corresponding to the overlap area is extracted as a corresponding area between the original images.

More specifically, the number of edges is equal to or more than a predetermined ratio with respect to the total number of edges in the original image, the error map takes a minimum value, and the second derivative at the minimum value of the error map is obtained. The overlapping region of the secondary differential image having the maximum value is extracted, and the region of each original image corresponding to the overlapping region is extracted as a corresponding region between the original images.

The image synthesizing apparatus according to the present invention obtains a second differential image obtained by secondarily differentiating each of the first original image and the second original image,
First means for extracting a corresponding area between the original images based on the second derivative image, second means for calculating a relational expression between the two original images in the corresponding area, and a first original image or A third method of converting the second original image and combining the two original images
Means.

The first means is means for calculating a second derivative image obtained by secondarily differentiating each of the original images. While changing an overlap area between the two second derivative images, at least one of the second derivative images in the overlap area is changed. Means for calculating the number of edges, obtaining an error map based on an error between the two secondary differential images in the overlapping area, and means for extracting a corresponding area between the two original images based on the number of edges and the error map Things.

More specifically, an overlap area where the number of edges is equal to or more than a predetermined value and an error map takes a minimum value is extracted, and an area of the original image corresponding to the overlap area is defined as a corresponding area between the original images. It is to extract.

More specifically, the number of edges is equal to or more than a predetermined value, the error map takes a minimum value, and the second derivative value at the minimum value of the error map becomes maximum. The overlapping area is extracted, and the area of each original image corresponding to the overlapping area is extracted as a corresponding area between the original images.

More specifically, the number of edges is equal to or more than a predetermined ratio with respect to the total number of edges in the original image, and
The overlap area where the error map takes the minimum value is extracted, and the area of the original image corresponding to the overlap area is extracted as a corresponding area between the original images.

More specifically, the number of edges is equal to or more than a predetermined ratio with respect to the total number of edges in the original image, and
The overlapping area of the secondary differential image in which the error map has a minimum value and the secondary differential value at the minimum value of the error map is maximum is extracted, and the area of each original image corresponding to the overlapping area is extracted between the original images. Is extracted as the corresponding area of

The computer-readable recording medium on which the image synthesizing program of the present invention is recorded obtains a second differential image obtained by secondarily differentiating each of the first original image and the second original image.
A first step of extracting a corresponding area between the original images based on the secondary differential image, a second step of calculating a relational expression between the two original images in the corresponding area, and a first original image or An image synthesizing program including a third step of converting the second original image and synthesizing the two original images is recorded.

The first step is a step of calculating a second derivative image obtained by secondarily differentiating each of the original images. While changing an overlap area between the two second derivative images, at least one of the second derivative images in the overlap area is changed. Calculating the number of edges, obtaining an error map based on an error between the two secondary differential images in the overlapping area, and extracting a corresponding area between the original images based on the number of edges and the error map. Things.

More specifically, an overlap area where the number of edges is equal to or more than a predetermined value and an error map takes a minimum value is extracted, and an area of the original image corresponding to the overlap area is defined as a corresponding area between the original images. It is to extract.

More specifically, the number of edges is equal to or larger than a predetermined value, the error map takes a minimum value, and the secondary differential value at the minimum value of the error map becomes maximum. The overlapping area is extracted, and the area of each original image corresponding to the overlapping area is extracted as a corresponding area between the original images.

More specifically, the number of edges is at least a predetermined ratio to the total number of edges in the original image, and
The overlap area where the error map takes the minimum value is extracted, and the area of the original image corresponding to the overlap area is extracted as a corresponding area between the original images.

More specifically, the number of edges is a predetermined ratio or more to the total number of edges in the original image, and
The overlapping area of the secondary differential image in which the error map has a minimum value and the secondary differential value at the minimum value of the error map is maximum is extracted, and the area of each original image corresponding to the overlapping area is extracted between the original images. Is extracted as the corresponding area of

[0034]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An example of an image synthesizing apparatus using a corresponding area extracting method according to the present invention will be described below with reference to the drawings.
In the present embodiment, an optical flow estimation method used for image synthesis in the present embodiment will be described, and then an image synthesis device will be described. [1] Description of Optical Flow Estimation Method

The optical flow estimation method presupposes hierarchical estimation in which images having different resolutions are formed several times in a pyramid hierarchy and the optical flow is calculated stepwise. The optical flow calculation method is Lucas- Kanade
It assumes a gradient method such as a method. That is, the optical flow estimation method based on the hierarchically structured gradient method is assumed. Here, Lucas-Kanade method is used as the gradient method.

In the optical flow estimating method, the optical flows obtained at each stage of the hierarchically structured optical flow estimating method using the Lucas-Kanade method are complemented by dilation processing. Hereinafter, this will be described in detail. Lucas-Kanade Method For details of the Lucas-Kanade method, see B. Lucas and
T. Kanade, "An Iterative Image Registration Techniq
ue with an Application to Stereo Vision, "In Sevent
h International Joint Conference on Artificial Int
See elligence (IJCAI-81), pp. 674-979, 1981.

An outline of the Lucas-Kanade method will be described below.

The grayscale pattern I (x, y, t) of the image coordinates P = (x, y) at a certain time t is changed to the coordinates (x + δx) after a certain short time (δt).
, Y + δy) while maintaining the density distribution constant, the following optical flow constraint equation 2 holds.

[0039]

(Equation 2) Optical flow 次 元 v = (δx / δt,
In order to calculate δy / δt) = (u, v)｝, one constraint equation is required because the number of unknown parameters is two.
Lucas and Kanade (Kinade) assumed that local areas of the same object had the same optical flow.

For example, assuming that the optical flow is constant in the local region ω on the image, the square error E of the gray pattern to be minimized is I ₀ (p) = I (x, y, t), I ₁ (p + v) = I (x + u, y + v, t +
δt), it can be defined by the following equation (3).

[0041]

(Equation 3) Here, when v is very small, the second-order or higher-order term of the Taylor expansion can be ignored, so that the following equation 4 holds.

[0042]

(Equation 4) Here, g (p) is the first derivative of I ₁ (p).

Since the error E is minimized when the differential value of E with respect to v is 0, the following equation 5 holds.

[0044]

(Equation 5) Therefore, the optical flow v is obtained by the following equation (6).

[0045]

(Equation 6) In addition, as shown in the following equation 7, it can be obtained with high accuracy by a Newton-Raphson-like iterative operation.

[0046]

(Equation 7) Hierarchical Estimation Method First, an image having several levels of different resolutions is created for each image in advance from two continuous images. next,
A rough optical flow is calculated between the images having the lowest resolution. Then, referring to the result, a more precise optical flow is calculated between the images with higher resolution. This processing is sequentially repeated up to the image with the highest resolution.

FIG. 1D shows the original image, and FIG.
1D shows an image having a lower resolution than the original image, FIG. 1B shows an image having a lower resolution than the low-resolution image shown in FIG. 1C, and FIG. 1A shows a low-resolution image shown in FIG. An image with a lower resolution is obtained for each patch having a size (for example, 13 × 13 pixels). 1 (a) to 1 (d), S is
One patch is shown.

The image of FIG. 1A (the image of the hierarchy 1), FIG.
(B) image (layer 2 image), FIG. 1 (c) image (layer 3 image), and FIG. 1 (d) image (layer 4 image)
The optical flow is obtained step by step in the following order. In FIGS. 1A to 1D, arrows indicate the obtained optical flow vectors obtained for each patch. (3) Completion by dilation processing One of the advantages of the Lucas-Kanade method is that the tracking results have reliability. Tomasi and Kanade have shown that the traceability of a certain area can be calculated from differential images as follows (C. Tomasi and Kanade, "Shape and Motion from Im
age Streams: aFactorization method-Part 3 Detectio
n and Tracking of Point Features, "CMU-CS-91-132, C
arnegie Mellon University, 1991.).

From the 2 × 2 coefficient matrix G of the following equation (8) having the square of the vertical and horizontal differentiation of a certain area image ω as an element,
By calculating the eigenvalue, the traceability of the sled area can be determined.

[0050]

(Equation 8) When both eigenvalues of the matrix G are large, the area has a change in the orthogonal direction, and unique positioning is possible. Therefore, from the smaller eigenvalue λmin and the gray level residual E between the areas after tracking, the reliability γ of the tracking result is calculated by the following equation (9).
Can be obtained by

[0051]

(Equation 9) Then, an optical flow is obtained with high accuracy even in an area having a small number of patterns (textures) using the reliability. In this method, the result of the one-stage coarse hierarchy is used only for the initial value of tracking, and nothing is used for the result of the current stage of interest. Instead, it is assumed that the optical flow in an area with a small texture has a value close to the optical flow around the area, and the flow field is complemented by morphological processing.

FIG. 2 shows how the flow vector is expanded.

The left figure shows a map of the reliability of the flow vector by shading. Here, it is assumed that the darker the black, the higher the reliability.

First, the obtained flow is subjected to threshold processing. The white part has been thresholded due to the low reliability of the result.

Next, a dilation process of the result is performed in the flow field as follows, simulating a filling process by a morphological operation in a binary image. The flow vector u (i, j) of a certain region i, j can be calculated as in the following equation 10 by weighting the flow vectors in the vicinity of the four in accordance with the reliability γ.

[0056]

(Equation 10) This process is repeated until all the low reliability areas subjected to the threshold processing are filled. This complementing process is performed in each layer. Note that the flow vector of a certain region i, j
u (i, j) may be calculated by performing weighting according to the reliability γ from the eight neighboring flow vectors.

FIG. 3A shows an optical flow obtained by performing threshold processing on an image of a certain hierarchy.
(B) shows the optical flow after the complement. In FIG. 3A, an arrow is an optical flow vector determined to have high reliability by the threshold processing, and an x mark indicates a portion having low reliability. [2] Description of Image Synthesizing Apparatus FIG. 4 shows a configuration of the image synthesizing apparatus in the present embodiment.

A display 21, a mouse 22, and a keyboard 23 are connected to the personal computer 10. The personal computer 10 has a CPU 1
1, a memory 12, a hard disk 13, and a driver 14 for a removable disk such as a CD-ROM.

The hard disk 13 stores image compositing software in addition to an OS (operating system). The image synthesis software is installed on the hard disk 13 using the CD-ROM 20 in which the image synthesis software is stored. Also, the hard disk 13 has
It is assumed that a plurality of images captured by a digital camera are stored in advance.

FIG. 5 shows a procedure of a panoramic image synthesizing process performed by the CPU 11 when the panoramic image synthesizing software is activated.

Here, for convenience of explanation, a case where two images are combined will be described. (I) First, two images specified by the user (first image)
The image and the second image) are read into the memory 12 (step 1). (II) Next, the corresponding area is extracted from the two images (step 2). This corresponding area extraction processing
This is performed according to the procedure shown in FIG.

First, the secondary differential images I ₁ and I _{2 obtained} by _{performing the} secondary differential processing on the two images I _1org and I _2org, respectively.
Is generated. In each of the second derivative images I ₁ and I ₂ , the original image I
_The pixel values greatly change in the positive and negative directions around portions corresponding to edges in _1org and _I2org . (Step 2
1) When the secondary differential images I ₁ and I ₂ are binarized with a predetermined threshold value, only the peripheral portion of the edge has a logical value of 1 and the other portions have a logical value of 0. Thus, edge detection is performed. (Step 22) Next, one secondary differential image I ₂ is replaced with the other secondary differential image I ₁
Each of the secondary differential images included in the overlapping area ω (size: M × N) where the two secondary differential images I ₁ and I ₂ overlap while moving two-dimensionally in the x direction and the y direction one pixel at a time. Number a1, a of edges (pixels having logical value 1) of I ₁ and I ₂
Count 2. However, assuming that the amount of movement of the secondary differential image I ₂ is (d), the amount of movement (d) is changed within a range where the two secondary differential images I ₁ and I ₂ have an overlapping area. (Step 23) In addition, while the number of edges is counted, 2 in the overlapping area ω (size: M × N) for each movement amount (d).
The square error (E) between the next differential images I ₁ and I ₂ is given by
Is calculated using

[0063]

[Equation 11] Then, based on this error (E), for example, an error map as shown in FIG. 9 is obtained. Each line segment in this error map is obtained by connecting coordinates corresponding to the amount of movement at which the error value in the overlap region is the same. In the region between the two line segments, the error value between the two line segments is Take the value of (Step 24) Finally, the number of edges a1 obtained for each movement amount (d),
Based on a2 and the error map, an overlapping area satisfying the following conditions Z1 and Z2 is extracted as a corresponding area.

Z1: the number of edges a1, a2 in the overlapping area ω of the secondary differential images I ₁ , I _{2 in} the movement amount (d)
Is a predetermined ratio α or more with respect to the total number of edges in each original image.

Z2: Movement amount (d) in the error map
Takes the minimum value (the minimum value in a region surrounded by a single line segment and having a smaller error value than the surroundings in FIG. 9), and The value obtained by secondarily differentiating the error map at the minimum value becomes the maximum. (Step 25) Since the overlapping area of the original image is extracted using the secondary differential image, the influence of the difference in the luminance level between the original images is eliminated, and the corresponding area can be extracted with high accuracy. It becomes possible.

Also, since the number of edges in the overlapping area as a corresponding area is extracted at a predetermined ratio α or more with respect to the total number of edges in each original image, an edge including a lot of sea and sky is extracted. It is possible to perform highly reliable extraction of a corresponding area without an erroneous extraction of an overlapping area having a small area.

Further, depending on the image to be synthesized and the value of the threshold value α of the edge, the error value may be smaller in the overlapping region with a small number of edges than in the overlapping region with a large number of edges. Since the change rate (slope) of the value of the error map before and after the minimum value serving as the corresponding area is larger than the slope before and after the other minimum values, the corresponding area is more accurately extracted. Becomes possible.

Here, when the overlapping area of the original image is extracted using the secondary differential image, the error map between the secondary differential images takes a minimum value, and the error map is converted to a secondary value at the minimum value. Although the case where the overlap area where the differentiated value is the maximum is extracted as the corresponding area has been described, the overlap area having the minimum value in the error map between the secondary differential images may be extracted as the corresponding area.

Further, here, the threshold value α of the edge is set to a predetermined ratio with respect to the total number of edges in each original image. However, the threshold value α may be a fixed value as the number of edges.

Further, here, the number of edges in the overlapping area is obtained for both images to be synthesized, but may be obtained for one of the images. (III) Next, feature point extraction is performed (step 3). That is, a plurality of partial images (rectangular regions) effective for tracking are extracted as feature points from the corresponding region of the first image. However, each feature point is extracted so as not to overlap each other. Specifically, a portion having a high reliability λmin is extracted as a feature point. (IV) Next, feature point tracking processing is performed (step 4).
That is, the positions of the extracted feature points of the first image on the second image are tracked.

Specifically, first, an appropriate size (for example, 1) is obtained by the optical flow estimation method described in [1].
An optical flow vector for each 3 × 13) patch is obtained. The positions on the second image corresponding to the feature points of the first image are obtained in pixel units or less by linear interpolation from the flow vectors of the patches near the four feature points of the first image. (V) Next, a plane projection transformation matrix is calculated (step 5).

If the target scene is far away, or if it is close to buildings, walls, blackboards, etc., it can be assumed that these are single planes. As shown in FIG.
A point M on a single plane in a dimensional space is defined by two different viewpoints C ₁ ,
When viewed from the C _2, the coordinates m _1, m of these in each image plane
Transformations between _two are known in projective geometry to be linear and are called homography (O. Faugera
s, "Three-Dimention Computer Viaion: a Geometric Vi
ewpoint ", MIT press, 1993.).

That is, the point m1 = (x1, y1, 1) t of the first image, which represents the image coordinates in homogeneous coordinates, is the corresponding point m2 = (x2, y2, 1) t on the second image. And their relationship is defined by the following equation (12).

[0074]

(Equation 12) Here, it is shown that they are projectively equal, and the scale factor is left. This transformation matrix can be rewritten as in the following equation (13).

[0075]

(Equation 13) (VI) Based on the obtained plane projection transformation matrix, an image is generated by superimposing the two images so that the positions of the two images coincide (step 6). (VII) Then, pixel value adjustment is performed on a portion where the two images overlap each other (hereinafter, referred to as a superimposed portion) (step 7). That is, the first image and the second image are:
Since the shooting conditions are not the same, the other may be darker than the other. Therefore, the pixel value of the point of interest in the overlapping portion of the two images is determined based on the distance from both ends of the overlapping portion of the two images to the point of interest, and the pixel value of the first image of the point of interest and the second pixel value of the point of interest. The pixel value of the image is a combined value.

As described above, according to the present embodiment, it is possible to perform image synthesis with high accuracy as the corresponding area between images to be synthesized is extracted with high accuracy. [3] Description of Other Application Examples The panoramic image synthesizing software used in the image synthesizing apparatus can be commercialized as a single unit. Further, the panoramic image synthesizing program can be incorporated into a printing device of a digital still camera image printing lab, a printer for outputting a digital camera image, a digital camera, a copying machine, and the like.

FIG. 8 shows the configuration of a digital still camera in which a panorama image synthesizing program is incorporated.

This digital still camera has a CCD 1, an A / D converter 2,
A video processing unit 3, a CPU 4, a ROM 5, a flash memory 6, an operation unit 7, and the like are provided. The panorama image synthesizing program is stored in the ROM 5.

By activating the panoramic image synthesizing program by operating the operation unit 7, a plurality of images already taken into the flash memory 6 are synthesized by the method described in [2]. One panoramic image is generated. The generated panoramic image is stored in the flash memory 6.

[0080]

According to the present invention, there is provided a method for extracting a corresponding area of an image which can accurately extract a corresponding area of two images without being affected by a change in luminance level, and a method for extracting the corresponding area. , An image synthesizing apparatus, an image synthesizing program, and a recording medium storing the program.

[Brief description of the drawings]

FIG. 1 is a schematic diagram for explaining a hierarchical estimation method.

FIG. 2 is a schematic diagram for explaining an expansion process performed in the optical flow estimation method according to the present invention.

FIG. 3 is a schematic diagram illustrating an example of an optical flow obtained by performing threshold processing on an image of a certain hierarchy and an optical flow after interpolation.

FIG. 4 is a block diagram illustrating a configuration of a panoramic image synthesizing apparatus.

FIG. 5 is a flowchart illustrating a panoramic image synthesis processing procedure;

FIG. 6 is a flowchart showing a procedure for extracting a corresponding region in the flowchart of FIG. 5;

FIG. 7 is an explanatory diagram for explaining a plane projection matrix.

FIG. 8 is a block diagram showing a configuration of a digital still camera in which a panorama image synthesizing program is incorporated.

FIG. 9 is a graph showing an example of an error map.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Personal computer 11 CPU 12 Memory 13 Hard disk 14 Removable disk driver 21 Display 22 Mouse 23 Keyboard

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Hiroshi Kamosono 2-5-5 Keihanhondori, Moriguchi-shi, Osaka Sanyo Electric Co., Ltd. F-term (reference) 5B057 CA12 CA16 CC01 CD02 CD03 CE20 DA08 DA20 DC16 DC30 5C076 AA19

Claims (25)

[Claims] 1. A second derivative of each of two original images
A method for extracting a corresponding area of an image, wherein a corresponding area between the original images is extracted based on the second derivative image. 2. The method according to claim 1, further comprising calculating an edge number of at least one of the secondary differential images in the overlapping area while changing an overlapping area between the secondary differential images, and based on an error between the secondary differential images in the overlapping area. A method for extracting a corresponding area of an image, wherein an error map is obtained and a corresponding area between original images is extracted based on the number of edges and the error map. 3. An overlapping area where the number of edges is equal to or more than a predetermined value and the error map takes a minimum value is extracted, and a region of each original image corresponding to the overlapping region is defined as a corresponding region between the original images. 3. The method according to claim 2, wherein the corresponding area is extracted. 4. An overlapping area of a secondary differential image in which the number of edges is equal to or more than a predetermined value, the error map has a minimum value, and a secondary differential value at the minimum value of the error map is maximum. 3. The method according to claim 2, wherein a region of each original image corresponding to the overlapping region is extracted as a corresponding region between the original images. 5. An overlapping region in which the number of edges in the previous period is equal to or more than a predetermined ratio with respect to the number of all edges in the original image in the previous period, and the error map takes a minimum value is extracted, and the extracted region corresponds to the overlapping region. 3. The method according to claim 2, wherein the area of each original image is extracted as a corresponding area between the original images. 6. A secondary differential image in which the number of edges in the previous period is equal to or more than a predetermined ratio with respect to the number of all edges in the original image in the previous period, and the second derivative at the minimum value of the error map is maximum. 3. The method according to claim 2, wherein the overlapping region is extracted, and a region of each original image corresponding to the overlapping region is extracted as a corresponding region between the original images. 7. A first step of obtaining a second differential image obtained by secondarily differentiating each of the first original image and the second original image, and extracting a corresponding region between the original images based on the second differential image, Image synthesis including a second step of calculating a relational expression between the two original images in the corresponding region, and a third step of converting the first original image or the second original image based on the relational expression and synthesizing the two original images Method. 8. The method according to claim 1, wherein the first step comprises:
Calculating the second-order differentiated image by performing second-order differentiation, calculating the number of edges of at least one second-order differential image in the overlap region while changing the overlap region between the two second-order differential images, 8. The image synthesizing method according to claim 7, further comprising: obtaining an error map based on an error between the differential images; and extracting a corresponding region between the two original images based on the number of edges and the error map. 9. An overlapping area where the number of edges is equal to or more than a predetermined value and the error map has a minimum value is extracted, and an area of an original image corresponding to the overlapping area is extracted as a corresponding area between the original images. 9. The image synthesizing method according to claim 8, wherein: 10. An overlapping area of a secondary differential image in which the number of edges is equal to or more than a predetermined value, the error map takes a local minimum value, and a secondary differential value at the local minimum value of the error map is maximum. 9. The image synthesizing method according to claim 8, further comprising extracting a region of each original image corresponding to the overlapping region as a corresponding region between the original images. 11. An overlapping area in which the number of edges in the previous period is equal to or greater than a predetermined ratio with respect to the number of all edges in the original image in the previous period, and the error map takes a minimum value is extracted. 9. The image synthesizing method according to claim 8, wherein an area of the original image to be extracted is extracted as a corresponding area between the original images. 12. A secondary differential image in which the number of edges in the previous period is equal to or more than a predetermined ratio with respect to the number of all edges in the original image in the previous period, and the second derivative at the minimum value of the error map is maximum. 9. The image synthesizing method according to claim 8, wherein an overlapping area is extracted, and an area of each original image corresponding to the overlapping area is extracted as a corresponding area between the original images. 13. Each of the first original image and the second original image is 2
First means for obtaining a second-order differentiated image obtained by performing second-order differentiation, and extracting a corresponding area between the original images based on the second-order differential image; second means for calculating a relational expression between the two original images in the corresponding area; An image synthesizing apparatus comprising third means for converting the first original image or the second original image based on the relational expression and synthesizing the two original images. 14. The first means is means for calculating a second derivative image obtained by secondarily differentiating each of the original images, and changing at least one of the two overlapped images while changing an overlap area between the two second derivative images. Means for calculating the number of edges of the secondary differential image and for obtaining an error map based on the error between the two secondary differential images in the overlapping area, and determining the corresponding area between the two original images based on the number of edges and the error map. 14. The image synthesizing device according to claim 13, further comprising: means for extracting. 15. An overlapping area where the number of edges is equal to or more than a predetermined value and the error map takes a minimum value is extracted, and an area of an original image corresponding to the overlapping area is extracted as a corresponding area between the original images. The image synthesizing device according to claim 14, wherein 16. An overlapping area of a secondary differential image in which the number of edges is equal to or more than a predetermined value, the error map takes a local minimum value, and a secondary differential value at the local minimum value of the error map is maximum. 15. The image synthesizing apparatus according to claim 14, wherein the image synthesizing unit extracts a region of each original image corresponding to the overlapping region as a corresponding region between the original images. 17. An overlapping area in which the number of edges in the previous period is equal to or more than a predetermined ratio with respect to the number of all edges in the original image in the previous period and the error map takes a minimum value is extracted, and the extracted region corresponds to the overlapping region. The image synthesizing apparatus according to claim 14, wherein an area of the original image is extracted as a corresponding area between the original images. 18. A secondary differential image in which the number of edges in the previous period is equal to or more than a predetermined ratio with respect to the number of all edges in the original image in the previous period, and the second derivative at the minimum value of the error map is maximum. 15. The image synthesizing apparatus according to claim 14, wherein an overlapping area is extracted, and an area of each original image corresponding to the overlapping area is extracted as a corresponding area between the original images. 19. Each of the first original image and the second original image is 2
A first step of obtaining a second-order differential image obtained by performing a second-order differentiation, and extracting a corresponding region between the original images based on the second-order differential image;
Calculating a relational expression between the two original images in the corresponding area;
A computer-readable recording medium storing an image synthesizing program, comprising: a step of converting a first original image or a second original image based on the relational expression to synthesize the two original images. 20. The method according to claim 19, wherein the first step is a step of calculating a second derivative image obtained by secondarily differentiating each of the original images, and changing at least one of the two overlapped images while changing the overlap region between the two second derivative images. Calculating the number of edges of the secondary differential image and obtaining an error map based on the error between the two secondary differential images in the overlapping area; and determining the corresponding area between the two original images based on the number of edges and the error map. 20. A computer-readable recording medium recording the image synthesizing program according to claim 19, further comprising an extracting step. 21. An overlapping area in which the number of edges is equal to or more than a predetermined value and the error map has a minimum value is extracted, and an area of an original image corresponding to the overlapping area is extracted as a corresponding area between the original images. A computer-readable recording medium on which the image synthesizing program according to claim 20 is recorded. 22. An overlapping area of a secondary differential image in which the number of edges is equal to or more than a predetermined value, the error map takes a minimum value, and a secondary differential value at the minimum value of the error map is maximum. 21. A computer-readable recording medium in which an image synthesizing program according to claim 20, wherein an area of each original image corresponding to the overlapping area is extracted as a corresponding area between the original images. 23. An overlap area in which the number of edges in the previous period is equal to or more than a predetermined ratio with respect to the number of all edges in the original image in the previous period and the error map takes a minimum value is extracted, and the overlap region corresponds to the overlap region. 21. A computer-readable recording medium storing an image synthesizing program according to claim 20, wherein an area of the original image is extracted as a corresponding area between the original images. 24. A secondary differential image in which the number of edges in the previous period is equal to or more than a predetermined ratio with respect to the number of all edges in the original image in the previous period, and the second derivative at the minimum value of the error map is the largest. 25. A computer-readable recording medium recording an image synthesizing program according to claim 24, wherein an overlapping area is extracted, and an area of each original image corresponding to the overlapping area is extracted as a corresponding area between the original images. 25. Each of the first original image and the second original image is 2
A first step of obtaining a second-order differential image obtained by performing a second-order differentiation, and extracting a corresponding region between the original images based on the second-order differential image;
Calculating a relational expression between the two original images in the corresponding area;
An image synthesizing program, comprising: a step of converting a first original image or a second original image based on the relational expression to synthesize the two original images.