JPH1083442A - Picture synthesis method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To precisely synthesize a plurality of pictures in a short time without the need of a troublesome operation by reducing the plurality of pictures stepwise and extracting stepwise corresponding points in the pictures which they indicate. SOLUTION: A corresponding point extraction part 20 sequentially extracts the corresponding points on reduced picture data in the picture sides of 1/8-fold, 1/4-fold, 1/2-fold and life size on the sizes of the pictures which inputted picture data (a) and (b) show. In this case, prediction is executed in an overlap part by using a converted parameter formed, based on the extraction of the corresponding points, which is executed before. A corresponding point vector selection part 30 selects and outputs only a highly reliable corresponding point vector. A picture conversion/synthesis part 50 coordinate-converts inputted picture data (a) and (b) in accordance with the coordinate conversion parameter estimated in a parameter estimation part 40, synthesizes them and forms synthesized picture data (c) showing the synthesized picture.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing method, and more particularly, to an image synthesizing method for forming an image having a wide angle of view by synthesizing two images whose images partially overlap each other. .

[0002]

2. Description of the Related Art Conventionally, as a method of forming an image having a wide angle of view from two images whose images partially overlap each other, the same point in an overlapping area of the two images coincides with each other. There is known a method of performing geometric transformation such as affine transformation on the two images and joining the two images on a plane.

[0003]

However, in the above-described conventional method, it is necessary to recognize an overlapping area between two images and extract the same point (hereinafter referred to as a corresponding point) in the two images. There is a method of displaying two images to be combined on a display and operating a mouse or the like to specify a corresponding point in the two images with a cursor. In the case of (1), there is a problem that the operability is poor because the operation is manual, and that the operation requires skill.

Therefore, as a method of automatically extracting corresponding points in two images, one image is divided into small regions, each divided small region is compared with the other image, and each of the small regions is compared with each other. In contrast, there is a template matching method in which the most similar area in the other image is a corresponding point.

[0005] However, in the case of the template matching method, if an overlapping area of two images to be combined is not specified, an erroneous corresponding point is extracted in an area other than the overlapping area of the two images. As a result, the extraction accuracy of the corresponding points deteriorates. Then, when the extraction accuracy of the corresponding points is poor, the two images cannot be connected accurately.

[0006] In the case of the template matching method, when an overlapping area of two images to be combined is specified, if the area specified as a search area for a corresponding point in the other image is wide, the other If there are many similar parts in the image, it takes time to extract an accurate corresponding point, and a problem arises that two images cannot be connected quickly.

According to the present invention, for example, even when there is a difference between the levels of the luminance components of a plurality of images represented by a plurality of input image data, a complicated operation is not required, and a plurality of images are automatically processed in a short processing time. It is an object of the present invention to provide an image synthesizing method capable of synthesizing the image with high accuracy without causing the seam of the image to be shifted so much.

[0008]

In order to achieve the above-mentioned object, an image synthesizing method according to a first aspect of the present invention includes:
A method of synthesizing a plurality of images having image areas partially overlapping each other to create one synthesized image, and inputting a plurality of image data each corresponding to a plurality of images, A reduced image data forming step of reducing a plurality of images indicated by the plurality of image data in a stepwise manner at different reduction ratios to form a plurality of reduced image data indicating the reduced images formed respectively; For the reduced image data formed in the reduced image data forming step, from the reduced image data indicating the reduced image having the highest reduction ratio, to the same size image indicating an image of the same size as the image indicated by the input plurality of image data A corresponding point extracting step of sequentially extracting corresponding points between the images indicated by the data, and extracting the corresponding points in the corresponding point extracting step. A parameter setting step of setting various parameters necessary for performing image synthesis conversion according to the corresponding points, and a reduced image indicated by the reduced image data in the corresponding point extracting step. If it is set for, using the corresponding points already extracted in the corresponding point extraction step and various parameters set in the parameter setting step,
Predicting an overlapping portion of a plurality of reduced images indicated by reduced image data whose reduction ratio is one step higher, setting a small area as a template for obtaining a corresponding point from one reduced image of the plurality of reduced images, In the corresponding point extracting step, a corresponding point is extracted from among a plurality of reduced images indicated by reduced image data having a reduction ratio one step higher, and the corresponding point is set for an image indicated by the same size image data in the corresponding point extracting step. In the case, according to various parameters set in the parameter setting step, an image conversion / combining step of converting at least one of a plurality of images indicated by the plurality of input image data and combining with another image And characterized in that:

The image synthesizing method according to the second aspect of the present invention is a method of synthesizing a plurality of images having image areas partially overlapping each other to create one synthesized image. A plurality of image data, each corresponding to a plurality of images, are input, and the plurality of images indicated by the input plurality of image data are reduced stepwise at different reduction rates, and each of the plurality of images is reduced. A reduced image data forming step of forming a plurality of reduced image data indicating the formed reduced image; and a feature point of the reduced image is extracted from the reduced image data formed in the upper reduced image data forming step. A feature point reduced image forming step of forming a reduced image; and a reduction of the feature point reduced image data formed in the feature point image formation step, the reduction rate being the highest. Corresponding points between the feature point images indicated in order from feature point reduced image data indicating an image to feature point equal-size image data indicating a feature point image having the same size as the image indicated by the plurality of input image data A corresponding point extraction step of extracting step by step, a parameter setting step of setting various parameters necessary for performing image synthesis conversion according to the corresponding points extracted in the corresponding point extraction step, and a setting step of the parameter setting step. If the set various parameters are set for the feature point reduced image indicated by the feature point reduced image data in the corresponding point extraction step, the corresponding points already extracted in the corresponding point extraction step and the parameter setting Using the various parameters set in the step, the reduction ratio is one step higher Predicting an overlapping portion of the plurality of reduced feature point images indicated by the reduced image data, setting a small area for obtaining a corresponding point from one reduced feature point image of the plurality of reduced reduced feature points as a template, In the corresponding point extraction step, a corresponding point is extracted from a plurality of reduced feature point images represented by the reduced feature point image data having a reduction ratio one step higher. If it is set, according to various parameters set in the parameter setting step, convert at least one of the plurality of images indicated by the plurality of input image data, and other images And an image conversion / combining step of combining.

(Operation) By the above-mentioned operation, a plurality of images can be automatically synthesized with high accuracy without a troublesome operation and with a short processing time.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION

(First Embodiment) FIG. 1 shows a configuration of an image synthesizing apparatus according to a first embodiment of the present invention.

In the figure, input terminals 10a and 10b are image data a and image data b, respectively, indicating an image to be synthesized, and are not shown so that a part of the image may be overlapped by an electronic still camera, a video camera or the like (not shown). For example, the image data a is generated by being imaged in FIG.
The image data b corresponds to an image as shown in FIG. 2B.

Numerals 11 and 12 denote image reduction units, for example, from input image data a and b to に 対 し て times, 倍 times and 1 / times of the size of the image indicated by the image data a and b. 8
The reduced image data corresponding to the reduced image obtained by reducing the size of the image such as double is hierarchically formed, and the formed reduced image data is stored in the image memories in the image reduction units 11 and 12. .

Reference numeral 20 denotes a corresponding point extracting unit, and the input terminal 10
A corresponding point between the respective images is extracted from the image data a, b input from a, 10b or the respective images indicated by the reduced image data formed in the image reduction units 11, 12.

Reference numeral 30 denotes a corresponding point vector selecting unit which selects and outputs only a highly reliable corresponding point vector from the corresponding point vectors indicated by the corresponding points extracted by the corresponding point extracting unit 20.

Reference numeral 40 denotes a parameter estimating unit for estimating a coordinate conversion parameter for performing coordinate conversion of an image from the corresponding point vector selected by the corresponding point vector selecting unit 30.

Reference numeral 50 denotes an image conversion / synthesis unit, which performs coordinate conversion on the input image data a and b in accordance with the coordinate conversion parameters estimated by the parameter estimating unit 40 and synthesizes the data, thereby indicating one synthesized image. The composite image data c is formed.

The control of the whole image synthesizing apparatus is performed by a control unit (not shown).

The operation of the image synthesizing apparatus according to the first embodiment of the present invention in the case where the synthesized image data c is formed from the input image data a and b will be described below.

In the present embodiment, input image data a and b will be described below as image data representing a gray image of 512 × 512 pixels.

First, the input image data a and b are, for example, １ ／ times the size of the image indicated by the image data a and b in the image reduction units 11 and 12, respectively.
Reduced image data corresponding to a reduced image obtained by reducing the size of the image such as ４ times or ８ times, that is, 256 × 256 pixels, 128 × 128 pixels, 64
It is reduced to reduced image data of an image size of × 64 pixels (these reduced image data are hereinafter referred to as a2, b2, a4, b
4, a8, b8).

First, the corresponding point extracting unit 20 sets the size of the image indicated by the input image data a and b to 1
Corresponding points are extracted from reduced image data a8 and b8 indicating a / 8-times reduced image, and then, １ ／ times, 倍 times,
Corresponding point extraction is performed on reduced image data of the same size of the image. When extracting corresponding points for each of the reduced image data having the size of 1/4, 1/2 and 1 times, the corresponding points are extracted based on the corresponding points extracted before that as described later. The prediction in the overlapping portion is performed using the formed conversion parameters.

Hereinafter, the corresponding point extraction algorithm in the corresponding point extraction unit 20 will be described with reference to the flowchart shown in FIG.

In FIG. 3, at s21, a template cutout area is first set for the reduced image data a8. By the way, when the corresponding points are extracted from the reduced image data a8 and b8, a predetermined region is set as a template cutout region because the overlapping region of the two reduced images is not known in advance.

In the present embodiment, when the input images indicated by the input image data a and b are arranged in the order of left and right as shown in FIG. 2, for example, the right end in the horizontal direction of the image is used. An area of 30% and 10% to 90% in the vertical direction of the image is defined as a template cutout area (area T in FIG. 4) in the reduced image data a8 on the left side.

Next, at s22 of FIG. 3, s2 of FIG.
As shown by the broken line in FIG. 4, that is, a block whose size of one block is about 10% of the original image size, that is, about 7 × 7 pixels is cut out from the template cutout area set in 1 as a template.

When the input images indicated by the input image data a and b are arranged in the order of top and bottom, 30% from the bottom in the vertical direction of the image and 10% from the bottom in the horizontal direction of the image. From the reduced image data a8 on the upper side
It is assumed that the template is set as the template cutout area in and the template is cut out.

As described above, for all templates cut out in s22 of FIG.
25 is performed.

In s23 of FIG. 3, for each template cut out as described above, a search area for searching for a corresponding point in the image indicated by the reduced image data b8 on the right side is set. By the way, reduced image data a
When the corresponding points are extracted for b8 and b8, a predetermined area is set as a corresponding point search area because the overlapping area of the two reduced images is not known in advance.

In the present embodiment, as described above, a search area for a corresponding point in the right reduced image data b8 with respect to an arbitrary template among a plurality of templates cut out from the left reduced image data a8 is used. In the horizontal direction, the area from the left end to the position shifted from the position where the arbitrary template exists by about 50% of the size of the template to the left, and in the vertical direction of the image, the position where the arbitrary template exists is defined as the position of the template. The area about 10% of the size up and down,
It is set as a search area in the reduced image data b8 on the right side.

FIG. 5 shows an example of a search area for a corresponding point in the right reduced image data b8 (corresponding to the right reduced image data b8 corresponding to the hatched template of the left reduced image data a8 in FIG. 5). The point search area is indicated by area S).

The setting of the search area for the corresponding point is such that the overlapping area of the image indicated by the input image data a and b is 50% or less of the input image size in the horizontal direction, and the input image size in the vertical direction. Are based on the condition that there is no shift, and the input image data a,
If the condition of the overlap area assumed in the image indicated by b is different from the condition, the initial setting of the search area of the corresponding point may be changed.

In s24 of FIG. 3, the image in the search area for the corresponding point in the right reduced image data b8 set as described above and the image indicated by an arbitrary template in the left reduced image data a8 are compared with the arbitrary image. The position of the template is sequentially shifted and compared, and a difference value between pixels constituting each image in the image indicated by the template and the image in the search area of the corresponding point is calculated. The position of the template where the sum of the absolute values of the difference values is minimum is detected as the corresponding point.

In this embodiment, the position of the template at which the sum of the absolute values of the differences is minimum is detected as the corresponding point. For example, the correlation calculation is performed between the image indicated by the template and the image in the search area for the corresponding point. And the position of the template at which the correlation value becomes maximum as a result of the correlation operation may be detected as the corresponding point.

At s25 in FIG. 3, the reliability of the corresponding point detected at s24 is determined. Note that the reliability is determined using the position of the template where the sum of the absolute values of the difference values is the smallest and the position of the template where the sum of the absolute values of the difference values is the second smallest.

That is, in the case of this embodiment, the minimum value of the sum of the absolute values of the difference values is equal to or less than a predetermined threshold value,
And if the difference between the minimum value and the sum of the absolute values of the difference values is the second smallest value is equal to or greater than a predetermined threshold value, it is determined that the detected corresponding point is reliable, Data relating to the coordinates of the corresponding points extracted in the images indicated by the reduced image data a8 and b8 is stored in the memory inside the corresponding point extracting unit 20.

The processing from s23 to s25 is performed in s2
Until it is detected in 6 that the processing in all the templates is completed, the processing is performed for each template cut out in s21.

Next, the processing of the corresponding point vector selecting section 30 will be described.

In the corresponding point vector selecting section 30, first, the coordinates of each corresponding point extracted in the image indicated by the reduced image data a8 and b8 extracted by the corresponding point extracting section 20 and stored in the internal memory are obtained. Read the data, calculate the corresponding point vector by calculating the difference between the horizontal direction and the vertical direction of the respective coordinates indicated by the read data, and calculate the histogram of all the calculated corresponding point vectors in the horizontal direction of the coordinates, Create for each vertical direction. Then, in the horizontal direction and the vertical direction of the coordinates, only the corresponding point vectors belonging to the most frequently occurring range are selected from the histogram of the created corresponding point vectors, and the data related to the selected corresponding point vectors are converted into the parameters of the next stage. Output to the estimation unit 40.

As described above, the corresponding point vector selection unit 30
By selecting only the corresponding point vectors belonging to the most frequently occurring range, even if an erroneous corresponding point is extracted in the corresponding point extracting unit 20, only the data relating to the more reliable corresponding point vector is used as a parameter. Estimation unit 4
0 can be output.

The parameter estimating unit 40 estimates the coordinate conversion parameters from the coordinates of the corresponding points of the reduced image data a8 and b8 indicated by the data on the corresponding point vectors supplied from the corresponding point vector selecting unit 30.

The estimation of the coordinate conversion parameters will be described later in detail.

The data relating to the corresponding point vectors obtained for the reduced image data a8 and b8 representing the reduced image having an image size 1/8 times the size of the input image obtained as described above is the input image to be executed next. This is used when predicting an overlapping area of two reduced images in corresponding point extraction processing of reduced image data a4 and b4 indicating reduced images having an image size that is times the image size.

That is, the data relating to the corresponding point vectors obtained for the reduced image data a8 and b8 are averaged in the horizontal and vertical directions by the parameter estimating section 40, and the reduced image data a4 and b4 are calculated.
Are output to the corresponding point extracting unit 20 for use in predicting the overlapping area of the two reduced images in the corresponding point extracting process.

Similarly, data relating to the corresponding point vector obtained with respect to the reduced image data representing the reduced image having the image size of ４ times and ２ times the two corresponding points in the corresponding point extraction processing of the next image data. It is used when predicting an overlapping area of an image.

As described above, the coordinate conversion parameters of the reduced image data a8 and b8 are
When output from 0, the corresponding point extracting unit 20 generates reduced image data a4, b4 indicating a reduced image having a 1 / 4-fold image size.
Corresponding points are extracted.

Note that the corresponding point extraction algorithm of the corresponding point extraction unit 20 for the reduced image data a4 and b4 indicating a reduced image having an image size 1/4 times that of the input image is shown in FIG.
However, the setting of the template cutout area in s21 and the setting of the corresponding point search area in s23 in FIG.
The average value in the horizontal direction and the vertical direction of the coordinates of the data regarding the corresponding point vector of the reduced image data a8 and b8 indicating the reduced image having an image size 1/8 times as large as the input image output from is used.

FIG. 6 is a diagram for explaining a method of setting a template cutout area for reduced image data a4.

The setting of the template cut-out area in s21 of FIG. 3 is performed in the horizontal direction of the coordinates of the data relating to the corresponding point vector output from the parameter estimating unit 40.
From the data indicating the average value in each of the vertical directions (arrows in FIG. 6), an overlapping area (shaded portion in FIG. 6) in the reduced image indicated by the reduced image data a4 is predicted, and the predicted reduced image data a4 is obtained. From the overlapping area of the reduced image shown, the positions of the left end, the upper end, and the lower end of the template cutout area for the reduced image data a4 are set, and the template is cut out.

FIG. 7 is a diagram for explaining a method of setting a search area for a corresponding point for reduced image data b4.

In s23 of FIG. 3, the reduced image data a is set for each of the plurality of templates cut out in s21.
4 is a point shifted from the position where the template exists in the image indicated by 4 by the average value in the horizontal and vertical directions of the coordinates of the data on the corresponding point vector of the reduced image data a8 and b8 supplied from the parameter estimating unit 40. (Points of arrows in FIG. 7) are set, and a predetermined width (for example, 5 pixels in each of up, down, left, and right directions) in the vertical and horizontal directions around the set point (the area S in FIG. 7). ′)
Only the corresponding point is set as a search area.

Hereinafter, corresponding point extraction is performed on reduced image data a4 and b4 indicating reduced images of 1/4 times the image size.
Reduced image data a8, b indicating selection of corresponding point vectors and parameter estimation indicating a reduced image having an image size of 1/8 of the aforementioned size
8 is performed in the same manner.

The corresponding point extraction processing for image data representing an image having an image size of 1/2 and 1 times the size of the input image is also performed in the reduced image data representing a reduced image having a 1/4 time image size. Is performed in the same manner as in the case of (a), and image data a2 and b
2, corresponding point extraction, selection of a corresponding point vector, and parameter estimation are sequentially performed. Next, corresponding point extraction processing is performed on image data a and b indicating images of the same size as the input image. And parameter estimation are sequentially performed hierarchically.

By the way, in the above-described processing, image data a and b indicating images of the same size as the input image are subjected to coordinate transformation by affine transformation in image transformation / combination processing performed as post-processing. , The horizontal direction of the coordinates,
Since the corresponding point vector in the vertical direction does not always fall within a certain range, the selection of the corresponding point vector based on the histogram need not be performed.

Here, the parameter estimation processing will be described in detail.

After the coordinate conversion parameters for the two images represented by the input image data a and b are obtained as described above, finally, the input image data a and b are Coordinate conversion is performed according to the coordinate conversion parameters estimated by the parameter estimating unit 40, and the images are combined into one image.

The coordinate transformation performed at this time is an affine transformation, and a coordinate transformation parameter is estimated from data on a corresponding point vector of an image of the same size as the image represented by the input image data a and b. In
The parameter estimating unit 40 outputs the parameters of the affine transformation by the following method instead of the average value of the data on the corresponding point vector.

Now, the image indicated by the input image data b is rotated by θ with respect to the image indicated by the input image data a, (dx, dy)
Assuming that the relationship has been obtained by performing parallel translation and magnifying conversion by a factor of m, the point (xa, xa) in the image indicated by the input image data a is the point (x) in the image indicated by the input image data b in the following equation (1).
b, yb).

Xb = (cos θ · xa + sin θ · ya−dx) × m = A · xa + B · ya + C yb = (− sin θ · xa + cos θ · ya−dy) × m = −B · xa + A · ya + D where A = m · cos θ , B = m · sin θ, C = −m · dx, D = −m · dy (1)

The parameters A, B, and C at this time are estimated by the parameter estimating unit 40 by the least square method.

However, in order to estimate the above parameters by the parameter estimating unit 40, at least two pairs of coordinates of corresponding points are required. However, if only one pair of coordinates of corresponding points is obtained, the input image What is necessary is just to set the average value of the corresponding point vector with respect to the reduced image data indicating the reduced image of the 1/2 image size as the translation parameter. That is, the average value of the corresponding point vectors for reduced image data representing a reduced image having an image size that is 倍 the size of the input image is represented by (ax,
ay), the parameter shown in the following equation (2) is set as the translation parameter.

A = 1, B = 0, C = −ax, D = −ay Expression (2)

If no pair of coordinates of the corresponding point is found, the subsequent processing cannot be performed. For example, a message indicating the fact is displayed on the monitor screen to inform the operator and to combine The processing operation ends.

The parameter estimating unit 40 according to the present embodiment estimates and outputs an affine transformation parameter for an image having the same size as the input image and a translation parameter for other reduced images. However, the affine transformation parameter may be estimated and output for both the image showing the same size as the input image and the reduced image.

Then, in the image conversion / synthesis unit 50, synthesized image data c is formed in accordance with the algorithm shown in FIG.

In s51 of FIG. 8, first, the synthesized image data c
The image area of the composite image indicated by is set.

In this embodiment, the image area is set as shown by a broken line in FIG. 9 with reference to the coordinate system of the image indicated by the input image data a. That is, the left end of the image area is the left end coordinate of the image indicated by the image data a,
The right end is the larger one of the coordinates obtained by converting the upper right end and lower right pixels of the image indicated by the image data b into the coordinates of the image indicated by the image data a. In order to convert the coordinates of the image represented by the image data b into the coordinates of the image represented by the image data a, the inverse transformation of the affine transformation of the above equation (1) is used. That is, the parameters of the inverse transformation are A ', B',
If C 'and D' are used, the conversion of the following equation (3) is performed.

Xa = A ′ · xb + B ′ · yb + C ′ ya = −B ′ · xb + A ′ · yb + D ′ where A ′ = A / (A ² + B ² ) and B ′ = − B / (A ² + B ² ) , C ′ = (− A · C + B · D) / (A ² + B ² ), D ′ = (− B · C−A · D) / (A ² + B ² ) Equation (3)

The upper end of the composite image indicated by the composite image data c is the upper end coordinate value of the image indicated by the image data a, and the upper right and upper left pixels of the image indicated by the image data b are the coordinates of the image indicated by the image data a. The lower coordinate is the lower coordinate value of the image represented by the image data a, and the lower right coordinate and the lower left pixel of the image represented by the image data b are those of the image represented by the image data a. The larger coordinate value of the coordinates obtained by converting the coordinates is used.

Next, in s52 of FIG. 8, the position of the joint is set as indicated by a dotted line L in FIG. That is, the smaller coordinate of the right end coordinate value of the image indicated by the image data a and the coordinates obtained by converting the upper left and lower left pixels of the image indicated by the image data b into the coordinates of the image indicated by the image data a The average value is used as the seam position.

In s53 of FIG. 8, each pixel value in the image area indicated by the composite image data c set in s51 is obtained and written in the memory area.

At s54 in FIG. 8, the image area indicated by the image data a in the image area indicated by the composite image data c is the image data a of the memory area corresponding to the image area indicated by the composite image data c. The pixel value indicated by the image data a is directly written into the memory area corresponding to the image area indicated by.

In s55 of FIG. 8, the image area indicated by the coordinate-converted image data b in the image area indicated by the composite image data c is stored in the memory area corresponding to the image area indicated by the composite image data c. The pixel value indicated by the coordinate-transformed image data b is directly written into a memory area corresponding to the image area indicated by the coordinate-converted image data b. At this time, if the pixel value indicated by the image data a has already been written, only the pixel value to the right of the joint position is written.

The composite image indicated by the composite image data c thus obtained is as shown in FIG. FIG.
The hatched portions in 0 indicate input image data a and b.
Is a region where a corresponding pixel is not assigned from any of the images indicated by..., For example, a dummy pixel such as a white pixel may be assigned.

As described above, in the image conversion / combination unit 50 according to the present embodiment, any one of the image data a and b is assigned to the overlapping area of the image. Calculate the average value of
The pixel values of the image data a and b may be variably weighted in the horizontal direction, for example, and then the average value may be obtained and assigned.

Then, the composite image data c formed in the image conversion / combination section 50 is output to a display device, a printer device or the like.

Although the present embodiment has been described with respect to a case where image data corresponding to two images each representing a grayscale image are combined, the present invention is applicable to two images representing a color image. The present invention can also be applied to a case where image data is combined. In this case, for example, when extracting corresponding points of two images, R (red), G
Of the images indicating the (green) and B (blue) components, corresponding points are extracted using only the image indicating the G component, or R, G,
By forming an image showing the average value of the three components of B, and performing corresponding point extraction using the formed image,
The present invention can be easily adapted.

Further, the present embodiment has been described with respect to a case where image data corresponding to two images generated by being picked up by an electronic still camera, a video camera, or the like is synthesized. It goes without saying that the present invention is also applicable to a case where so-called CG (computer graphic) image data generated by a computer or the like is synthesized.

As described above, according to the image synthesizing apparatus of the present embodiment, it is not necessary to perform a troublesome operation of designating the same corresponding point in a plurality of images with a pointer or the like, and a plurality of images can be automatically converted. It can be synthesized with high accuracy.

Further, in the image synthesizing apparatus according to the present embodiment, the template cut-out area for cutting out a small area used for obtaining a corresponding point from one of a plurality of images as a template is limited. A plurality of images can be synthesized with high accuracy without erroneous corresponding points being extracted outside the overlapping region of the images.

In the image synthesizing apparatus according to the present embodiment, a template for extracting a small area used for obtaining a corresponding point from one of a plurality of images as a template by using information output from the parameter estimating unit. Since the search area for searching the corresponding point corresponding to the cut-out area from other images is limited, a plurality of images can be combined in a short processing time, and a similar part in the plurality of images is used. Even if there are many, the corresponding points can be accurately extracted, and a plurality of images can be synthesized with high accuracy.

Further, in the image synthesizing apparatus according to the present embodiment, a small area used for obtaining a corresponding point from one of a plurality of images is cut out as a template by using a result of extracting the corresponding point from the reduced image. Since the search area for searching the corresponding point corresponding to the template cut-out area from other images is limited, a plurality of images can be synthesized in a short processing time, and a similar part in the plurality of images can be obtained. Can be accurately extracted, and a plurality of images can be synthesized with high accuracy.

Furthermore, in the image synthesizing apparatus according to the present embodiment, since many corresponding points are extracted near the seam where the image is synthesized, it is possible to reduce the displacement at the seam of the synthesized image.

(Second Embodiment) FIG. 11 shows a configuration of an image synthesizing apparatus according to a second embodiment of the present invention.
In the configuration shown in FIG. 11, those having the same functions as those of the image synthesizing apparatus according to the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted. Omitted.

In the figure, reference numerals 13 and 14 denote feature point extraction units for enhancing the edge portions of the reduced image indicated by the reduced image data with respect to the reduced image data output from the image reduction units 11 and 12. By performing various processes, feature point image data is formed and stored in the image memory in the feature point extraction units 13 and 14. (Hereinafter, these feature point image data are referred to as a2 ', b2', a4 ', b4', a8 ', b8'
It is expressed as )

As described above, in the image synthesizing apparatus according to the second embodiment of the present invention, the image reduction units 11 and 12 do not have an image memory, and the next-stage feature point extraction units 13 and 14 store image data. A memory is provided to store the formed feature point image data. Note that, in order to form feature point image data, in the present embodiment, the reduced image data is
By performing a convolution operation with a 3 × 3 pixel two-dimensional filter shown in FIG. 1, the edge portion of the reduced image indicated by the reduced image data is emphasized.

Reference numeral 21 denotes a corresponding point extracting unit, and the input terminal 10
A corresponding point between the respective images is extracted from the image data a, b input from a, 10b or the image indicated by the feature point image data formed by the feature point extraction units 13, 14.

First, the corresponding point extracting section 21 sets the size of the image indicated by the input image data a and b to 1
Feature point image data a8 ', b indicating a feature point image of / 8 times
The corresponding points are extracted for 8 ', and then 1/4 times, 1
Corresponding points are extracted for feature point image data having an image size of / 2 and 1: 1. When the corresponding points are extracted for each of the corresponding point image data having the image size of 1/4, 1/2, and 1: 1, the same as in the first embodiment,
Prediction in the overlapping portion is performed using the conversion parameters formed based on the corresponding point extraction performed before that.

Hereinafter, the corresponding point extracting algorithm in the corresponding point extracting section 21 will be described with reference to the flowchart shown in FIG.

In FIG. 13, at s31, a template cutout area is first set for the feature point image data a8 '. By the way, feature point image data a8 ', b8'
When the corresponding point is extracted, the overlapping area of the two feature point images is not known in advance, so that a predetermined area is set as the template cutout area as in the first embodiment.

Then, the following processes s33 to s36 are performed on all the pixels in all the template cutout regions cut out in s32 of FIG.

In s33, a template is cut out centering on the feature point.

That is, when corresponding points are extracted from feature point image data a8 'having an image size of 1/8 times, as shown in FIG. 14, the template cutout area set in s31 is vertically moved to T1, T2. Divided into regions
The processing is sequentially performed on the respective areas of T1 and T2 along the arrow in the figure, that is, from the right end of the feature point image data a8 ', and by the processing of s33 to s36 in FIG.
The process is continued until a predetermined number of corresponding points are extracted for each of the regions 1 and T2.

Whether to extract corresponding points is determined by
It is determined whether or not the pixel value of the feature point image indicated by the feature point image data a8 'is equal to or greater than a predetermined threshold. If the pixel value of the feature point image is equal to or greater than the predetermined threshold, it is determined that the image is a feature point image. Corresponding points are extracted according to the processing of s34 to s36 in FIG. 13. If the pixel value of the feature point image is not equal to or greater than a predetermined threshold, it is determined that the image is not a feature point image.
Without performing the corresponding point extraction by the processing of s34 to s36,
The process proceeds to processing of a feature point image having the next image size.

In the present embodiment, the processing is continued until a predetermined number of corresponding points are extracted, and the processing is terminated when the predetermined number is reached, so that an unnecessary processing time is omitted. Since the processing time can be shortened and the processing is sequentially performed from the right end of the feature point image data a8 ', for example, even if the processing is interrupted in the middle, the processing is performed near the joint between the two images. Since many points can be extracted, 2
The displacement of the seam between the two images can be reduced, and the template cut-out region is divided into upper and lower parts in advance, so that the seam between the two images can be connected in a well-balanced manner in the upper and lower parts.

The same processing is performed when extracting corresponding points for feature point image data having 1/4 times and 1/2 times the image size. When the extraction is performed, the seam between the two images is set by the next-stage image conversion / synthesis unit 50 as shown in FIG. 9 in the same manner as in the first embodiment. Extract. The position of the seam between the two images is calculated by the same processing as the processing of s52 in FIG. 8 in the description of the operation of the image conversion / combination unit 50 according to the first embodiment.

As described above, when the position of the joint between the two images is calculated, as shown in FIG. 15, the template cutout region set in s31 is divided into regions T3 and T4 on the left and right of the position of the joint. The region is divided, and processing is sequentially performed on each of the regions along the arrow in the drawing, that is, from near the position of the joint (L 'in FIG. 15) of the feature point image data a.

When extracting corresponding points for feature point image data a2 'and a4' having image sizes of 1/4 times and 1/2 times, the template cutout area is set in the same manner. When corresponding points are extracted from image data a having an image size, the image data a having the same size is not subjected to the processing by the feature point extracting units 13 and 14 in FIG. After performing the convolution operation processing with the indicated 3 × 3 pixel two-dimensional filter, feature points are extracted by threshold processing, and a template is cut out around the extracted feature points.

Next, in s34 of FIG. 13, a search area for searching for a point corresponding to the template cut out in s33 is set from the right image data b. by the way,
As for the feature point images a8 'and b8' having an image size of 1/8, the overlapping area is not known in advance, so that the predetermined area is set to the corresponding point search area as in the first embodiment. In addition, for the feature point image data indicating the feature point image having the image size of ４ times, ， times, and 1 ×, the first coordinate conversion parameter output from the parameter estimating unit 40 is used. Is predicted and set in the same manner as in the first embodiment.

In s35 of FIG. 13, the image in the search area for the corresponding point in the right-side feature point image data b8 'set as described above and the image indicated by the template in the left-side feature point image data a8' are displayed. Comparison is performed while sequentially shifting the template in parallel, and a difference value between pixels constituting each image in the image indicated by the template and the image in the search area for the corresponding point is calculated,
The position of the template at which the sum of the absolute values of the difference values is minimum is detected as a corresponding point.

In s36 in FIG. 13, the reliability of the corresponding point detected in s35 is determined, and data on the coordinates of the extracted corresponding point is stored in the memory inside the corresponding point extracting unit 21.

In the present embodiment, since the corresponding points are extracted from the reduced image using the feature point image, even if there is a slight difference in luminance level between the input images, the difference in the luminance level can be reduced. The corresponding points can be extracted without being affected,
Further, since the corresponding point extraction processing for the input image itself is not performed on the feature point image, the corresponding point can be extracted without emphasizing image noise.

The operation of the other parts of the image synthesizing apparatus according to the second embodiment of the present invention is the same as that of the first embodiment of the present invention, and the description is omitted.

(Third Embodiment) FIG. 16 shows a configuration of an image synthesizing apparatus according to a third embodiment of the present invention.
In the configuration shown in FIG. 16, those having the same functions as those of the image synthesizing apparatus according to the first embodiment shown in FIG. Omitted.

That is, the present embodiment is different from the first embodiment in that the corresponding point extracting unit 22 performs the corresponding point extracting process using the coordinate data of the corresponding points output from the corresponding point vector selecting unit 30.

In the present embodiment, the corresponding point extracting unit 2
Since the corresponding point extraction algorithm in FIG. 2 is the same as that of the first embodiment of the present invention shown in FIG. 3, the corresponding point extraction algorithm in the corresponding point extraction unit 22 will be described below with reference to the flowchart shown in FIG.

In FIG. 3, at s21, a template cutout area is first set for the reduced image data a8. Then, the following processes s23 to s25 are performed for all templates cut out in s22. However, when extracting reduced image corresponding points of 1/4, 1/2, and 1 × image size, 1/8 times, 1 /
Corresponding point extraction is performed using only the template from which the corresponding point can be extracted by the corresponding point extraction performed on the reduced image data indicating the reduced image having the image size of 4 times and 1/2 times.

In s23 of FIG. 3, for each template cut out as described above, a search area for searching for a corresponding point in the image indicated by the reduced image data b8 on the right side is set. By the way, reduced image data a
When the corresponding points are extracted for 8 and b8, since the overlapping area of the two reduced images is not known in advance,
As in the first embodiment, a predetermined area is set as a corresponding point search area. When extracting a corresponding point from reduced image data indicating an image having an image size of 1/4, 1/2 or 1: 1, the coordinates of the corresponding point output from the corresponding point vector selection unit 30 The search area is set so that only the area of a predetermined width (for example, ± 5 pixels) is searched in the vertical and horizontal directions centering on the coordinate position of the corresponding point with reference to the data regarding the corresponding point.

In s24 of FIG. 3, the image in the search area for the corresponding point in the right reduced image data b8 set as described above and the image indicated by an arbitrary template in the left reduced image data a8 are compared with the arbitrary image. Are compared while sequentially shifting the position of the template, a difference value between pixels constituting each image is calculated between the image indicated by the template and the image in the search area of the corresponding point, and the difference value of the difference value is calculated. The position of the template where the sum of the absolute values is minimum is detected as the corresponding point.

In s25 of FIG. 3, the reliability of the corresponding point detected in s24 is determined, and data on the coordinates of the extracted corresponding point is stored in the memory inside the corresponding point extracting unit 21.

The operation of the other parts of the image synthesizing apparatus according to the third embodiment of the present invention is the same as that of the first embodiment of the present invention, and the description is omitted.

As described above, the image synthesizing apparatus according to the present embodiment uses the information output from the corresponding point vector selection unit to obtain the small point used to obtain the corresponding point from one of the plurality of images. Since the search area for searching for a corresponding point corresponding to the template cutout area from another image is limited as a template, a plurality of images can be combined in a shorter processing time, and a plurality of images can be synthesized. Even if there are many similar parts, the corresponding points can be accurately extracted, and a plurality of images can be synthesized with high accuracy.

[0113]

As described above, according to the image synthesizing method of the present invention, for example, even when there is a difference in the level of the luminance component between a plurality of images indicated by a plurality of input image data,
This makes it possible to automatically combine a plurality of images with high accuracy without any troublesome operation and with a short processing time, without much deviation of the seams of the images.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image composition device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of an input image.

FIG. 3 is a flowchart illustrating a processing algorithm of a corresponding point extracting unit according to the first embodiment of this invention.

FIG. 4 is an explanatory diagram of a template extracting method for extracting corresponding points according to the first embodiment of the present invention.

FIG. 5 is an explanatory diagram of a method of setting a search area in corresponding point extraction according to the first embodiment of the present invention.

FIG. 6 is an explanatory diagram of a method for setting a template cutout region in corresponding point extraction according to the first embodiment of the present invention.

FIG. 7 is an explanatory diagram of a method of setting a search area in corresponding point extraction according to the first embodiment of the present invention.

FIG. 8 is a flowchart illustrating a processing algorithm of an image conversion / combination unit according to the present embodiment.

FIG. 9 is an explanatory diagram of an image synthesizing method according to the present embodiment.

FIG. 10 is a diagram illustrating an example of a composite image according to the present embodiment.

FIG. 11 is a block diagram illustrating a configuration of an image synthesizing apparatus according to a second embodiment of the present invention.

FIG. 12 is a diagram illustrating a two-dimensional filter according to a second embodiment of the present invention.

FIG. 13 is a flowchart illustrating a processing algorithm of a corresponding point extraction unit according to the second embodiment of this invention.

FIG. 14 is an explanatory diagram of a template extracting method for extracting corresponding points according to the second embodiment of the present invention.

FIG. 15 is an explanatory diagram of a template extracting method for extracting corresponding points according to the second embodiment of the present invention.

FIG. 16 is a block diagram illustrating a configuration of an image synthesizing apparatus according to a third embodiment of the present invention.

[Explanation of symbols]

 11,12 Image reduction unit 13,14 Feature point extraction unit 20,21,22 Corresponding point extraction unit 30 Corresponding point vector selection unit 40 Parameter estimation unit 50 Image conversion and synthesis unit

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Kenji Hatori 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc.

Claims (8)

[Claims] 1. A method of composing a plurality of images having image areas partially overlapping each other to create one composite image, wherein a plurality of image data each corresponding to the plurality of images are generated. Reduced image data that is input, and the plurality of images indicated by the plurality of input image data are reduced stepwise at different reduction rates, and form a plurality of reduced image data indicating the reduced images formed respectively. Forming the reduced image data formed in the reduced image data forming step, from the reduced image data indicating the reduced image with the highest reduction ratio, the image having the same size as the image indicated by the plurality of input image data A corresponding point extraction step of sequentially extracting corresponding points between the images indicated by the same size image data shown in order, and the corresponding point extracting step A parameter setting step of setting various parameters necessary for performing image synthesis conversion in accordance with the corresponding points extracted in the above, and the various parameters set in the parameter setting step are the reduced image data in the corresponding point extraction step. In the case where the reduced image data is set for the indicated reduced image, the reduced image data having the reduction ratio one step higher is used by using the corresponding points already extracted in the corresponding point extracting step and various parameters set in the parameter setting step. Is predicted, and a small area for obtaining a corresponding point from one of the plurality of reduced images is set as a template. In the corresponding point extracting step, a reduction ratio of 1 is set. Extraction of corresponding points between multiple reduced images indicated by stepped reduced image data Align, wherein when the corresponding point extraction step in which the equal magnification image data set for the image shown is
An image conversion / combining step of converting at least one of a plurality of images indicated by the plurality of input image data according to various parameters set in the parameter setting step, and combining the converted image with another image Synthesis method. 2. The method according to claim 1, wherein the corresponding point extracting step sets, using the various parameters set in the parameter setting step, one of a plurality of reduced images indicated by reduced image data having a reduction ratio higher by one step. 2. The image synthesizing method according to claim 1, wherein a search area for searching for a corresponding point corresponding to the template to be performed in another reduced image is set. 3. The corresponding point extracting step includes: using a corresponding point already extracted, a template set in one reduced image of a plurality of reduced images indicated by reduced image data having a reduction ratio one step higher. The method according to claim 1, wherein a search area for searching for a corresponding corresponding point in another reduced image is set. 4. The method according to claim 1, wherein the corresponding point extracting step extracts a number of corresponding points near a joint between the plurality of images. 5. A method for composing a plurality of images having image areas partially overlapping each other to create one composite image, comprising: a plurality of images each corresponding to a plurality of images. Data reduction, a plurality of images represented by the plurality of input image data are reduced stepwise at different reduction rates, and a plurality of reduced image data representing reduced images formed respectively are formed. An image data forming step; and a feature point reduced image forming step of extracting a feature point of the reduced image from the reduced image data formed in the reduced image data forming step to form a feature point reduced image; For the feature point reduced image data formed in the image forming step, the input plurality of feature point reduced image data indicating the reduced image with the highest reduction rate A corresponding point extracting step of sequentially extracting corresponding points between the feature point images indicated by each of the feature point images, in order up to feature point equal-size image data indicating a feature point image having the same size as the image indicated by the corresponding image data; A parameter setting step of setting various parameters required for performing image synthesis conversion according to the corresponding points extracted in the extraction step; and various parameters set in the parameter setting step are feature point reduction in the corresponding point extraction step. When the image data is set for the feature point reduced image indicated by the image data, the reduction ratio is set to 1 using the corresponding points already extracted in the corresponding point extraction step and the various parameters set in the parameter setting step.
A small area for predicting an overlapping portion of a plurality of reduced feature point images indicated by the stepped reduced feature point image data and obtaining a corresponding point from one reduced feature point image of the plurality of reduced feature point images is used as a template. Setting, and causing the corresponding point extracting step to perform extraction of corresponding points between a plurality of reduced feature point images indicated by the reduced feature point image data whose reduction ratio is one step higher. If it is set for the image shown, according to various parameters set in the parameter setting step, to convert at least one of the plurality of images indicated by the plurality of input image data, An image conversion method for synthesizing the image with another image. 6. The corresponding point extracting step includes using one of various parameters set in the parameter setting step to select one of a plurality of reduced feature point images represented by reduced feature point image data having a higher reduction ratio by one step. 6. The image synthesizing method according to claim 5, wherein a search area for searching a corresponding point corresponding to the template set in the reduced feature point image in another reduced feature point image is set. 7. The corresponding point extracting step includes: using the corresponding points already extracted, one of the plurality of reduced feature point images represented by the reduced feature point image data whose reduction ratio is one step higher. 6. The image synthesizing method according to claim 5, wherein a search area for searching for a corresponding point corresponding to the template set in step (a) in another reduced feature point image is set. 8. The image synthesizing method according to claim 5, wherein the corresponding point extracting step extracts a number of corresponding points near a joint between the plurality of feature point images.