JP3530652B2 - Image composition method and image composition device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing method and an image synthesizing apparatus, and more particularly, to an image synthesizing method and an image synthesizing method for synthesizing a plurality of images in which a part of the images overlap to synthesize a panoramic image with a wide angle of view. It relates to the device.

[0002]

2. Description of the Related Art Conventionally, in the image synthesizing method and the image synthesizing apparatus as described above, two images are planarized by performing geometric transformation such as affine transformation so that the same points are coincident in the overlapping areas of the images. The ones that are spliced together are known.

This is, for example, for a subject as shown in FIG.
The two input images captured as shown in FIGS. 24 and 25 in the frame of No. 2 are subjected to geometric transformation such as affine transformation in the overlapping area of the images to create a composite image as shown in FIG. Is.

In this case, since the subject of the frame f1 is entirely dark, the exposure amount at the time of image pickup is corrected so that the dark place becomes bright, and an image as shown in FIG. 24 is picked up. Also,
Since the entire subject of the frame f2 is bright, the exposure amount at the time of imaging is corrected so that the bright part becomes dark, and an image as shown in FIG. 25 is captured.

[0005]

However, the above-mentioned conventional example has the following problems.

Since the image pickup conditions of each image differ depending on factors such as the subject, even when the images related to the combination are accurately connected, a seam is generated on the combined image.

This is particularly noticeable when the exposure conditions are different. For example, even when the above-mentioned input images are connected with high precision, due to the difference in the brightness of the input images for the same subject, as shown in FIG. It becomes a composite image, and seams are conspicuous on the composite image.

Further, even when a plurality of images picked up by the image pickup optical system in which the characteristic of the image plane illuminance ratio is significantly different in the vicinity of the optical axis and in the periphery thereof, a seam occurs due to the difference in image state.

Therefore, the present invention has been made in order to solve such a problem, and an image synthesizing method capable of satisfactorily synthesizing a plurality of images having different image states due to differences in image pickup conditions and the like. Another object of the present invention is to provide an image synthesizing device.

[0010]

In order to solve the above-mentioned problems, the present invention has the following means.

[0011] The image synthesizing apparatus according to claim 1, wherein, in the image synthesis apparatus comprising an image synthesizing means for synthesizing a plurality of images, creating a broad single composite image angle having image areas overlapping each other, the image The synthesizing means determines the distance from the center point of each image in the overlapping image area of the plurality of images.
The image composition is performed after the plurality of images are corrected in accordance with the image states of points having the same.

[0012]

According to a second aspect of the present invention, there is provided the image synthesizing apparatus according to the first aspect.
In the image synthesizing apparatus described above, a detection unit that detects gradations of the plurality of images as the image state, and a correction unit that corrects the plurality of images based on the detection result so that gradations of overlapping image regions become equal to each other. And having. The image synthesizing apparatus according to claim 3 is the image synthesizing apparatus according to claim 2.
In the image synthesizing apparatus, the correction means corrects the gradation.
The corrected image is only the increased dynamic range
It is characterized by normalizing to the gradation of and.

An image synthesizing method according to a fourth aspect is an image synthesizing method for synthesizing a plurality of images having mutually overlapping image regions to create one synthesized image having a wide angle of view, wherein the image synthesizing is performed by the plurality of image synthesizing methods. Within the overlapping image area of the image
Then, it is performed after the plurality of images are corrected according to the image states of the points having the same distance from the center point of each image .

[0015]

The image synthesizing method according to claim 5 is the method according to claim 4.
In the image synthesizing method described above, the gradations in the overlapping image areas are detected as the image states, and the plurality of images are corrected based on the detection result so that the gradations of the overlapping image areas become equal to each other. And The image according to claim 6.
The image synthesizing method is the image synthesizing method according to claim 5,
Increase the dynamic range of the gradation-corrected image
The feature is that the gradation is normalized to the original gradation.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First Embodiment) First, a first embodiment of the present invention will be described.

FIG. 1 is a block diagram showing the configuration of an embodiment of the image synthesizing apparatus of the present invention.

The present apparatus synthesizes input images a and b picked up by an electronic still camera, a video camera or the like, and outputs them as a synthesized image c to, for example, a CRT or a printer, and has a control unit (not shown). In addition, the corresponding point extraction unit 20 and the parameter estimation unit 4 are controlled by the control unit.
0, the gradation correction coefficient determination unit 30, the gradation conversion units 31 and 32, and the image conversion synthesis unit 50.

Here, in the input images a and b, a subject as shown in FIG. 2 is captured by an electronic still camera, a video camera or the like in a part of the frames fa and fb, respectively, with some of the images overlapping. Has an overlapping image area (hereinafter also referred to as “overlapping part”), and in the present embodiment, as shown in FIGS. 3 and 4, the images have different exposure conditions at the time of imaging. There is. In the following embodiments, the case where the image data of the input images a and b is grayscale image data of N gradations will be described.

The corresponding point extraction unit 20 extracts overlapping corresponding points in the images between the input images a and b and determines the reliability of the extraction of the corresponding points. The coordinates in and b are stored in the internal memory. Also, when extracting the corresponding points,
Since the overlapping portion is not known in advance, a predetermined area in the input image a is set as the template cutout area.

When the input images a and b are arranged in the order of left and right, the corresponding point extracting unit 20 divides the template cutout area from the right end in the horizontal direction of the input image a by 30%. Input image a from 10% to 90% in the vertical direction
The template cutout area (area T in FIG. 5) is cut out as a template, and a small area Ta having a size of about 10% of the image size in block units is cut out as a template. At this time, for example, when the input images a and b are arranged in the order of top and bottom, 30% from the lower end in the horizontal direction of the input image a and 10% to 90% in the horizontal direction are input images a. The template cut-out area in is.

Further, the corresponding point extraction unit 20 determines that the input image b
Since the overlapping portion is not known in advance, a predetermined area in which to search for a point corresponding to the cut template is set as a search area.

This search area is set, for example, like the area S shown in FIG. 7, and in the present embodiment, from the left end of the input image b in the horizontal direction from the position in the input image a of the template. Up to 50% to the left, the vertical direction is set to an area from the position in the input image a of the template to a position of ± 10%. In this area setting, the overlapping portion of the input images a and b is set. It is based on the condition that it is 50% or less in the horizontal direction and ± 10% or more in the vertical direction. When the assumed overlapping conditions of the input images a and b are different, the setting of the search area for extracting the corresponding points may be changed.

The parameter estimating section 40 estimates a parameter for image conversion from the corresponding point vector,
From the corresponding point position extracted by the corresponding point extracting unit 20, the parameter of coordinate conversion is estimated.

Further, the parameter estimating section 40 is adapted to perform this coordinate transformation by affine transformation, and the input image b is rotated by θ with respect to the input image a, (dx, dy) is translated, and is enlarged m times. Assuming that there is a converted relationship, the point (xa, ya) in the input image a is the input image b of the following formula.
It corresponds to the middle point (xb, yb).

Xb = (cos θ · xa + sin θ · ya-dx) × m = A · xa + B · ya + C, yb = (− sin θ · xa + cos θ · ya-dy) × m = −B · xa + A · ya + D .. = M · cos θ, B = sin θ, c = −m · dx, D
= -M · dy.

The parameter estimating section 40 is adapted to estimate the parameters A, B, C, D at this time by the method of least squares. However, at least two pairs of corresponding point coordinates are required to obtain the parameters, and when the obtained corresponding points are one pair, the average value of the corresponding point vectors obtained for the 1/2 image is set as the translation parameter. That is, when the average value of the corresponding point vectors is (ax, ay), the parameters are set and output as in the equation.

A = 1, B = 0, C = −ax, D = −ay Formula ... Note that the parameter estimation unit 40 does not perform the subsequent process if no corresponding pair is found. Then, for example, a message is output to the CRT and the process is terminated, and the estimated parameter is used when predicting the overlapping portion.

The image conversion / synthesis unit 50 is configured to perform a synthesis process so that the center of the overlapping portion becomes a seam, and the input images a and b whose brightness is corrected by the gradation conversion units 31 and 32 are processed. The image is converted by the conversion parameter, the image area of the combined image c is set, and then combined into one image.

The image conversion / synthesis unit 50 sets the image area with reference to the coordinate system of the input image a, and sets the area as shown by the broken line in FIG. That is, the left end is the left end coordinate of the input image a, and the right end is the larger coordinate value of the coordinates obtained by converting the pixels at the upper right end and the lower right end of the input image b into the coordinates of the input image a. ing.

Further, the image conversion / synthesis unit 50 is adapted to convert the coordinates of the input image b into the coordinates of the input image a by using the inverse transform of the affine transformation of the above equation. That is, the parameters of the inverse transformation are A ′, B ′, C ′, D ′.
Then, the following equations are converted.

[0034]     xa = A ′ · xb + B ′ · yb + C ′,     ya = -B '* xb + A' * yb + D '                                                               ……formula However, A '= A / (A2 + B2), B '=-B / (A2 + B2), C '= (-AC + BD) / (A2 + B2), D '= (-BC + AD) / (A2 + B2) Is.

The upper end of the composite image c has the upper end coordinate value of the input image a and the smaller one of the coordinates obtained by converting the pixels at the upper right end and the upper left end of the input image b into the coordinates of the input image a. , The lower end is the coordinate value of the lower end of the input image a and the larger one of the coordinates obtained by converting the pixels at the lower right end and the lower left end of the input image b into the coordinates of the input image a.

Here, the image of the composite image c of the input images a and b obtained by the image conversion / synthesis unit 50 as described above is as shown in FIG. It should be noted that the hatched portion in FIG. 9 is a region in which pixels are not assigned from either of the input images a and b, and dummy pixels (for example, white pixels).
Is supposed to enter.

The gradation converting units 31 and 32 are for correcting the images of the input images a and b according to the image states in the overlapping image areas, and the gradation correction coefficient determining unit 30 as a detecting means described later. It has a function as a correction unit that corrects the plurality of images so that the gradations of the overlapping image areas become equal based on the detection result of 1. In this embodiment, gradation conversion is performed on the input images a and b, respectively. Is performed to correct the brightness in the overlapping image areas of the two images to be equal.

When the gradation average values of the pixel values determined to be in the overlapping part are Pa and Pb, the gradation conversion units 31 and 32 calculate the pixel values pa 'and pb' based on the following equations. It is designed to perform the required correction.

Pa ′ = pa × N / (N + Pa−Pb), pb ′ = (pb + Pa−Pb) × N / (N + Pa−Pb) ... where p a and p b are pixel values of the input images a and b, respectively. , N is the number of gradations.

It should be noted that, since the subjects in the overlapped portions are the same, Pa and Pb should be equal if the input images a and b are imaged under the same exposure condition, but FIGS. If the brightness of the subject is plotted on the horizontal axis and the gradation of the image data at the time of image pickup is taken on the vertical axis, the characteristics of the input images a and b at the time of image pickup are as shown in FIG. (However, in FIG. 10, the characteristic for the input image a is represented by 10a, and the characteristic for the input image b is represented by 10b).

Therefore, in the present embodiment, the gradation conversion unit 31
And 32, this characteristic is approximated as shown in FIG. 11, and the gradation correction is performed according to the equation so that the gradations of the subject in the overlapping portions between the respective input images become equal (however, 11, the characteristic for the input image a is represented by 11a, and the characteristic for the input image b is represented by 11b). That is, since the gradation of the input image b is increased by the amount of the arrow in FIG. 11 and the dynamic range of pixel values is increased by the amount of the arrow, the original gradation is normalized.

The tone correction coefficient determination unit 30 determines whether the coordinate values in each input image are in the overlapping portion or not by the parameter estimation unit 4.
This is determined using an output parameter of 0. That is, each coordinate value of the input image a is affine-transformed according to an equation to be converted into the coordinate value of the input image b, and it is determined whether or not the coordinate value is within the area of the input image b.

Further, the gradation correction coefficient determining unit 30 has a function as a detecting means for detecting the gradations of the input images a and b, and the pixel values determined to be in the overlapping part are input image a and b. For each pixel, and after the processing is completed for all the pixels in the overlapping part, the gradation average value P of the respective pixel values is added.
a and Pb are calculated respectively. However, in the case of an image in which the characteristics of the image plane illuminance ratio are significantly different in the vicinity of the optical axis and in the periphery, the average value is not calculated in the entire area of the overlapping portion. 12
The average value is calculated in the vicinity of the center of the overlapping portion shown in the area V.

Further, the gradation correction coefficient determination unit 30 determines the correction coefficient for performing the gradation correction in the gradation conversion units 31 and 32 from the image data of the overlapping portions of the input images a and b. is there. However, this correction coefficient is the coefficient (Pa-Pb) and N / (N + Pa-P) in the above equation.
It is represented by b).

In the present embodiment, as described above, the image conversion / synthesis unit 50 performs the synthesis process so that the center of the overlapping portion becomes a seam. In a general lens, the image plane illuminance ratio near and around the axis depends on the distance from the optical axis to the image point as shown in FIG. 13, and the image plane illuminance for the same subject near the seam, which is the center of the overlapping portion. It is also convenient in that the ratio becomes equal.

That is, in FIG. 14, at the point Q, the distances from the centers O and O'of the input images a and b are hQ and
If hQ ′, hQ <hQ ′, so that the brightness of the subject does not match, but at the point R near the center of the overlapping portion, the distances from the centers O and O ′ of the input images a and b are hR,
If hR ′, then hR = hR ′, so that the brightness of the subject matches.

Therefore, as shown in FIG. 13, when synthesizing an input image picked up by an image pickup optical system such as a wide-angle lens in which the characteristic of the image plane illuminance ratio is significantly different between the vicinity of the optical axis and the periphery, Rather than calculating the average value of each input image in the entire overlapping area and performing gradation correction, calculate the average value in the area where the brightness of the subject matches near the center of the overlapping area The tonality correction can be performed more accurately and the seam caused by the exposure condition becomes less noticeable.

Next, the operation of the image synthesizing apparatus according to the first embodiment of the present invention when the synthetic image c is generated from the input images a and b will be described.

First, the operation when the corresponding point extraction unit 20 extracts the corresponding points will be described.

FIG. 15 is a flow chart showing the processing algorithm of the corresponding point extraction unit 20.

In the figure, first, the corresponding point extraction unit 20
Sets the template cutout area from the input image a (step S21).

Then, the processes in the following steps S22 to S24 are performed on all the cut out templates.

First, a search area for searching for a point corresponding to the cut template is set from the input image b (step S22). At this time, since the overlapping area is not known in advance, a predetermined area is set as the search area, and, for example, the hatched area S shown in FIG. 7 is set as the search area.

Next, the template is shifted in parallel within this search area, and the difference between the input images a and b is calculated. Then, the position where the sum of the absolute values of the differences is the minimum is set as the corresponding point position (step S23).

Then, the reliability of the result of step S23 is determined (step S24).

This reliability determination is performed using the sum of absolute values of the differences that have become the minimum value and the sum of absolute values of the second smallest differences. In the case of the present embodiment, the minimum sum of absolute differences is less than or equal to a first predetermined threshold, and the difference between the second smallest sum of absolute differences and the minimum is greater than or equal to a second predetermined threshold. If so, the corresponding point is determined to be reliable, and the coordinates of the corresponding point in the input images a and b are stored in the memory in the corresponding point extracting unit 20.

In the above description, the position where the sum of the absolute values of the differences is the minimum is the corresponding point position. However, for example, correlation calculation may be performed and the position where the correlation value is the maximum may be the corresponding point position. .

Although the corresponding points of the input images a and b were extracted from the image data by the above processing, for example, two images are displayed on the display and the same points in the images are designated by a cursor or the like and extracted. You may.

Next, the operation of the gradation correction coefficient determining section 30 and the gradation converting sections 31 and 32 will be described.

FIG. 16 is a flow chart showing the processing algorithm of the gradation correction coefficient determination unit 30 and the gradation conversion units 31 and 32.

In the figure, first, the gradation correction coefficient determination unit 30 determines whether or not the coordinate values in each input image are in the overlapping portion by using the output parameter of the parameter estimation unit 40 (step S1). That is, each coordinate value of the input image a is affine-transformed according to an equation to be converted into the coordinate value of the input image b, and it is determined whether or not the coordinate value is within the area of the input image b.

Next, the pixel values determined to be in the overlapping portion are added to the input images a and b, respectively, and after the processing is completed for all the pixels in the overlapping portion, the gradation average value Pa of the respective pixel values, Calculate Pb respectively (step S
2).

Then, the gradation correction coefficient determination unit 30 further uses the coefficients (Pa-Pb) and N / (N) in the above equation.
When + Pa−Pb) is calculated (step S3), the gradation conversion units 31 and 32 perform gradation correction processing for each input image on the entire screen based on the formula (step S4).

In the above embodiment, the average value of the pixel values of the overlapping portions of the input images a and b is used for the gradation correction, but it is almost the same even if the median value of the pixel values of the overlapping portion is used. The effect of is obtained.

Further, although the determination of the overlapped portion pixel for which the pixel averaging is performed is performed using the output parameter of the parameter estimation unit 40, the determination may be made directly from the corresponding point coordinates output from the corresponding point extraction unit 20. At this time, the pixel values of the corresponding points from which the corresponding points have been extracted may be added separately to obtain the respective average values.

Finally, the operation relating to the generation of the composite image c by the image conversion / composition unit 50 will be described.

FIG. 17 is a flowchart showing the processing algorithm of the image conversion / synthesis unit 50.

In the same figure, first, the image conversion / synthesis unit 50.
Sets the image area of the composite image c (step S5).
1).

Next, the position of the joint is set as shown by the broken line L in FIG. 8 so as to be the center of the overlapping portion (step S).
52). That is, the average value of the right edge coordinate value of the input image a and the smaller coordinate value of the coordinates obtained by converting the pixels at the upper left corner and the lower left edge of the input image b into the coordinates of the input image a is the position of the seam. And

Then, the pixel value is obtained for each area of the composite image c set at step S51 (step S53).

First, the pixel values of the input image a are written as they are in the area of the input image a (step S54).

Then, the coordinate value is converted into the coordinate value of the input image b, and the pixel value of the input image b at that position is written as it is (step S55). At this time, if the pixel values of the input image a have already been written, only those on the right of the position of the seam are written.

The image of the composite image c thus obtained is as shown in FIG. The hatched portion in FIG. 9 is a region in which pixels are not assigned from either the input images a or b, and dummy pixels (for example, white pixels) enter.

In the above, when the image plane illuminance ratio of the image pickup optical system is known in advance, the image plane illuminance ratio of each input image is 1 in the entire image area as a pre-process for performing the image combining process of the present invention. It goes without saying that the correction may be performed so that

Further, in the above, in the image conversion / synthesis unit 50 of the present embodiment, the images at the time of synthesis are the input images a and b.
However, the average value of the input images a and b may be allocated in the overlapping portion. Also, the pixel values of the input images a and b may be weighted in the horizontal direction in a variable manner in the overlapping portion, and the average value may be assigned.

As described above, according to the present embodiment, the gradation correction is performed in the area where the same subject is imaged in the overlapping portion of the input images, so that even if the exposure conditions of the input images are different, Images can be combined so that the seams are inconspicuous.

Since the correction is performed using the image data in the vicinity of the center of the image overlapping portion, even when an image is picked up by an image pickup optical system in which the characteristic of the image plane illuminance ratio is significantly different between the vicinity of the optical axis and the periphery. Images can be combined so that the seams are inconspicuous.

(Second Embodiment) Next, the second embodiment of the present invention will be described.
The embodiment will be described.

The configuration of the image synthesizing apparatus according to the second embodiment of the present invention is almost the same as the configuration of the first embodiment shown in FIG. 1 and only the gradation correction method is different.
A gradation correction method different from that of the first embodiment will be described.

In the first embodiment, the gradation conversion section 3
In 1 and 32, the gradation correction is performed based on the gradation average value of the pixel values determined to be within the overlapping portion based on the expression using Pa and Pb. However, in the second embodiment, the input image a
And b having relatively bright pixel values,
The gradation average value of dark ones is Pa1, Pb1, Pa2 and P
As b2, pixel values pa ', pb' are calculated based on the following equations.
Is calculated and gradation correction is performed.

Pa ′ = pa × K2, pb ′ = (K1 × pb + K0) + K2 ... where K1 = (Pa2-Pa1) / (Pb2-Pb1), K0 = (Pa1 × Pb2-Pa2 × Pb1 / ( Pb2-
Pb1) and K2 = N / (K1 * (N-Pb1) + Pa1).

The derivation of this equation will be described below.

First, since the subjects in the overlapping portion related to the image combination are the same, if the input images a and b are imaged under the same exposure condition, the gradation average values Pa1 and Pb1, Pa2 and P2 are obtained.
b2 should be equal. However, as in the first embodiment, when the image is captured as described with reference to FIGS. 3 and 4, the horizontal axis indicates the brightness of the subject and the vertical axis indicates the gradation of the image data at the time of image capturing. The characteristics when the input images a and b are captured are as shown in FIG.

In the present embodiment, the characteristics of the brightness of the subject and the gradation of the image data at the time of image pickup as shown in FIG. 10 are approximated as shown in FIG. 18, and the overlapping portion between the respective input images is considered. The gradation correction is performed so that the gradations of the subject are the same (however, in FIG. 18, the characteristic for the input image a is represented by 18a and the characteristic for the input image b is represented by 18b). That is, when the brightness of the subject corresponding to the two gradation average values is L1 and L2 in FIG. 18, the gradation of the input image b is increased by the amounts of arrows D1 and D2 in FIG. 18, respectively. Then, the gradation of the input image b is corrected by the following equation.

Pb ′ = K1 × pb + K0 Equation where K1 = (Pa2-Pa1) / (Pb2-Pb1), K0 = (Pa1 × Pb2-Pa2 × Pb1 / (Pb2-
Pb1).

Further, when the dynamic range of the pixel value in this expression is increased to the original gradation, the above expression is derived.

The gradation correction coefficient determining unit 30 performs gradation correction in the gradation converting units 31 and 32 from the image data of the overlapping portion of the input images a and b, as in the first embodiment. The correction coefficient for the execution is determined.

However, the correction coefficient in the second embodiment is represented by the coefficients K0, K1 and K2 in the above equation.

Next, the operation according to the second embodiment of the present invention will be described with reference to the gradation correction coefficient determination unit 30 and the gradation conversion units 31 and 3.
The operation in 2 will be mainly described.

FIG. 19 is a flow chart showing the processing algorithm of the gradation correction coefficient determination unit 30 and the gradation conversion units 31 and 32.

In the figure, first, the gradation correction coefficient determination unit 30 determines whether or not the coordinate value in each input image is in the overlapping portion by using the output parameter of the parameter estimation unit 40 (step S11). That is, each coordinate value of the input image a is affine-transformed according to an equation to be converted into the coordinate value of the input image b, and it is determined whether or not the coordinate value is within the area of the input image b.

Next, the pixel values determined to be within the overlapping portion are added to the input images a and b, respectively. In the present embodiment, the gradation average values Pa1 of relatively bright and dark pixel values, Pb1, Pa2, and Pb2 are calculated for each of the input images a and b (step S1
2).

At this time, the pixel value of the input image a having a pixel value larger than a predetermined threshold value and the pixel value of the corresponding input image b are added respectively, and similarly, the pixel value smaller than the predetermined threshold value and the corresponding input image The pixel values of b are added together. And after the processing is completed for all the pixels in the overlapping part,
The gradation average values Pa1, Pb1, Pa2, Pb2 corresponding to the two brightnesses of the respective pixel values are calculated.

Then, the gradation correction coefficient determination unit 30 calculates the coefficients K0, K1 and K2 in advance (step S1).
3), the gradation conversion units 31 and 32 perform gradation correction processing for each input image on the entire screen based on the formula (step S14).

(Third Embodiment) Next, the third embodiment of the present invention will be described.
The embodiment will be described.

FIG. 20 shows the configuration of the image composition apparatus according to the third embodiment of the present invention. The configuration of the present embodiment is almost the same as the configuration of the first embodiment shown in FIG. 1, but in the present embodiment, it corresponds to the case where the gradation correction coefficient determination unit 30 obtains the gradation correction coefficient. The corresponding point coordinates output from the point extraction unit 20 are used as they are.

Since only the gradation correction method is different from that of the first embodiment of the present invention, only the gradation correction method will be described below.

In the present embodiment, similar to the second embodiment, the characteristics of the input images a and b shown in FIG. 10 at the time of image pickup are approximated as shown in FIG. The gradation correction is performed so that the gradations of the subject are the same in the overlapping. Therefore, similarly to the equations, the pixel values pa ′ and pb ′ are obtained by the method shown in the following equations, and gradation correction is performed.

[0099]   pa ′ = pa × K2 ′,   pb ′ = (K1 ′ × pb + K0 ′) × K2 ′                                                               ……formula However, K2 '= N / (K1'x (N-Pb') + Pa ') Is.

However, in the present embodiment, the gradation correction coefficient determination unit 30 uses the corresponding point coordinates output from the corresponding point extraction unit 20 as they are, and the coefficients K0 ', K1', K of the equation.
2'is required.

The method will be described below. First,
Coefficients K0 'and K1' in the following equations corresponding to the equations
Ask for.

Pb ′ = K1 ′ × pb + K0 ′ Equation (3) Here, the corresponding points of the input images a and b have the same gradation when imaged under the same exposure condition, and therefore the corresponding point extraction unit 2
The absolute value of the square of the difference between the pixel value calculated by substituting the pixel value of the input image b into pb on the right side of the equation among the pixel values of the corresponding point coordinates that are 0 output is the pixel value of the input image a. From the pixel values of all the corresponding points extracted to be the minimum, the coefficient K
0'and K1 'are obtained.

Then, the average values of the pixel values of all the corresponding points extracted at the same time are obtained as Pa 'and Pb'.

The gradation correction coefficient determination unit 30 performs the above processing, and the gradation conversion units 31 and 32 correct the gradation of each input image according to the formula.

(Fourth Embodiment) Next, the fourth embodiment of the present invention will be described.
The embodiment will be described.

The configuration of the image synthesizing apparatus according to the fourth embodiment of the present invention is the same as that of the first embodiment of the present invention shown in FIG. 1, and only the gradation correction method is different. ,
The gradation correction method will be described.

In this embodiment, the gradations that the input images a and b can take are N gradations, but the minimum and maximum values of the image data in the entire image area are Namin, Namax, and
A gradation correction method for Nbmin and Nbmax will be described.

That is, let us consider a case where the characteristics of the input images a and b shown in FIG. 10 at the time of imaging are as shown in FIG. 21, and further approximate this characteristic as shown in FIG. The gradation correction is performed so that the gradations of the subject are the same (however, in FIGS. 21 and 22, the characteristics for the input image a are represented by 21a and 22a, and the characteristics for the input image b are represented by 21b and 22b. Exist).

The processing algorithm of the gradation correction coefficient determining unit 30 and the gradation converting units 31 and 32 at this time is the same as that shown in FIG. 16, but before the processing of step S1 is performed, first, the input image a And the minimum for the entire image area of b,
Find the maximum value. Then, the same processing as that of the first embodiment of the present invention shown in FIG. 16 is performed. However, since the actual dynamic lens of the pixel value is expanded to N gradations, the minimum and maximum values Namin, N of the image data are normalized.
amax, Nbmin, and Nbmax are used. That is, the pixel values pa 'and pb' are obtained by the following equation and gradation correction is performed.

Pa ′ = (pa−Nmin) × N / (Nmax−Nmin + 1), pb ′ = (pb + Pa−Pb−Nmin) × N / (Nmax−Nmin + 1), where Nmin = MIN (Namin, Nbmin + Pa) -P
b) Nmax = MAX (Namax, Nbmax + Pa-P
b), where MIN () represents the minimum value and MAX () represents the maximum value.

As described above, according to the present embodiment, even when the dynamic range of the image data of the input images a and b is smaller than the gradation that can be taken as the image data, the overlapping portion of the input images a and b does not occur. The gradation average is used and the minimum and maximum values of the image data in the entire image area of the input images a and b are used to perform gradation correction and combine, so that a joint image with no noticeable seam and high contrast is obtained. To be

The present invention is not limited to the above embodiment, and various modifications can be made.

For example, in the description of each of the above embodiments, a grayscale image was described as an object, but it goes without saying that a color image can also be applied. At that time, the input image a
When the white balance is automatically performed at the time of capturing the images b and b, the same phenomenon as the mismatch of the exposure conditions described in the problems of the related art occurs in the color balance. For each of the above, the image data may be corrected from the image data in the overlapping portion of the images or the image data of the pixels from which the corresponding points are extracted in the same manner as in the case of the grayscale image.

[0114]

According to the present invention, which has been described in detail above, the following effects can be obtained by the above-mentioned configuration.

According to the first aspect of the present invention, the image synthesis is performed in the center of each image within the overlapping image area of a plurality of images.
Since it is performed after correcting the image according to the image condition of the points with the same distance from the point , when combining a plurality of images having different image conditions due to the difference in the imaging conditions etc. An image synthesizing device capable of synthesizing can be provided. Also, for example, the characteristic of the image plane illuminance ratio is
There is a significant difference between the vicinity of the axis and the periphery,
Even when synthesizing several images, the seams are not conspicuous.
Image synthesizing device capable of performing good image synthesizing
Can be provided.

[0116]

According to the second aspect of the present invention, the gradations of a plurality of images are detected as the image state, and the plurality of images are corrected so that the gradations of the overlapping image areas are equal based on the detection result. For example, it is possible to provide an image synthesizing device capable of performing image synthesizing so that seams are not conspicuous even in the case of images captured under different exposure conditions. According to the invention of claim 3, the correction means is a floor.
The tonal-corrected image is
However, since it is normalized to the original gradation, the image correction is accurate.
Can be performed with multiple exposures caused by different exposure conditions.
Image composition device that can make image seams inconspicuous
Can be provided.

According to the fourth aspect of the present invention, the image synthesis is performed by using the center of each image within the overlapping image area of a plurality of images.
Since it is performed after correcting the image according to the image state of the points with the same distance from the point , it is possible to combine well so that the seams are not conspicuous even when combining multiple images with different image states due to different imaging conditions etc. It is possible to provide an image synthesizing method capable of Also, for example, the characteristic of the image plane illuminance ratio is with the optical axis
Multiple images taken with the imaging optics that have a marked difference between near and surrounding
Even when combining images, the seams are not noticeable
Image synthesizing method that enables good image synthesizing
Can be provided.

[0119]

According to the fifth aspect of the present invention, the gradations of a plurality of images are detected as the image state, and the plurality of images are corrected so that the gradations of the overlapping image areas become equal based on the detection result. For example, it is possible to provide an image synthesizing method capable of performing image synthesizing so that seams are not conspicuous even in the case of images captured under different exposure conditions. According to the invention described in claim 6, the gradation is corrected.
The corrected image is increased by the dynamic range
Since it is normalized to the original gradation, the image can be corrected accurately.
Multiple images resulting from different exposure conditions
Image composition method that can make the seams of
Can be provided.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 2 is a diagram showing an imaging situation of images to be combined.

FIG. 3 is a diagram showing an example of an input image.

FIG. 4 is a diagram showing an example of an input image.

FIG. 5 is a diagram showing a method of cutting out a template when extracting corresponding points.

FIG. 6 is a diagram showing a template for extracting corresponding points.

FIG. 7 is a diagram showing a method of setting a search area in extraction of corresponding points according to the present invention.

FIG. 8 is a diagram illustrating a region of a composite image of the present invention.

FIG. 9 is a diagram showing an example of a composite image of the present invention.

FIG. 10 is a diagram showing the characteristics of the brightness of the subject and the gradation of the image data during imaging.

FIG. 11 is a diagram illustrating a gradation correction method according to the first embodiment.

FIG. 12 is an explanatory diagram of a case where a gradation average is obtained in the vicinity of the center of an overlapping portion.

FIG. 13 is a diagram showing characteristics of an image plane illuminance ratio of an imaging optical system.

FIG. 14 is a diagram illustrating a difference in brightness depending on a characteristic of an image plane illuminance ratio of an input image.

FIG. 15 is a flowchart showing a processing algorithm of a corresponding point extraction unit.

FIG. 16 is a flowchart showing a processing algorithm for gradation correction.

FIG. 17 is a flowchart showing a processing algorithm of an image conversion / synthesis unit.

FIG. 18 is a diagram illustrating a method of gradation correction according to the second embodiment.

FIG. 19 is a flowchart showing a gradation correction processing algorithm according to the second embodiment.

FIG. 20 is a block diagram showing a configuration of an image composition device according to a third embodiment.

FIG. 21 is a diagram showing the characteristics of the brightness of a subject and the gradation of image data when an image is captured with a small number of gradations.

FIG. 22 is a diagram illustrating a method of gradation correction according to the fourth embodiment.

[Fig. 23] Fig. 23 is a diagram illustrating a conventional imaging situation of images to be combined.

FIG. 24 is a diagram showing an example of a conventional input image.

FIG. 25 is a diagram showing an example of a conventional input image.

FIG. 26 is a diagram showing an example of a conventional composite image.

[Explanation of symbols]

20 Corresponding point extractor 30 gradation correction coefficient determination unit 31, 32 gradation converter 40 Parameter estimation unit 50 Image conversion and synthesis section

─────────────────────────────────────────────────── ─── Continuation of front page (72) Kenji Hatori Kenji Hatori 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (56) Reference JP-A-7-85246 (JP, A) JP-A-3 -182976 (JP, A) JP 5-342344 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 1/40 G06T 3/00

Claims (6)

(57) [Claims] 1. An image in which a plurality of images having mutually overlapping image areas are combined to create one combined image with a wide angle of view.
In an image synthesizing device including image synthesizing means , the image synthesizing means , in the overlapping image regions of the plurality of images, forms the plurality of images in accordance with image states of points having an equal distance from the center point of each image. An image synthesizing device characterized by performing image synthesizing after correction. 2. A detection means for detecting gradations of the plurality of images as the image state, and a correction means for correcting the plurality of images based on the detection result so that the gradations of the overlapping image areas become equal to each other. The image synthesizing apparatus according to claim 1, further comprising: 3. The gradation is corrected by the correction means.
The original gradation of the image as much as the dynamic range is increased
The image synthesis according to claim 2, wherein the image synthesis is performed as follows.
apparatus. 4. An image synthesizing method for synthesizing a plurality of images having mutually overlapping image regions to create a single synthesized image having a wide angle of view, wherein the image synthesizing is performed within an overlapping image region of the plurality of images.
Then, the image synthesizing method is performed after correcting the plurality of images according to the image states of the points having the same distance from the center point of each image . 5. The gradation of the overlapping image areas is detected as the image state, and the plurality of images are corrected based on the detection result so that the gradations of the overlapping image areas become equal to each other. The image synthesizing method according to claim 4 . 6. The correction is an image in which the gradation is corrected.
To the original gradation as much as the dynamic range increases.
The image synthesizing method according to claim 5, wherein the image is normalized.
Law.