JP2008003683A - Image generation device and its method and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To synthesize accurately an image in synthesizing a single sheet of a panoramic image from a plurality of images even when local nonlinear distortion or geometric transformation between the images occur. <P>SOLUTION: The device is equipped with a pixel extraction part for extracting two or more pixels from an area including a common imaging area with a second image and its peripheral M pixels (here, 0≤M) in a first image, a pixel selection part for selecting two or more pixels constituting the area defined in advance based on the extracted pixel and the area satisfying a designated condition from the common imaging area with the first image and the area including its peripheral N pixels (here, 0≤N) in the second image, an image correction part for correcting at least one of the first image and the second image, and an image synthesizing part for synthesizing the image in which at least one of the first image and the second image is corrected, and generates the image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for generating one image from a plurality of images.

  Conventionally, in order to generate a panoramic image by joining a plurality of images, there has been a method in which aberration correction and rotation correction are performed and the image is corrected (see Patent Document 1). FIG. 22 is a block diagram of the prior art described in Patent Document 1. In FIG.

First, a corresponding point detection unit 2203 detects a point corresponding to a common subject portion between images (hereinafter referred to as a corresponding point). For detection of corresponding points, the focal length value and distortion value are estimated by the distortion aberration focal length estimation means 2204 based on the corresponding points, and the rotation correction angle is estimated by the rotation correction control means 2206. Finally, using these values, first, the distortion correction means 2205 corrects the distortion of the image, and then the rotation correction means 2207 performs rotation correction.
Japanese Patent Laid-Open No. 6-4660

  In the conventional configuration described above, the accuracy of correction decreases unless the corresponding points are accurately extracted. Here, the extraction of corresponding points is usually performed by matching processing. However, even if the pixels are at the same position among a plurality of images due to the influence of the light source or the like, the brightness value and the color are different, so that it is not easy to accurately extract the corresponding points in units of pixels.

  In addition, since it is difficult to completely correct the non-linear distortion generated in each region, the images are not completely connected at the time of synthesis.

  Therefore, the present invention estimates the difference between images by an easy process of matching only without using affine transformation or geometric transformation of perspective transformation, and corrects the panoramic image accurately by using the distortion function obtained by this difference. An object of the present invention is to provide an image processing apparatus that generates the image.

  The image generating apparatus according to the first aspect of the present invention is a pixel that extracts two or more pixels from a region including a common imaging region with the second image and its surrounding M pixels (where 0 ≦ M) in the first image. In the second image and the extraction unit, a region defined in advance based on the extraction pixel and a predetermined region are selected from a region including a common imaging region with the first image and its surrounding N pixels (where 0 ≦ N). A pixel selection unit that selects two or more pixels constituting a region that satisfies the condition, an image correction unit that corrects at least one of the first image and the second image based on the selected pixel, An image synthesis unit that synthesizes the first image and an image in which at least one of the second images is corrected, and generates an image.

  According to the second invention, the predetermined condition is to minimize a difference in pixel values in a region defined in advance based on the extracted pixels.

  According to the third aspect, the difference in the region defined in advance based on the extracted pixel is calculated based on at least one of the luminance information, color information, and geometric information of the image.

  According to the fourth invention, the region is at least one of a straight line, a curve, and a plane.

  According to the fifth aspect, the pixel selection unit calculates a distortion function that represents a correspondence relationship with the pixel extracted by the pixel extraction unit.

  According to the sixth invention, the correction is performed by using at least one of the selected two or more points to calculate the coordinates of a new point of the corresponding point outside the common imaging region. In addition, the image data outside the common imaging region is corrected using two or more points and the new point.

  According to the seventh aspect, the image correction unit performs pixel correction by weighting the distortion function in a region other than the common imaging region of the first image and the second image.

  According to the eighth aspect of the invention, the image composition unit performs a clipping process to compose at least one of the common imaging regions of the first image and the second image.

  According to the ninth aspect, the image composition unit synthesizes at least one of the common imaging region of the first image and the second image by transmission processing.

  Furthermore, according to the tenth aspect, it is possible to provide a common area extraction unit that extracts a common imaging area of the first image and the second image by image matching processing.

  According to the present invention, it is possible to generate a panoramic image by performing correction with high accuracy by an easy process of only matching without using affine transformation or geometric transformation of perspective transformation.

  Further, local matching processing is performed in a predetermined region, and correction processing is performed on the image based on the result. By this local matching processing, it is possible to cope with various geometric transformations including nonlinear distortion of the lens.

  Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment)
FIG. 1 is a block diagram of an image generation apparatus according to an embodiment of the present invention.

  In FIG. 1, the image generation apparatus includes a pixel extraction unit 101, a pixel selection unit 102, an image correction unit 103, and an image composition unit 104.

  FIG. 3 is a flowchart of the image generation apparatus according to Embodiment 1 of the present invention.

  First, the first image and the second image are input, and the pixel extraction unit 101 in the first image has a common imaging region with the second image shown in FIG. Two or more pixels of the first image are extracted from a predefined region including ≧ 0) (step 301).

  Next, the pixel selection unit 102, in the second image, from a region including a common imaging region (hatched portion in FIG. 4) with the first image and its peripheral N pixels (where N ≧ 0). Two or more pixels of the second image constituting a region defined in advance and a region satisfying a predetermined condition are selected based on the extracted pixels (step 302). The predetermined condition will be described later.

  Then, the image correction unit 103 corrects at least one of the first image and the second image (step 303). The correction is performed based on a distortion function obtained from the selected pixel described later.

  Finally, the image synthesis unit 104 synthesizes the first image and the second image, at least one of which has been corrected (step 304).

  Details of each step will be described below.

  Here, an example is shown in which the number of peripheral pixels is 0 (M = N = 0), and only the common imaging area that is the hatched portion in FIG. 4 is focused and synthesized as a predefined area.

First, in step 301, pixels P ₁ (x ₁ , y ₁ ) and P ₂ (x ₂ , y ₂ ) are extracted from the first image (FIG. 5). The hatched area in FIG. 5 is a common imaging area in the first image. As shown in FIG. 5, P1 and P2 are extracted so as to form a line segment having a length in the horizontal direction (y1 = y2).

In step 302, the predetermined condition described above, that is, a line that minimizes the norm with the luminance value of the line segment connecting the pixels P ₁ (x ₁ , y ₁ ) and P ₂ (x ₂ , y ₂ ). Pixels Q ₁ (u ₁ , v ₁ ) and Q ₂ (u ₂ , v ₂ ) constituting the minute are selected from the second image (FIG. 6).

In calculating the norm, first, the line segment P ₁ P ₂ and the line segment Q ₁ Q ₂ are equally divided into n, and new points P ′ _n (x _n , y _n ), Q ′ _n (u _n , v _n ) is obtained (FIG. 7). And it calculates by (Formula 1). Note that f (ξ) represents a luminance value at ξ.

_{However, x n, y n, u} n, v n to note that take the real number. Thus, _{(if x n, y n, u n} , v n is not an integer) when the pixel is not present, the nearest neighbor method (Nearest Neighbor Method), bi-linear method (Bi-Linear Method), bicubic method (Bi- It is necessary to perform interpolation using a conventional sampling method such as Cubic Method). As an example, a bilinear interpolation method is shown below. First, four neighborhoods A _{1 to} A ₄ of the coordinate A and luminance values f (A ₁ ) to f (A ₄ ) of the respective coordinates are obtained (FIG. 8). Then, the luminance value f (A) of the coordinate A is obtained from (Equation 2).

The above process is repeated to obtain a distortion function (correspondence between line segment P ₁ P ₂ and line segment Q ₁ Q ₂ ) in the common imaging region. As an example, as shown in FIG. 9, a line segment P ₁ ^(k) P ₂ ^(k) is selected in units of one line ⁽ y ₁ = y ₂ for each line), and the corresponding line segment Q ₁ ^{(k )} Q ₂ ^(k) may be selected. However, when the selection of the line segment ^{_{Q 1 (k) Q 2 (}} k), in consideration of the result of the line segment ^{_{Q 1 (k-1) Q}} 2 (k-1), so that the overall optimum. That is, (Formula 3) is used for calculating the norm.

  Next, in step 303, the image is corrected based on the distortion function. Here, a method of correcting only the second image is shown.

(Correction of common imaging area)
First, correction is performed by replacing the pixel data of the common imaging region in the second image with the pixel data of the line segment Q ₁ Q ₂ (FIG. 10).

  By this process, the common imaging area of the first image and the second image becomes substantially equal.

(Correction of areas other than the common imaging area)
Next, an area other than the common imaging area in the second image is corrected. The correction is performed by weighting the distortion function and propagating it to surrounding pixels. Here, a method of performing linear weighting is shown.

The right end point of the common imaging area is T ₁ (k ₁ , l ₁ ), the distortion propagation range (number of pixels) is β, T ₁ (k ₁ , l ₁ ) and T _β (k _β , l _β ) The difference in the vertical direction with respect to is α (FIG. 11). If be weighted linear, it can be realized by replacing the luminance value of the pixels on y = l _beta in luminance value on the straight line T ₁ T _beta. Here, a procedure for replacing the luminance value of T ′ _n (k _n , l _β ) with the luminance value of T _n (k _n , l _β ) will be described.

First, the line segment T ₁ T _β is obtained by (Equation 4).

Then, a _{l n} by equation (5).

Note that l _n is a real number. Thus, (if l _n is not an integer) when the pixel is not present, the nearest neighbor method described above, bilinear, it is necessary to perform interpolation using the sampling method such as bi-cubic method. The luminance value of T _n thus obtained is replaced with the luminance value of T ′ _n . By applying this to the luminance value on y = l _β (k ₁ ≦ x ≦ k _β ), linear weighted distortion propagation is completed.

  Finally, image synthesis is performed (FIG. 12).

  First, the common imaging area of the second image is deleted. Then, the first image and the second image are joined.

  As described above, according to the present invention, it is possible to generate a panoramic image by performing correction with high accuracy by an easy process of matching only without using affine transformation or geometric transformation such as perspective transformation.

  Further, local matching processing is performed in a predetermined region, and correction processing is performed on the image based on the result. By this local matching processing, it is possible to cope with various geometric transformations including nonlinear distortion of the lens.

  Further, since the correction is partial, when a 360 ° panoramic image is generated by joining 1 to N pieces of image data continuously captured, the first image (image having an angle of 0 °) and the last image are also generated. These images (images having an angle of 360 °) can also be smoothly synthesized.

  In the present embodiment, M = N = 0 (that is, the common imaging region) is set as the range designated in advance, but the range is not limited to this. For example, as shown in FIG. 13, it is possible to select a pixel with a range exceeding the common imaging area as a designated range. However, when calculating the norm, it should be noted that the calculation is not performed for the region R where no pixel exists.

In the present embodiment, the norm is calculated based on the luminance signal, but the present invention is not limited to this. You may calculate based on color information. For example, an R signal, a G signal, a B signal, and a color difference signal can be used. Further, geometric information may be used. For example, it is possible to determine the direction of the edge from several neighboring pixels (FIG. 14) and select a set of line segments Q ₁ ^(k) Q ₂ ^(k) so that this direction is maintained.

In the present embodiment, two pixels are extracted, but the present invention is not limited to this. For example, as shown in FIG. 15, three pixels can be extracted and matched as a polygonal line. When there are three or more points, matching may be taken as a region of ΔP ₁ P ₂ P ₃ and ΔQ ₁ Q ₂ Q ₃ as shown in FIG. In this way, increasing the number of extraction points increases the degree of freedom and increases the matching accuracy.

In the present embodiment, the norm is calculated by dividing the line segment into n equal parts, but the present invention is not limited to this. For example, as shown in FIG. 17, Q ′ ₂ (u ₂ , v ₂ ) to Q ′ ₄ (u ₄ ) such that the norms of the line segment P ′ ₁ P ′ ₅ and the line segment Q ′ ₁ Q ′ ₅ are minimized. , V ₄ ), and non-linear division may be performed. By allowing non-linear division, the degree of freedom of the interpolation points Q ′ ₂ (u ₂ , v ₂ ) to Q ′ ₄ (u ₄ , v ₄ ) is increased, and the matching accuracy can be increased.

  In the present embodiment, weighting is performed linearly when propagating the distortion function, but the present invention is not limited to this. For example, non-linear weighting as shown in FIG. 18 may be performed. By using such a curve, the image can be corrected more smoothly.

  In the present embodiment, when compositing images, one common imaging region is deleted, but the present invention is not limited to this. Transmission processing may be performed and superposition may be performed. By blurring the boundary portion, the discontinuity of the joint portion can be relaxed, and the effect of the present application can also be achieved by setting the transmittance to 0 or a value substantially close to 0.

  In the present embodiment, only the second image is operated, but the present invention is not limited to this. As shown in FIG. 19, the first image and the second image may be operated simultaneously to search for an optimum pattern. The amount of change due to the correction can be divided into the first image and the second image, and the amount of change per image can be suppressed.

  In the present embodiment, the common imaging area is designated in advance, but the present invention is not limited to this. As shown in the configuration example of FIG. 2, the common area extraction unit 201 may be added to obtain this area by calculation. For example, the cross correlation between the first image and the second image may be calculated, and the region having the highest correlation value may be set as the common region.

(Modification)
It is also possible to construct each component of the image generation apparatus according to the first embodiment as a program and install it in a computer used as the image generation apparatus or distribute it via a network. In this program, the pixel extraction unit 101, the pixel selection unit 102, the image correction unit 103, and the image synthesis unit 104 are mounted as processes or program modules.

  A configuration example according to the present embodiment is shown in FIG. 2001 is a CPU, 2002 is an image capturing unit such as a camcorder or digital camera, 2003 is an image capturing device connection I / O, 2004 is a display unit such as a television, 2005 is a display unit connection I / O, 2006 is a RAM, 2007 Is an external storage device such as a hard disk, 2008 is an external storage device connection I / O, 2009 is a storage medium such as a CD-ROM or DVD-ROM, 2010 is a drive for reading the storage medium, 2011 is a drive connection I / O, 2012 is an I / F that performs network communication.

  Details will be described below with reference to the flowchart of FIG.

  First, load the program.

  This program is stored in the recording medium 2009 in a format that can be executed on a computer. This is read by the drive 2010. Alternatively, this program may be downloaded through an I / F 2012 such as a network. The program may be stored in the external storage device 2007 in advance, or may be stored in the external storage device 2007 through the connection I / O 2011 and 2008 after being read through the drive and the network ( Step 2101).

  The read program is read into the RAM 2006 (step 2102). In addition, when reading using the drive 2010, it expand | deploys to RAM2006 through connection I / O2011. On the other hand, when data is read using the external storage device 2007, it is expanded in the RAM 2006 through the connection I / O 2008.

  Next, image data is captured.

  The capturing is optically performed by the image capturing unit 2002 (step 2103).

  The captured image is stored in the external storage device through the connection I / O 2003 and 2008 (step 2104).

  The stored image data is expanded to the RAM 2006 through the connection I / O 2008 (step 2105).

  The CPU 2001 performs image generation processing using the program and image data read into the RAM (step 2106). The processing here proceeds according to the above-described embodiment.

  The generated result is displayed on the display unit 2004 through the connection I / O 2005. The generated image data can be transmitted through the I / F 2012.

(Other variations)
Although the present invention has been described based on the above embodiments and modifications, it is needless to say that the present invention is not limited to the above embodiments. The following cases are also included in the present invention.

  (1) A part or all of the constituent elements constituting each of the above devices may be constituted by one system LSI (Large Scale Integration). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components on one chip, and specifically, a computer system including a microprocessor, a ROM, a RAM, and the like. . A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to the computer program.

  (2) A part or all of the constituent elements constituting each of the above devices may be constituted by an IC card that can be attached to and detached from each device or a single module. The IC card or the module is a computer system including a microprocessor, a ROM, a RAM, and the like. The IC card or the module may include the super multifunctional LSI described above. The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may have tamper resistance.

  (3) The above embodiment and the above modifications may be combined.

  The image generation apparatus according to the present invention makes it possible to synthesize a panoramic image even from an image in which various geometric transformations including local nonlinear distortion have occurred. This is useful in the case where parallax occurs because the optical centers do not coincide with each other or when the lens aberration correction is insufficient.

Block diagram of a digital watermark embedding apparatus in an embodiment of the present invention Block diagram of a digital watermark embedding apparatus in an embodiment of the present invention Flowchart of a digital watermark embedding apparatus according to an embodiment of the present invention The figure which shows the common imaging region in embodiment of this invention The figure which shows the extraction method of the pixel in embodiment of this invention (in the case of 2 point extraction) The figure which shows the selection method of the pixel in embodiment of this invention (in the case of 2 point selection) The figure which shows the division | segmentation method of the line segment in embodiment of this invention (in the case of a linear) The figure which shows the interpolation method (Bi-Linear Method) of the pixel in embodiment of this invention The figure which shows the matching in the common imaging area in embodiment of this invention (when a periphery is not included) The figure which shows the correction | amendment in the common imaging area in embodiment of this invention (in the case of a linear) The figure which shows the correction | amendment other than the common imaging area in embodiment of this invention The figure which shows the composition method of the image in embodiment of this invention The figure which shows the matching in the common imaging area | region in embodiment of this invention (when a periphery is included) The figure which shows the direction of the edge in embodiment of this invention The figure which shows the extraction method of the pixel in embodiment of this invention (in the case of 3 point extraction) The figure which shows the selection method of the pixel in embodiment of this invention (in the case of 3 point selection) The figure which shows the division | segmentation method of the line segment in embodiment of this invention (in the case of nonlinear) The figure which shows the correction | amendment in the common imaging area in embodiment of this invention (in the case of nonlinear) The figure which shows the matching in the common imaging area in embodiment of this invention (when making both variable) The block diagram of the recording medium in embodiment of this invention Flowchart of recording medium in the embodiment of the present invention Block diagram of prior art (Patent Document 1)

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Pixel extraction part 102 Pixel selection part 103 Image correction part 104 Image composition part 201 Common area extraction part 202 Pixel extraction part 203 Pixel selection part 204 Image correction part 205 Image composition part 2001 CPU
2002 Image capture unit 2003 I / O
2004 Display unit 2005 I / O
2006 RAM
2007 External storage device 2008 I / O
2009 ROM
2010 drive 2011 I / O
2012 I / F
2201 Image memory 2202 Image memory 2203 Corresponding point detection means 2204 Distortion aberration focal length estimation means 2205 Distortion aberration correction means 2206 Rotation correction control means 2207 Rotation correction means 2208 Image composition means

Claims (14)

An apparatus for generating an image using a first image and a second image,
Pixel extraction for extracting two or more pixels from a region including a common imaging region of the first image and the second image and surrounding M pixels (where 0 ≦ M) in the first image. And
Of the second image, a region defined in advance based on the extracted pixel and a predetermined condition are selected from a region including a common imaging region with the first image and its surrounding N pixels (where 0 ≦ N). A pixel selection unit that selects two or more pixels constituting a region to be filled;
An image correction unit that corrects at least one of the first image and the second image based on the selected pixel;
An image generation apparatus comprising: an image synthesis unit that synthesizes the first image and the second image, at least one of which is corrected. The image generation apparatus according to claim 1, wherein the predetermined condition minimizes a difference in pixel values in a region defined in advance based on the extracted pixels. The image generation apparatus according to claim 2, wherein the difference in the region defined in advance based on the extracted pixel is calculated based on at least one of luminance information, color information, and geometric information of the image. The image generation apparatus according to claim 1, wherein the region is at least one of a straight line, a curve, and a plane. The image generation apparatus according to claim 1, wherein the pixel selection unit calculates a distortion function representing a correspondence relationship with the pixel extracted by the pixel extraction unit. The correction uses at least one point of the selected two or more points to calculate the coordinates of a new point of the corresponding point outside the common imaging region, and the two or more selected points and The image generation apparatus according to claim 1, wherein the image data outside the common imaging region is corrected using the new point. The image generation unit according to claim 1, wherein the image correction unit performs pixel correction by weighting the distortion function in a region other than the common imaging region of the first image and the second image. apparatus. The image generating apparatus according to claim 1, wherein the image combining unit combines at least one of the common imaging areas of the first image and the second image by clipping processing. The image generating apparatus according to claim 1, wherein the image composition unit performs composition by transmitting at least one of the common imaging regions of the first image and the second image. The image generation apparatus according to claim 1, further comprising a common area extraction unit that extracts a common imaging area of the first image and the second image by image matching processing. A method of generating an image by combining a first image and a second image,
Pixel extraction for extracting two or more pixels from a region including a common imaging region of the first image and the second image and surrounding M pixels (where 0 ≦ M) in the first image. Steps,
Of the second image, a region defined in advance based on the extracted pixel and a predetermined condition are selected from a region including a common imaging region with the first image and its surrounding N pixels (where 0 ≦ N). A pixel selection step of selecting two or more pixels constituting a region to be filled;
An image correction step of correcting at least one of the first image and the second image based on the selected pixel;
An image generation method including an image synthesis step of synthesizing the first image and the second image, at least one of which is corrected. The image generation method according to claim 11, further comprising a common area extraction step of extracting a common imaging area of the first image and the second image by image matching processing. A program for causing a computer to generate an image by combining a first image and a second image,
Pixel extraction for extracting two or more pixels from a region including a common imaging region of the first image and the second image and surrounding M pixels (where 0 ≦ M) in the first image. Steps,
Of the second image, a region defined in advance based on the extracted pixel and a predetermined condition are selected from a region including a common imaging region with the first image and its surrounding N pixels (where 0 ≦ N). A pixel selection step of selecting two or more pixels constituting a region to be filled;
An image correction step of correcting at least one of the first image and the second image based on the selected pixel;
A program for executing an image synthesis step of synthesizing the first image and the second image, at least one of which is corrected. A semiconductor device that controls to generate an image by combining an input first image and a second image,
Of the first image, two or more pixels are extracted from a common imaging region of the first image and the second image and a region including surrounding M pixels (however, 0 ≦ M),
Of the second image, a region defined in advance based on the extracted pixel and a predetermined condition are selected from a region including a common imaging region with the first image and its surrounding N pixels (where 0 ≦ N). Select two or more pixels that make up the region to fill,
Correcting at least one of the first image and the second image based on the selected pixel;
A semiconductor device that controls to combine the first image and the second image, at least one of which is corrected.

Images