ES2784438T3 - Image combining method and device - Google Patents

Abstract

A method of combining a plurality of globally aligned sub images in a region to be combined, the method comprises: finding a set of corresponding pixels in the plurality of globally aligned sub images; determining, by means of optimization, an interpolation position corresponding to a weighted sum of the positions of the corresponding pixels; determining a pixel value of a pixel at the interpolation position in the region to be combined using pixel values of the corresponding pixels; where the weighting factor for the position of a corresponding pixel in a given sub image depends on the relative distance of said interpolation position to an edge of the region to be combined associated with that sub image.

Description

DESCRIPTION

Image combining method and device

The present invention is related to an image combining method. It is also related to an image combination device and a computer-readable medium that carries instructions for carrying out said method.

Methods are known in the art whereby multiple images covering a different part of a scene can be combined into one panoramic image. The viewing angle of such an image can reach up to 360 degrees, giving the user a complete impression of the surroundings from a given point of view. For example, applications are known where vehicle-mounted cameras are used to obtain information about the surroundings. Such information can be used in, for example, surveying and measurement tasks, remote work planning and (public) asset management. Other applications of, for example, panoramic sequential images or video include entertainment, for example, 360-degree content for virtual reality displays. Typically, a plurality of cameras are used, where the cameras are each directed in a different direction. The images obtained by these cameras must be combined into one panoramic image. This process involves combining images.

When creating a perfect, high-quality panoramic image from multiple sub images, it is usually intended to minimize parallax errors. This improves the geometric correction of the result and reduces the visibility of blending artifacts such as streaks or ghosting. Parallax errors can be minimized by maintaining the optical center of the camera (s) by recording the images in the same spatial position. In practice, however, this is avoided by misalignment, for example when shooting manually without a tripod and panoramic rotator, physical restrictions, for example to align the optical center of multiple cameras, or the requirements of multiple perspectives, for example. For example, some parallax may be necessary to build a depth map or stereo views. To improve panoramic image quality when parallax (residual) is present, algorithms are needed to minimize the visibility of artifacts.

Figure 1 illustrates the general combination problem of combining two 1,2 images into a single image. Hereinafter, the images to be combined are called sub images, where the result of the combination, for example, the panoramic image, is known as the final image.

Sub images 1, 2 have been globally aligned and overlap in an overlapping region 3. After global alignment, the sub images typically no longer rotate or switch to each other when these images are used to build the final image. Global alignment is known in the art and comprises finding a suitable relative position and orientation of the sub images based on the corresponding features in the sub images. Alternatively, the positioning or orientation can be determined using the geometric relationship between the cameras responsible for obtaining the cameras or between the shutters of those cameras or cameras at the time of recording the sub images.

Due to global alignment, a reference frame can be introduced by which the positions in the sub images and the final image can be expressed as vectors relative to a common OR origin.

The final image has a shape that can correspond to the sum of the contours of the sub images 1, 2. To determine the content of this final image, three regions can be identified. In the first region I, the pixels of the final image are determined using pixels from only sub picture 1. In the second region II, the pixels of the final image are determined using pixels from only sub picture 2. In the third region III, the pixels of the final image are determined using pixels from sub image 1 and sub image 2. This last region, hereinafter referred to as the region to be combined, can correspond to the overlapping region 3, but can also be smaller or larger than this region.

In Figure 1, the vectors p ¹ and p ² indicate positions of the corresponding pixels in sub-images 1 and 2, respectively. For example, these vectors point to pixels that represent or are part of the same feature in both sub images. Accordingly, these vectors point to pixels in the respective sub images that are associated with features that appear in both sub images. In the absence of parallax, vector p ¹ and vector p ² should ideally coincide. It is observed that both the vector p ¹ and the vector p ² relate to a common origin denoted by O, and may refer to pixels that are outside the overlapping region and / or region to be combined.

Corresponding pixels in two or more sub-images can be found using techniques known in the art, such as optical flow, motion estimation, and / or (dense) (stereo) matching. In most techniques, not only the pixels themselves are examined, but also the surrounding pixels to determine whether a pair or combination of pixels constitutes a corresponding pair or group of pixels.

The combination problem can be summarized as how to determine the pixel values for the pixels in the region to be combined, for example region III in figure 1. For example, in figure 1, the pixel value of the pixel at a position corresponding to the vector p, must be determined using the information from sub-images 1 and 2. In this case, it is important to determine a) which pixels in sub-images 1,2 should be used to determine the value of the pixel in p, and b) if these pixels are known, how is the pixel value at position pi determined using the pixel values of these pixels.

A known technique to address this problem is known as alpha blending. In this technique, the value of the pixel at position p, in the final image is determined using the pixel values at this same position in sub-images 1 and 2, that is, p, = p ¹ = p ² . A weighting factor can be used to impart more or less weight to a particular pixel value of a pixel in a given sub image. One drawback of this known approach is that ghosting can appear in the final image.

In US5986668A and US6097854A, a local alignment step is described as a ghost image removal method in which pairwise matches are used to calculate a strain to an average (virtual) position. This known method can handle n overlapping input images, warping each image to the same average interpolation position, eliminating ghosting in the overlay region. However, because the overlap region deforms to an average interpolation position and non-overlapping parts are not deformed, this approach can produce visual artifacts at this boundary.

Additional methods for combining images have been described in US7006709B2 and US2014307045A1.

"Joint View Triangulation Relief Mosaics", by Lhuillier et al, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Part 1, December 8, 2001, discloses a combination method according to the preamble of the claim 1. Here, a relief mosaic is represented by a collection of images with an associated Joint View Triangulation (TVC) for pairs of adjacent images. Overlapping sections of two adjacent images are interpolated locally. This interpolation calculates the final position of a vertex in the new mosaic coordinates using a weighting that depends on the distances of the vertices corresponding to the limits of their respective images in the overlapping region.

A drawback of the above interpolation approach is that it requires the complex construction of a triangulation, which may not perfectly follow the per-pixel shift between sub images resulting in artifacts in the final image.

It is an object of the present invention to provide a solution to the aforementioned combination problem in which the occurrence of artifacts is minimized.

This objective is achieved with the method according to the invention, which is characterized in that the interpolation position corresponds to a weighted sum of positions of the corresponding pixels, where the weighting factor for the position of a corresponding pixel in a sub-image depends on a relative distance from said interpolation position to an edge of the region to be combined associated with that sub image.

According to the present invention, the corresponding pixels are used to determine the pixel value of a pixel at the interpolation position in the final image. The invention further stipulates that the distance between each pixel between the corresponding pixels and the interpolation position depends on the relative distance of the interpolation position to an edge of the region to be combined associated with the sub-image corresponding to said pixel.

The relative distance may correspond to a distance from said interpolation position to an edge of the region to be combined associated with that sub image in relation to the sum of these relative distances for all sub images. For example, the relative distance is given by:

gave

Equation 1

L n = i d n

where d, is the distance of at least one of said corresponding pixel and said interpolation position to an edge of the region to be combined associated with sub image i, and where N is the total number of sub images.

The weighting factor for the position of a corresponding pixel in a given sub image can be equal to the relative distance associated with that sub image.

In one embodiment, the weighting factor decreases as the distance from the interpolation position to the edge decreases.

Edge distance can generally be calculated as the shortest edge distance. Furthermore, the edge may comprise an arbitrary curve or a collection of curves, which are connected to form a larger curve or contour.

The corresponding pixels in the plurality of sub images may be pixels that relate to the same region or point of an object or feature that is displayed in the plurality of sub images. For example, the feature may comprise a house that is visible in each of the sub images. However, the position of this house appears to be different after globally aligning the sub images. The pixels can be identified in each of the sub images that relate to the same place in the house. These pixels can be identified using techniques known as motion estimation.

The edge of the region to be combined associated with a sub picture may correspond to an edge of that sub picture in the aligned plurality of sub pictures. Here, the aligned plurality of sub-images may refer to an imaginary composite image of the plurality of sub-images. The edge of the sub image refers to the edge of that sub image that is present in the composite image. The region to be combined can correspond to a complete region in which at least two or more sub images overlap.

Alternatively, the region to be combined could be centered around a curve through a region in said plurality of images in which at least two of said sub images overlap. In this way, the overlapping region cannot be fully used for combining. In such a case, the method may further comprise determining a correction reference point using the differences in pixel values between the respective corresponding pixels for a plurality of pixels in the region to be combined, comparing the correction reference point with a threshold pre-defined, and if the correction reference point exceeds the pre-defined threshold, extending the region to combine.

For example, the region to be combined comprises 100 pixels corresponding to 100 interpolation positions. The pixel values for these pixels are determined using two sub images. For each pixel in the region to be combined, there are two available pixel values of the two sub images by which a pixel value can be determined. When there is a large difference between these pixel values, a large parallax correction can be expected to have occurred. In such a case, it may be advantageous to use a wider region to be combined. To that end, the differences for each of the 100 pixels can be combined into a single correction benchmark that can be compared to a threshold.

The method may additionally or alternatively comprise determining a displacement vector field in the region to be combined that indicates a position difference between corresponding pixels in different sub-images, and calculating the curve using the displacement vector field. Here, the displacement vector field is a vector field comprising vectors between corresponding pixels. The curve could be calculated, for example, using a lower cost optimization technique and / or in which the curve is configured to avoid regions where the displacement vectors are relatively large. Therefore, the region to be combined can be chosen so as to exclude regions in the overall sub-image alignment that are characterized by large position differences between the corresponding pixels. Therefore, you can exclude regions that comprise features indicative of a large parallax artifact. In such a case, it may be advantageous to use only pixel information from one of the sub images to determine the pixels in the final image. It may be possible to vary, preferably gradually, the width of the region to be combined, for example depending on the magnitude of the displacement vectors.

The method comprises determining an interpolation position in the region to be combined and finding, by means of optimization, the corresponding pixels of which the pixel values will be used to determine the pixel value of the pixel at the determined interpolation position.

Finding the corresponding pixels may comprise finding the corresponding pixels such that an estimated interpolation position, which is calculated using a weighted sum of the positions of those corresponding pixels, corresponds substantially to the determined interpolation position, where the weighting factors a used for this calculation are calculated using the determined interpolation position.

Additionally or alternatively, the finding of corresponding pixels may comprise in this case a) guessing a set of positions of the corresponding pixels, b) determining the weighting factors for the corresponding pixels, c) calculating an estimated interpolation position applying the weighted sum of the guessed positions of the corresponding pixels, and d) determining an error between the estimated interpolation position and the determined interpolation position and repeating steps a) -d) if the error exceeds a predefined threshold. Alternatively, the weighting factors may depend solely on the determined interpolation position. In this case, the method may further comprise calculating the weighting factors using the determined interpolation position.

Alternatively, finding the corresponding pixels may comprise a step a) of guessing weighted test vectors to find corresponding pixels such that when an interpolation position is calculated using those pixels, it would at least substantially correspond to the determined interpolation position, said vectors of Weighted test vectors are calculated using the determined interpolation position, and a step b) to determine an error between pixels found using the weighted test vectors and repeating steps a) -b) if the error exceeds a predefined threshold.

An alternative method comprises finding a set of corresponding pixels in the plurality of globally aligned sub-images and determining, by means of optimization, an interpolation position that corresponds to the weighted sum of the positions of the corresponding set of pixels.

The interpolation position can be determined using the steps of a) guessing an interpolation position, b) determining the weighting factors for the corresponding pixels, c) calculating an estimated interpolation position applying the weighted sum of the corresponding pixel positions using the determined weighting factors, and d) determining an error between the estimated interpolation position and the guessed interpolation position, and repeating steps a) -d) if the error exceeds a predefined threshold. Alternatively, the weighting factors may depend solely on the corresponding pixel positions. In this case, the method may further comprise calculating the weighting factors using the determined positions of the corresponding pixels.

The pixel value of the pixel at the interpolation position in the region to be combined can be determined by weighting the pixel values of the corresponding pixels associated with that interpolation position. Said weighting may comprise applying additional weighting factors to the pixel values of the corresponding pixels. These weighting factors can be chosen so that the weighting factors, which are applied to the pixels between the corresponding pixels that are far from the edge, are relatively large. This reflects the notion that pixels that are far from the edge may be more appropriate candidates for blending than pixels that are close to the edge. This allows a gradual transition from one image to another. In a preferred embodiment, these additional weighting factors are identical to the weighting factors.

According to a second aspect, the present invention provides a device for combining a plurality of globally aligned sub-images in a region to be combined, comprising a memory for containing said plurality of sub-images, a unit for determining the combination region for determining the region to be combined, and a processor configured to construct a final image using the plurality of sub images, said construction comprises performing the method as defined above to combine the sub images in the region to be combined.

According to a third aspect, the present invention provides a computer-readable medium comprising instructions which, when executed in a processor, instruct this processor to perform the method as described above.

In the following, the present invention will be described in more detail with reference to the accompanying drawings, in which:

Figure 1 illustrates the problem of combining images;

Figure 2 illustrates the general concept of the present invention using two sub images;

Figure 3 illustrates different regions that will be combined when three sub images overlap;

Figure 4 illustrates a region to be combined based on a curve;

Figure 5 illustrates a region to be combined based on a displacement vector field;

Figure 6 illustrates a first method for performing image combining according to the present invention; Figure 7 illustrates a second method for performing image combining according to the present invention; and Figure 8 illustrates an embodiment of a device according to the present invention.

Figure 2 illustrates the general concept of the present invention in case two sub images need to be combined. The pixel value at an interpolation pi position in the final image should be determined using sub images 1, 2. Vectors Vi ¹ and Vi ² point to the corresponding pixels in sub images 1 and 2, respectively, which pixels are used for determine the pixel value of the pixel at the interpolation pi position in the final image. Thus:

Equation 2

According to the present invention, the distance between each of the corresponding pixels and the interpolation position depends on:

a distance from the interpolation position to a respective edge associated with a respective sub image (distance method 1);

According to the examples that do not belong to the invention, the distance between each of the corresponding pixels and the interpolation position depends on:

a distance from each of the pixels corresponding to a respective edge associated with the respective sub image (distance method 2);

a mixture of these distances (distance method 3).

The distances indicated in methods 1 and 2 above are illustrated on the right hand side of Figure 2. Here, dn indicates a distance from the corresponding pixel with position pn to the edge Bn associated with sub image n. On the other hand, dn indicates a distance from the interpolation position with the position pi to the edge Bn associated with the sub image n.

Applicant has found that using distance method 1 or 3 may provide slightly improved results in terms of artifacts that are not as clearly visible in the final image as when using distance method 2.

Hereafter, the distance that is used during the merge is called dn, regardless of the distance method used. In this case, the general concept of the present invention, when applied to two sub images, can be described by the following equation:

Equation 3 ^ ¿1 _ ^ ¿2

d2 di

In other words, as the distance d ¹ increases, the corresponding vector Vi ^{1 will} decrease in length. Finally, when d ¹ becomes zero, for example, when a corresponding pixel is at edge B ¹ (distance method 2), Vi ² becomes zero. The latter indicates that the interpolation position pi corresponds to the position of the corresponding pixel in sub image 2, that is, p ² .

Figure 2 on the left side illustrates a distance d ¹ from a corresponding pixel at position p ¹ in sub picture 1 to the edge of this sub picture, the edge of which is indicated by the dashed line. Here it can be seen that in Figure 2 the region to be combined comprises the entire region in which the sub-images 1,2 overlap.

Assuming that the vectors Vi ¹ and Vi ² are arranged in the opposite way, that is, d ¹ Vi ¹ = <hVi ² , equation 2 and equation 3 can be combined into:

^{Equation 4} Pi _ • P iw 2 • p 2 ^{W l}

where w ¹ and w ² are weighting factors given by:

__ d 2

^{Equation 6} W2 d ^ ^ 2

where it is noted that the weighting factors may depend on the interpolation position (for example, distance method 1) and / or the corresponding pixel positions (for example, distance method 2). Therefore, according to the present invention, the interpolation position corresponds to a sum of weighted positions of the corresponding pixels that are used to determine the pixel value of the pixel at the interpolation position, where the weighting factor for the position of each corresponding pixel depends on the relative distance from the interpolation position to an edge of the region to be combined associated with that sub image. This concept can easily be generalized for N overlapping images in:

Equation 7

where

Equation 8

If a combination of the corresponding pn pixels and the interpolation position pi is found that satisfies Equation 7, then those corresponding pixels can be used to determine the pixel value of the pixel at that interpolation position.

Figure 3 illustrates the situation where three sub images 1,2,4 show partial overlap. The table below shows which sub images overlap in which region after global alignment.

Sub image 1 is relevant to four different regions. In region I, the weighting factors for the pixels in sub image 1 can be set to 1 since the other sub images do not contain image information for that position. In region III, the weighting factors for the pixels in sub-images 1 and 2 are determined by comparing the distance from those pixels and / or the interpolation position to the edge associated with sub-image 1 and to the edge associated with sub-image 2, respectively. For region IV, three sub images are taken into account. If a corresponding pixel is found in a given subimage that is outside the relevant region to combine, the associated weighting factor can be set to zero. For example, the position of pixel 5 in sub image 4 is associated with a weighting factor equal to zero when this position indicates a pixel that corresponds to another pixel in sub image 1 or 2 when trying to interpolate within region IV .

In addition to using the entire overlap region, limited regions can be used. In such a case, only a limited region can be used in which the image blending is applied. Such a region can be defined, for example, around a center line or curve 6 in the overlapping region, as shown in Figure 4. Here, the weighting factors for sub picture 1 in region A are set to one and in region C at zero. In region B, the weighting factors depend on the distance to the edge associated with sub image 1.

Figure 4 illustrates a straight line 6 as an example of a curve. However, the invention is not limited thereto.

There are alternative methods to define the region to combine. For example, the region to be combined can be formed based on a displacement vector field, as illustrated in Figure 5. Here, a displacement vector indicates the vector from a pixel in a given sub-image to a corresponding pixel in another sub image. A curve 7 can be found through this field based on a lower cost approach in which the sum of the displacements of the pixels crossed by the curve, or crossed by the curve and a certain region is minimized. Once this curve is found, a certain region surrounding the curve can be used to stitch the image. Additionally or alternatively, the construction of the region to be combined can be done adaptively. For example, for a given starting region, the average amount of correction per pixel can be calculated. Said correction can be calculated using the difference between the pixel values of the corresponding pixels in the different sub images. If the average correction per pixel is too high, the region to be combined can be expanded and the image stitching process can be repeated.

To find a combination of corresponding p ¹ ..pn pixels and the interpolation pi position that satisfies Equation 7, two different methods will be discussed. In the second method, the interpolation position is chosen and the corresponding pixels that should be used for that interpolation position are searched, while in the first method, the corresponding pixels are determined first after which an interpolation position is searched. .

An advantage of determining the interpolation position pi first (second method), is that you can choose pi to align with the regular pixel grid of the composite image you want to create, that is, each integer pixel position in the region of combination of the composite image. Their corresponding pixel p ¹ ..pn positions are actually valued when precise sub pixel matching techniques are used to derive the matches. Known sampling / interpolation techniques, eg bilinear, bicubic or Lanczos sampling, can be used to obtain the pixel or vector values in the 1..n images based on these valued actual position vectors. This method resembles reverse deformation.

If one first determines p ¹ ..pn and then derives a valued real pi position (first method), the latter might not be aligned with the pixel grid of the composite image. Forward warping (or "spatter") techniques can be used to obtain alignment with the regular grid of the composite image. for example, compute for each pixel or vector at a valued real pi position its weighted contributions (distance) to the pixels or vectors of the surrounding regular / integer grid. This is usually done in a contribution pass, where a separate weight map is kept, and a normalization pass where the weights are used to normalize the values on the regular grid.

In both methods, known techniques are used to find the corresponding pixels. An important aspect of these techniques is detailed in the next section, specifically the position of origin. After this explanation, the two methods are further detailed.

Find matches

Techniques for finding dense pixel matches such as motion estimation, optical flow estimation and (stereo) generally produce their result in the form of a vector field containing a displacement vector for each pixel or block of pixels. For example, a full search block match motion estimation technique can be used to test (for each pixel or block of pixels) all possible displacement vectors using block correlation and assign the vector with the best value of correlation.

In practice, many techniques rely on more refined (and faster) optimization and combine a match term (e.g. pixel / block / feature correlation, gradient constraint equations) with additional terms / constraints, such as a smoothness term that prevents the vector from deviating too much from the estimated vectors at neighboring positions.

When applying these techniques, it is important to consider the "position" from which the vectors or the vector field or fields originate. For example, it is possible to estimate a valid displacement vector field vf ¹² at the "position" of image 1, which contains for a pixel position p ¹ the displacement vector at a corresponding pixel position p ² in image 2 according to equation 9. Images 1 and 2 are assumed to overlap with the same coordinate system.

^{Equation 9 P 2 = P l} V12 = Pl ⁺ Vfl2 ( P l)

Similarly, it is possible to estimate a vector field originating from the "position" of image b according to equation 10.

Equation 10 P l = P 2 ^ 21 = P 2 V f 2 l (P 2)

With some techniques, it is also possible to estimate a vector field originating from an "intermediate" virtual image position x. For example, in the middle of images 1 and 2:

^{Equation 11 P i =} Px ^{0 .5} * ^vx = ^px + ^{0 .5 *} vfx ( Px)

^{Equation 12 P 2 = P x - 0 .5 *} V ^x = P ^x - ^{0 .5 *} vf ^x ( P ^x )

Or in an arbitrary virtual # position of the image, where # is between zero and one (indicating the position between image a and b):

^{Equation 13} P ⁱ = P ^x + r * V ^x = P ^x + r * vf ^x ( P ^x )

^{Equation 14} P ² = Px ^{(i - r) * V x = P x (i - r) * v f x (P x)}

For a block coincidence motion estimation, the above equations can be used to derive for a displacement vector being tested, the set of positions at which the block correlation should be applied. The test vectors "pivot" around the px position.

In general, vector fields estimated at different source image positions # contain the same vector values, but at offset spatial positions. In practice, they may differ somewhat due to numerical, occlusion, and other (estimation) effects, such as poorly stated conditions.

Alternatively or additionally, the forward warp technique discussed in conjunction with the first method can be used to "project" a vector field to a different origin position. Due to the effects of occlusion, holes and overlaps can occur in this projection, this effect can be addressed with known occlusion management techniques.

The first method; method 1 away

In a variant of the first method, illustrated in Figure 6A, the first distance method is used. Consequently, direct calculation of the interpolation position is not possible due to the fact that the required weighting factors depend on the unknown interpolation position. In this case, the interpolation pi position can be found by optimization using equation 15:

In detail, after determining the corresponding pixels in step S60, the interpolation position can be found by first guessing an interpolation position in step S61. Next, in step S62, the weighting factors are determined. In step S63, an estimated interpolation position can be found by applying equation 7 using the determined weighting factors and corresponding pixels already found. In step S64, this estimated interpolation position is compared with the guessed interpolation position. The resulting error is compared with a threshold in step S65. If the error is above a certain threshold, the process re-guesses the interpolation position.

For three sub images 1-3, equation 15 can be rewritten as:

where pi, V ¹² and V ¹³ are known since these parameters are related to the corresponding pixels, which have already been found in the first method. Depending on the distance method used, the weighting factors w ² and w ^{3 are} already known (distance method 2) and the interpolation position can be calculated directly, or the weighting factors depend on the interpolation position (methods 1 and 3 distance) and the interpolation position is calculated by optimization. A generic optimization algorithm, such as stochastic hill climbing, can be used to find the best solution for pi.

For two sub images 1 and 2, equation 15 reduces to:

Optimization of this equation, when required if distance method 1 or 3 is used, is usually done along the line from p ^{1 to} p ² .

The first method is very suitable, but is not limited to, finding the corresponding pixels by estimating the vector fields in pairs (eg vf ¹² , vf ²¹ in the previous section).

In the first method, the calculated interpolation position can be off the grid. In such a case, a pixel value can be calculated. Distance weighting can be used to determine how the pixels in the final image surrounding the off-grid interpolation position are influenced by this pixel value.

The first method; method 2 away

In a variant of the first method, illustrated in Figure 6B, the corresponding pixels are found using known techniques in step S66. In Fig. 6B, the distance method 2 is used so that the weighting factors can be calculated directly in step S67, after which the interpolation position can be calculated in step S68 using equation 7.

The second method; method 1 away

In the second method, the interpolation position is determined and the corresponding pixels are searched, so that equation 7 is satisfied. The weighted positions of the corresponding pixels must be equal to the interpolation position. Equivalently, by means of equation 2, the vectors vi ¹ ..vin are searched, which satisfy equation 7.

In a variant of the second method, distance method 1 is used as shown in FIG. 7A. In this case, the weighting factors can be calculated directly in step S71 after determining the interpolation position in S70. The corresponding pixels are determined later in step S72.

The second method is well suited, but is not limited to, finding the corresponding pixels by motion estimation with the interpolation position as the origin position (eg, determining vn, vi ² , vi3). In this case, the estimate follows the same principle of symmetry as the "intermediate" estimate explained above in relation to equations 11-14. The "pivot" point is based on Equation 7 and can be generalized for N images. More particularly, equation 7 can be used to express the position pj of the corresponding pixel j at the position pi of interpolation and the positions pn of the other corresponding pixels:

For three images, equation 18 can be rewritten into:

Equation 19 V ¿i - ^{(1 - W} i ^{) p} i ^{+ W} 2 ^V 2 ^{+ w} 3 ^p 3

^{Equation 20} V ^{¿2 - W} i ^V i ^{+ (1 - W} 2 ^{) V} 2 ^{+ W} 3 ^V 3

Equation 21 V ^¿3 - ^W i ^V i ^{+ W} 2 ^V 2 ^{+ (1 - w *) v *}

where vin are test vectors originating at the pi interpolation position. To find suitable test vectors to initiate a search, it may be advantageous to use information from an interpolation at an adjacent interpolation position pi 'that used the position vectors p ¹ ', p ² ', p3'. These vectors point to known corresponding pixels. The weighting for the interpolation pi position can be applied to these vectors to find the vectors vin ':

Equation 22 V i n _ Z m = i, m. n W m • p m (1 W n) • p n

The vectors vij 'are then translated to the interpolation pi position to find suitable test position vectors:

Equation 23 V n - V i V ¿^

Thereafter, variations can be applied to these test position vectors to find the best matching set of corresponding pixels (or blocks) for which the weighted sum of the position vectors corresponds to the interpolation position.

In the aforementioned embodiment, the error used to control optimization was related to the difference in image information of the pixels signaled by the test vectors. The required weight, which is determined by the interpolation position, was accounted for in the test vectors themselves. In other words, the pointed pixels necessarily met the condition that if the weighted sum of the positions of those pixels is to be used to calculate an interpolation position, then that calculated position would correspond to the interpolation position established at the beginning of the optimization.

Alternatively, the weighting can be applied after the corresponding pixels have been found. In such an embodiment, one could first, in a step a) guess a set of corresponding pixel positions. Therefore, at this stage, it has been verified that the pixels are indeed corresponding pixels, but it remains uncertain whether these pixels correspond to the interpolation position established at the beginning of the optimization. Then, in a step b), the weighting factors for the corresponding pixels can be determined. It should be noted that these weighting factors are already known when the weighting factors depend solely on the interpolation position. In a step c), an estimated interpolation position can be calculated by applying the weighted sum of the guessed positions of the corresponding pixels. In step d) an error is calculated between the estimated interpolation position and the determined interpolation position. If the error exceeds a predefined threshold, steps a) -d) are repeated and a new set of corresponding pixels is tested.

The second method; method 2 away

In a variant of the second method, distance method 2 is used as illustrated in Figure 7B. The interpolation position is determined in step S73 and the corresponding pixels must be found. In this case, Equation 7 applies equally. However, in this case, the corresponding pixel positions are guessed in step S74.

As a next step, the weighting factors are determined in step S75. Because the distance method 2 is used, the weighting factors can only be calculated after guessing the corresponding pixels. Next, an estimated interpolation position is calculated based on the guessed positions of the pixels corresponding values and the weighting factors determined in step S76. In step S77, this estimated interpolation position is compared with the interpolation position that was determined in step S73. The resulting error is compared with a threshold in step S78. If the error is above a certain threshold, the process re-guesses the positions of the corresponding pixels.

For all the methods described above, where an optimization step is used, such as guessing vectors or positions, it may be beneficial to use a previous combination of an interpolation position and corresponding pixels, preferably relative to an adjacent interpolation position, such as starting point for the optimization process.

In the above examples, methods have been described for interpolating between more than two images simultaneously. Alternatively, the interpolation process could be cascaded in a two-image interpolation process. For example, suppose that 4 images are to be combined into one image. As a first step, a first intermediate image can be constructed by interpolating between the first and second images. A second intermediate image can then be constructed by interpolating between the first intermediate image and the third image. The final image can be constructed by interpolating between the second intermediate image and the fourth image.

Determining the pixel value at the interpolation position

The process of finding a combination of interpolation position and corresponding pixels as described above has to be repeated for each position in the region to be combined. Once the combination of the interpolation position and the corresponding pixels is known, the pixel values of the final image can be calculated. For this purpose, a weighting can be used using additional weighting factors vn:

Equation ^{2 4}

where In is the pixel value of sub image n at position pn. As an example, the weighting factors wn mentioned above can be used as other weighting factors, that is, vn = wn. Alternatively, the additional weighting factors can be determined using a distance method since it is different from the distance method used to determine the weighting factors wn.

It should be noted that the pixel value can be related to an intensity value and / or a color channel value such as an RGB value. Also, in case there is more than one pixel value for each pixel, the combining method described here can be applied to each pixel value separately. However, it is preferred to determine a suitable combination of interpolation position and corresponding pixels for one pixel value, such as intensity, and to use this same combination to determine the other pixel values.

Figure 8 illustrates one embodiment of a device 100 in accordance with the present invention. It comprises a memory 101 for containing a plurality of sub images and a unit for determining the combining region 102 for determining the region to be combined. The device 100 also comprises a processor 103 configured to construct a final image using the plurality of sub-images, said construction comprises performing the method as defined above to combine the sub-images in the region to be combined. In addition, processor 103 can also be configured to perform global alignment on stored sub images. The result of this global alignment and / or the final image resulting from the combination can also be stored in memory 101.

Although the present invention has been described using embodiments thereof, it should be apparent to one skilled in the art that various modifications to those embodiments are possible without departing from the scope of the invention as defined by the appended claims.

As an example, the present invention relates to the interpolation of images that have been globally aligned. Within the context of the present invention, the global alignment comprises a first alignment of the images. This alignment may comprise a coordinate transformation to transform the coordinates of the raw images into the coordinates of the final image. For example, two images, each having 3000 x 2000 pixels, can be combined into a single image comprising 5500 x 2000 pixels. In this case, the pixels of the first image, corresponding to a left part, cannot be transformed. The pixels of the second image, corresponding to the right part, can be transformed by the translation (2500, 0). Consequently, an overlap region will be generated that extends between (2500, 0) - (3000, 2000).

In the above description, references have been made to corresponding pixel position vectors that are relative to an origin that is common for all images to be combined. It should be apparent to the person skilled in the art that these vectors can also be expressed in the coordinates of the underlying raw images in combination with the transformation related to that image. In the example above, a vector that points to position (500, 0) relative to the origin of the second raw image is identical to a vector that points to a pixel in the second image at position (3000, 0) relative to the common origin of the first and second images after the transformation associated with the global alignment.

Claims (15)

1. A method for combining a plurality of globally aligned sub-images in a region to be combined, the method comprises: finding a set of corresponding pixels in the plurality of globally aligned sub images; determining, by means of optimization, an interpolation position that corresponds to a weighted sum of the positions of the corresponding pixels; determining a pixel value of a pixel at the interpolation position in the region to be combined using pixel values of the corresponding pixels; where the weighting factor for the position of a corresponding pixel in a given sub-image depends on the relative distance of said interpolation position to an edge of the region to be combined associated with that sub-image. A method according to claim 1, wherein said determination comprises: a) guess an interpolation position; b) determining the weighting factors for the corresponding pixels; c) calculating an estimated interpolation position by applying the weighted sum of the corresponding pixel positions using the determined weighting factors; d) determining an error between the estimated interpolation position and the guessed interpolation position and repeating steps a) -d) if the error exceeds a predefined threshold. 3. A method of combining a plurality of globally aligned sub-images in a region to be combined, the method comprises: determining an interpolation position in the region to be combined; find, by means of optimization, the corresponding pixels of which the pixel values should be used to determine a pixel value of one pixel at the given interpolation position, so that the interpolation position corresponds to a weighted sum of the positions of the corresponding pixels; determining the pixel value of the pixel at the interpolation position in the region to be combined using the pixel values of the corresponding pixels found; where the weighting factor for the position of a corresponding pixel in a given sub-image depends on a relative distance from said interpolation position to an edge of the region to be combined associated with that sub-image. The method according to claim 3, wherein said searching for the corresponding pixels comprises finding the corresponding pixels such that an estimated interpolation position, which is calculated using a weighted sum of the positions of those corresponding pixels, corresponds substantially to the determined interpolation position, where the weighting factors to be used for said calculation are calculated using the determined interpolation position. The method according to claim 3 or 4, wherein said searching for corresponding pixels comprises: a) guessing weighted test vectors to find the corresponding pixels so that when an interpolation position will be calculated using those pixels, l minus would correspond substantially to the determined interpolation position, said weighted test vectors being calculated using the determined interpolation position; b) determining an error between pixels found using the weighted test vectors and repeating steps a) -b) if the error exceeds a predefined threshold. The method according to claim 3 or 4, wherein said searching for corresponding pixels comprises: a) guess a set of corresponding pixel positions; b) determining the weighting factors for the corresponding pixels; c) calculating an estimated interpolation position by applying the weighted sum of the guessed positions of the corresponding pixels; d) determining an error between the estimated interpolation position and the determined interpolation position and repeating steps a) -d) if the error exceeds a predefined threshold. The method according to any of the preceding claims, wherein said relative distance corresponds to a distance from said interpolation position to an edge of the region to be combined associated with that sub-image in relation to the sum of those distances for all the sub images, where the relative distance is preferably given by: gave _¿ y _u ^N _{n = l} d _{u, n} where d, is the distance from said interpolation position to an edge of the region to be combined associated with sub image i, and where N is the total number of sub images. The method according to any of the preceding claims, wherein the weighting factor for the position of a corresponding pixel in a given sub image is equal to the relative distance associated with that sub image, and / or where the factor weighting decreases if the distance from said interpolation position to said edge decreases. The method according to any of the preceding claims, wherein said distance is the shortest distance to the edge. The method according to any of the preceding claims, wherein the corresponding pixels in the plurality of sub images are pixels that relate to the same region or point of an object or feature that is displayed in said plurality of sub images, the preferably, preferably, the method further comprises using a motion estimation technique to find corresponding pixels in said plurality of sub images, or a technique to find dense matches, such as motion estimation, optical flow estimation and / or disparity estimation stereo. The method according to any of the preceding claims, wherein the edge of the region to be combined associated with a sub image corresponds to an edge of that sub image in the aligned plurality of sub images, wherein the region to be combined corresponds preferably to a complete region in which at least two or more sub images overlap. The method according to any of claims 1-10, wherein the edge of the region to be combined associated with a sub-image preferably corresponds to an edge of that sub-image in the aligned plurality of sub-images, wherein the region to be combined is a region centered around a curve through a region in said plurality of images in which at least two of said sub images overlap, the method preferably further comprising: determining a correction reference point using the differences in the pixel values between the respective corresponding pixels for a plurality of pixels in the region to be combined; comparing the correction reference point to a predefined threshold; if the correction reference point exceeds the predefined threshold, extending the region to be combined; and / or the method preferably further comprises: determining a displacement vector field in the region to be combined that indicates a position difference between corresponding pixels in different sub images; calculating the curve using the displacement vector field, where the curve is preferably calculated using a lower cost optimization technique, and where the curve is more preferably configured to avoid regions where the displacement vectors are relatively large. The method of any of the predefined claims, further comprising determining a pixel value of the pixel at the interpolation position in the region to be combined by weighting the pixel values of the corresponding pixels associated with that interpolation position using weighting factors. additional weightings, wherein the additional weighting factors are preferably identical to the weighting factors. 14. A device for combining a plurality of globally aligned sub-images in a region to be combined, comprising: a memory for containing said plurality of sub images; a combining region determining unit for determining the region to be combined; Y a processor configured to construct a final image using the plurality of sub images, said construction comprises performing the method as defined in any of the preceding claims to combine the sub images in the region to be combined. 15. A computer-readable storage medium comprising instructions which, when executed in a processor, instruct this processor to perform the method as described in any of claims 1-13.