JP2010512696A - Image joining method and device for joining digital images together - Google Patents

Abstract

An image joining method for joining overlapping input images in an overlapping region includes obtaining a low frequency band sub-image from each input image; and obtaining a high frequency band sub-image from each input image. The image joining method further includes blending the low frequency sub-images across at least a portion of the overlap region to produce a low frequency band output image; and at least part of the overlap region to create a high frequency band output image. Joining the high frequency band sub-images across; combining the low frequency band output image and the high frequency band output image to produce a full band output composite image.

Description

  The present invention relates to a method and apparatus for stitching two or more digital images together.

  Digital images depicting overlapping fields of view can be combined to form a composite image of a larger integrated field of view by a process known as “joining”. Several original images can be joined together by using splicing to create a composite image with a larger field of view than any original image.

  One problem that may arise in some prior art stitched images is the emergence of visible “seams” where the fields of view of the merged image intersect. Known image stitching methods typically require an accurate or near-accurate overlay of the input image to produce a composite image. Most known image stitching methods require the same image exposure and, if so, will not work for high dynamic range images. Some known image stitching methods are computer intensive and poorly memory efficient when executed with computer software.

  In order to obtain a high quality composite image, an effective method of joining digital images is generally required. In particular, there is a need for equipment that can merge a high dynamic range image into a composite image.

  The following embodiments and their features are meant to be exemplary and exemplary and are illustrated and illustrated in a way that does not limit the scope. In various embodiments, one or more of the above problems are alleviated or eliminated, but other embodiments are directed to other improvements.

  One feature of the present invention relates to an image joining method for joining overlapping images together. The image joining method includes obtaining a low-frequency band sub-image from each input image and obtaining a high-frequency band sub-image from each input image. Furthermore, the image merging method creates a low-frequency band composite image by blending low-frequency band sub-images, and joins the high-frequency band sub-images at the joint position to create a high-frequency band output composite image. Combining the band sub-images and creating a full-band output composite image by combining the low-frequency band output composite image and the high-frequency band composite image.

  Another feature of the present invention relates to an image joining device that joins overlapping images together. The image splicing device includes a first image blend processor, an image splicing processor, and an image merger. The image merger is coupled to the image blend processor and the image splicing processor, respectively.

Exemplary embodiments are illustrated in the accompanying drawings. The embodiments and drawings disclosed herein are illustrative and not limiting.
Detailed information is provided in the following description to provide a better understanding of the present invention. However, the present invention may be practiced without these details. In other instances, well-known elements have not been shown or described in detail so as not to unnecessarily obscure the present invention. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

  An apparatus and method according to some embodiments of the invention described herein blends a low spatial frequency component of one input image with a low spatial frequency component of another input image, and further features the input image. The high spatial frequency components of the input images are discontinuously spliced at the position selected to match. Low spatial frequency components can be smoothly blended without producing significant artifacts. High spatial frequency components can be spliced along each scan line in the overlap region at the junction location on each scan line closest to the edge feature of the scan line.

  One feature of the present invention relates to an image joining method for joining images. In the following exemplary description, the example method is described as being applied to two overlapping input images, where the field of view shares a common sub-field of view, but it should be understood that the method according to embodiments of the present invention is: It can also be used to merge any number of input images. The portion of each image that depicts a common sub-field may be referred to as an overlapping region. The overlapping area in each image includes a plurality of points corresponding to the points in the overlapping area of other images. Points in the overlapping area of one image and corresponding points in the overlapping area of the other image may be collectively referred to as corresponding points.

FIGS. 1a to 1c show overlapping input images and output composite images that share a common sub-field of view. FIGS. 2A to 2D are scanning lines of an example of overlapping areas shared by input images. 5 is a flowchart illustrating a method according to an example embodiment of the invention. 4a to 4c are non-rectangular output composite images. The data flow diagram which illustrates the example of a joining of an image. 6a-6c are two sub-image scan lines joined to create an output sub-image scan line. 1 is a block diagram of an image joining device according to an embodiment of the present invention. FIG. 3 is a block diagram of an image joining device according to another embodiment of the present invention.

  FIGS. 1a and 1b show a first input image 10a and a second input image 10b that share a common sub-field depicted as a first input region 11a and a second input region 11b, respectively. FIG. 1c shows a composite image 10c obtained by applying the method as described herein to the first input image 10a and the second input image 10b. The composite image 10c includes a common sub-field that is depicted as an overlapping region 11c. The method according to embodiments of the present invention can be applied to stitch together any number of images as long as any of the images shares an overlapping area with at least one of the other images.

  2a to 2d show overlapping area examples 111a to 111d, respectively. Each overlapping region 111 comprises a plurality of pixels, each pixel comprising a first pixel value derived from the first input image and an associated second pixel value derived from the second input image. Each overlapping region 111 is partitioned between the first boundary line 123 and the second boundary line 124 and can be divided by a plurality of scanning lines. Scan line examples 121a to 121d are indicated by hidden lines in FIGS. 2a to 2d, respectively. The scanning lines of the overlapping region example 111a shown in FIG. 2a are generally defined perpendicular to the first boundary line 123a and the second boundary line 124a that are parallel to each other.

In some situations, the input image can be oriented such that the borders of the overlapping regions are not parallel to each other. In such a situation, the input image may be subjected to at least one of trimming and processing by a rotation function so that the boundary lines of the overlapping regions are parallel to each other, but this is not necessarily required for all embodiments. Absent.

  In the embodiment shown in FIGS. 2b to 2d, the input image is oriented so that the first boundary lines 123b to 123d and the second boundary lines 124b to 124d of the overlapping regions 111b to 111d are not parallel to each other. For example, the scan line of the overlap region 121b of FIG. 2b is defined generally perpendicular to the first boundary line 123b, and the scan line of the overlap region 121c of FIG. 2c is defined generally perpendicular to the second boundary line 124c. The scanning line of the overlapping region 121d in FIG. 2d is defined at an oblique angle with respect to both the first boundary line 123d and the second boundary line 124d.

  In some situations, the input images can be normalized to account for different exposure levels. For example, in some embodiments, the input image can be normalized based on exposure information associated with the input image. Alternatively or in addition, normalization can be achieved by calculating an average exposure and further normalizing the exposure level of the overlapping area.

  FIG. 3 shows an example of an image joining method 20 for joining the first input image 10a and the second input image 10b. The image joining method 20 may include aligning the first input image 10a and the second input image 10b according to image correspondence data that can be obtained using known techniques. In some embodiments, the image joining method 20 includes aligning the input images according to corresponding point pairs between the images. Corresponding point pairs can be selected manually or can be automatically generated by using known techniques such as Lucas and Kanade motion estimation, or optimal scan line feature matching, for example. . In general, a single feature match in the overlap region is sufficient for input image alignment.

  In block S22, the image joining method 20 obtains a first low-frequency band sub-image and a second low-frequency band sub-image from the first input image 10a and the second input image 10b. The first low-frequency band sub-image and the second low-frequency band sub-image are respectively overlap regions formed by a common sub-field shared by both the first low-frequency band sub-image and the second low-frequency band sub-image. At least a part of. In some embodiments, the low frequency band sub-image is obtained by filtering the input image with a low frequency spatial filter. The low frequency band sub-image may have a resolution that is lower than the resolution of the input image. In one particular embodiment, the low frequency band sub-image comprises a resolution equal to 1 / 16th of the input image resolution.

  In block S24, the image joining method 20 obtains a first high-frequency band sub-image and a second high-frequency band sub-image from the first input image 10a and the second input image 10b. Each of the first high-frequency band sub-image and the second high-frequency band sub-image also includes at least a part of an overlapping region defined by a common sub-field shared by both the first input image 10a and the second input image 10b. Including. In some embodiments, the high frequency band sub-image is obtained by filtering the input image with a high frequency spatial filter. In one specific embodiment, the high frequency band sub-image comprises a resolution equal to the resolution of the input image.

  In block S26, the image joining method 20 blends the first low frequency band sub-image and the second low frequency band sub-image to produce a low frequency band output sub-image. The blending of the first low frequency sub-image and the second low frequency sub-image may proceed along one or more scan lines. In one embodiment, blending comprises creating a low frequency band output sub-image by using a sinusoidal fusion function. Other suitable fusion functions can also be used, such as, for example, an exponential fusion function, or a linear fusion function.

The blending of the first low frequency band sub-image and the second low frequency band sub image is performed when the exposure amounts of the first input image 10a and the second input image 10b are different from each other.
and reduce the appearance of artifacts. First input image 10a and second input image 10b
In order to further minimize the appearance of Machband artifacts caused by exposure differences between images, the image stitching method 20 further normalizes multiple sub-images based on some or all of the shared overlap region. With stages.

  In block S28, the image joining method 20 joins the first high-frequency band sub-image and the second high-frequency band sub-image to create a high-frequency band output sub-image. The joining of the first high-frequency band sub-image and the second high-frequency band sub-image can proceed along one or more scanning lines. In the following description of exemplary embodiments, the present invention may be implemented by joining a plurality of scan lines sequentially or simultaneously, but the junction is described for a single scan line.

  The joining employs the first scanning line and the second scanning line from the first high-frequency band sub-image and the second high-frequency band sub-image, respectively, as an input, and generates a combined scanning line as an output. A part of each of the first scanning line and the second scanning line is derived from the overlapping region of the corresponding high frequency band sub-image. The pixel value derived from the first input scan line at the junction is assigned to the pixel on the first side of the output scan line between the first boundary line of the overlapping region and the junction position, and the pixel value derived from the second input scan line. Are assigned to pixels on the second side of the output scan line between the junction location and the second boundary of the overlap region.

  The junction position of the output scan lines can be selected based on the edge characteristics of the first input scan line and the second input scan line. The position of the edge feature in the scan line can be obtained by using a known algorithm. In one embodiment, edge features are identified at a pixel interface where the intensity difference between pixels adjacent to the pixel interface exceeds a threshold. Other embodiments may identify edge features by using gradient Laplace or Sobel methods.

The junction position of the output scan line can include, for example, a pixel interface corresponding to an edge feature common to both the first input scan line and the second input scan line. If the output scan line includes only one pixel interface corresponding to a common edge feature, that pixel interface can be selected as the junction location for that scan line. If the output scan line includes multiple pixel interfaces corresponding to common edge features, the selection of a particular pixel interface as a junction location will have the following factors:
i) Intensity gradient of edge features including pixel interface;
ii) whether edge features corresponding to the pixel interface are present in adjacent scan lines;
iii) the number of scan lines in the sub-image including edge features corresponding to the pixel interface;
iv) may be determined according to one or a combination of random selections between a plurality of suitable pixel interfaces.

  The joint positions of adjacent output scan lines may be advantageously selected close to each other in some embodiments so that artifacts caused by switching the joint positions on each scan line are reduced. In some embodiments, when a multiple pixel interface corresponding to a common edge feature is selected as the current scanline junction location, the pixel interface corresponding to the edge feature closest to the nearest scanline junction location is , Selected as the junction position of the current scan line.

Some embodiments may, for example, scramble scan lines to maximize the possibility that edge features may be continuous from one scan line to the next, and therefore the joint positions of adjacent scan lines are close to each other. It may also include changing the length of the scan line by either singing or stretching. Squashing or stretching can be achieved by using any suitable scaling method. The length of the scan line can be varied, for example, up to 15% in some embodiments. In embodiments where the scan line length is changed, the corresponding scan line length of the low frequency band output sub-image can be changed by the same amount to minimize dislocation artifacts. Also, any deviation in the length of one scan line can be taken into account when changing the length of adjacent scan lines. Changing the length of the scan line may be desired in situations where the input image has visible vertical lines in the overlap region. The length of the scan line can be changed manually or automatically.

  If one input scan line of the high frequency band sub-image does not include any pixel interface corresponding to the edge feature, the spline is joined to obtain an output scan line so that no false edges occur in the high frequency band output sub-image. Instead of blends can be used. As described in blending low frequency sub-images, blending can be performed using a sinusoidal fusion function or other suitable fusion function.

  In block S30, the image joining method 20 creates a full-band output composite image by combining the low-frequency band output sub-image and the high-frequency band output sub-image. In some circumstances, the full-band output composite image may include a square image as shown in FIG. However, the input image can be oriented such that the full-band output composite image is non-square as shown in FIGS. 4a-4c.

  FIG. 4 a shows a non-rectangular output image 40 generated from two input images 41 and 42. As shown in FIG. 4b, the non-rectangular output image 40 can be cropped as shown by the crop area 43 to obtain a square image. However, cropping results in loss of some crop image data 44 that can be inconvenient. Another solution is to define an enlarged square image as shown by the enlarged image area 45 in FIG. Enlarging the image results in a number of blank border image areas 46 that can distract.

In some embodiments of the present invention, the edge image region of the enlarged output image is a random neighborhood sampling method random neighborhood sampling method.
Can be filled. In such an embodiment, for each unfilled pixel, an input pixel may be randomly selected from a square or circular area centered around the unfilled (destination) pixel. Reject sampling can be used until the fill input pixel is located. If each is not successful (rejected), the radius of the search area can be slightly increased so that the final filled input pixels are reliably located. If an unfilled destination pixel is filled, the method proceeds to the next (usually adjacent) unfilled pixel and the process may be repeated from a slight previous search radius. This keeps the sampling radius from increasing unconstrained, so the search radius is averaged and kept at a good size to find valid neighborhood pixels. Eventually it has a border image area, pixels whose color and characteristics match the nearest valid pixel, but the appearance is “snowy” (snowy white).

FIG. 5 shows an exemplary embodiment of the above method. The first input image 210a and the second input image 210b adjacent to each other share a common sub-field of view captured by the first image region 211a and the second image region 211b, respectively. The first input image 210a and the second input image 210b are applied to the first high-pass filter 212a and the second high-pass filter 212b to obtain a first high-frequency band sub-image H-210a and a second high-frequency band sub-image H-210b. Entered. The first input image 210a and the second input image 210b are divided into a first low-pass filter 213a and a second low-pass filter to obtain a first low-frequency band sub-image L-210a and a second low-frequency band sub-image L-210b. Also input to 213b. The first high frequency band sub-image H-210a and the second high frequency band sub-image H-210b are merged by the junction unit 214. The first low frequency band sub-image L-210a and the second low frequency band sub-image L-210b are merged by the blender 215. The high frequency band junction output H-210c and the low frequency band blend output L-210c are merged by an adder 216 by merging to produce a full band output image 210c that includes a common subfield of view depicted by the overlap region 211c. The The merging of the high-frequency band junction output H-210c and the low-frequency band blend output L-210c avoids negative numbers and rounding errors that may occur when the outputs are added. It may include multiplication of the band blend output L-210c.

  Figures 6a, 6b and 6c show an exemplary embodiment of the above junction applied to a scan line. FIGS. 6a and 6b show a first input scan line 330a and a second input scan line 330b, respectively, and a first correspondence graph 340A and a second correspondence graph 340b of pixel values along each scan line. The first input scanning line 330 a extends beyond the first boundary line 331, crosses the overlapping region 332, and reaches the second boundary line 333. The second input scanning line 330 b extends from the first boundary line 331 across the overlap region 332 to the end of the second boundary line 333. The first input scan line 330a and the second input scan line 330b include a first pixel interface 334a and a second pixel interface 334b, respectively, corresponding to common edge features 335a and 335b, respectively. In the exemplary embodiment, the first pixel interface 334a and the second pixel interface 334b are the only pixel interfaces in each scan line that are detected as corresponding to the edge feature, so the first pixel interface 334a and the second pixel interface 334b. This position is selected as the joining position.

  FIG. 6c shows an output composite scan line 330c that results from joining the first input scan line 330a and the second input scan line 330b. The output composite scan line 330c results from joining the first input scan line 330a and the second input scan line 330b around the junction position at the pixel interface 334c. Pixel values derived from the first scan line 330a are used for a part of the output composite scan line 330c on the left side of the pixel interface 334c, and pixel values derived from the second scan line 330b are on the right side of the pixel interface 334c. Used for part of the output composite scanning line 330c. Accordingly, as shown in the pixel value graph 340c along the output composite scan line 330c, the joint output to the left of the edge feature 335c corresponds to the first correspondence graph 340A and the joint to the right of the edge feature 335c. The output corresponds to the second correspondence graph 340b.

  Another feature of the present invention provides an image splicing device that stitches images together to form a composite or “joined” image. The functional elements of the image splicing device may be provided by at least one of software and hardware elements executing on one or more data processors (which may include a microprocessor, an image processor, etc.). FIG. 7 shows an image joining device 50 according to an example embodiment. The image splicing device 50 includes a scan line splicing processor 52 and a first scan line blending processor 54 that are both coupled to the image merger 56. The high frequency band sub-image, the image correspondence data, and the image edge position data are input to the first scanning line joining processor 52. The low frequency band sub-image and the image correspondence data are input to the first scanning line blend processor 54. The image joining device 50 may include a second scanning line blend processor to which a high frequency band sub-image and image correspondence data are input. The image splicer 50 may also include a high pass image filter coupled to the scanline splice processor 52 to obtain a high frequency band sub-image from the input image. Image splicer 50 may further include a low pass image filter coupled to the first scan line blend processor to obtain a low frequency sub-image from the input image. The image stitching device 50 further comprises an image alignment detector that generates image correspondence data to be input to one or a combination of the first scanline blend processor, the second scanline blend processor, or the scanline splice processor 52. obtain. The image splicer 50 may further comprise an edge feature detector that locates a pixel interface corresponding to the edge feature for input to the scanline splice processor 52.

FIG. 8 shows an image stitching device 440 according to another embodiment of the present invention. The image splicing device 440 includes a scanning line joining processor H-441, a low-frequency scanning line blending processor L-442, and a high-frequency scanning line blending processor H-442. The high-frequency image filter H-443 filters the input image 450 to obtain a high-frequency band sub-image H-450 that is input to the scanning line splicing processor H-441 and the high-frequency scanning line blend processor H-442, respectively. The low pass image filter L-443 filters the input image 450 to obtain a low frequency band sub-image L-450 that is input to the low pass scan line blend processor L-442. The edge feature detector 445 locates the pixel interface corresponding to the edge feature in the input image 450 and forwards the pixel interface edge data 455 to the joint position selector 446. The joining position selection device 446 is connected to the scanning line joining processor H-441 and the high-frequency scanning line blend processor H-442. The image alignment detector 444 generates image correspondence data 454 that is input to the joint position selection device 446, the low-frequency scan line blend processor L-442, and the high-frequency scan line blend processor H-442. The high-frequency band joint output image H-451S, the high-frequency band blend output image H-451B, and the low-frequency band blend output image L-451B are merged by an image merger 447 that outputs a full-band synthesized image 451.

  Many changes and modifications can be made in the practice of the invention without departing from its spirit and scope, as will be apparent to those skilled in the art in view of the foregoing disclosure. Accordingly, the scope of the invention should be construed in accordance with the content defined by the following claims. Those skilled in the art will recognize certain modifications, substitutions, additions, and subcombinations thereof. Accordingly, the following appended claims and claims subsequently introduced are intended to be construed as including such modifications, substitutions, additions, and subcombinations as falling within the true spirit and scope thereof.

Claims (27)

An image joining method for joining input images that overlap each other in an overlapping region, wherein the image joining method includes:
Obtaining a low frequency sub-image from each said input image;
Obtaining a high frequency band sub-image from each said input image;
Blending the low frequency sub-images across at least a portion of the overlap region to produce a low frequency output image;
Joining the high frequency band sub-images across at least a portion of the overlap region to produce a high frequency band output image;
An image joining method comprising combining the low-frequency band output image and the high-frequency band output image to produce a full-band output composite image.   The image joining method according to claim 1, wherein each of the input images includes a high dynamic range image. Bonding between the high frequency band sub-images,
Selecting a joint position corresponding to an edge feature common to the input images as a selected joint position;
The image joining method according to claim 1, further comprising joining the high-frequency band sub-images at the selective joining position.   The image joining method according to claim 3, wherein the image joining method includes blending the high-frequency band sub-images whose edge characteristics common to each other cannot be identified. The image joining method includes defining a plurality of scanning lines for each of the high frequency band sub-images,
The image joining method according to claim 3, wherein joining of the high-frequency band sub-images at the selective joining position includes joining the high-frequency band sub-images on a scanning line basis.   6. The image joining method includes selecting a pixel interface corresponding to an edge feature common to both of the input images as a joint position for each scanning line of the high-frequency band sub-image. Image joining method.   The image joining method according to claim 3, wherein the image joining method includes identifying an edge feature in the input image.   The image joining method according to claim 3, wherein the image joining method includes identifying corresponding points in the input image.   The image joining method according to claim 1, wherein the blending of the low frequency band sub-images includes using a sinusoidal fusion function.   The image joining method according to claim 1, wherein the image joining method includes normalization of the input image.   The image joining method according to claim 1, wherein the image joining method includes normalization of the low frequency band sub-image.   The image joining method according to claim 1, wherein the image joining method includes normalization of the high-frequency band sub-image.   The image joining method according to any one of claims 1 to 12, wherein the image joining method includes changing a size of the high-frequency band sub-image.   The image joining method further includes defining a rectangular enlarged image region and filling a blank edge image region by random neighborhood sampling when the full-band output composite image includes a non-rectangular image. The image joining method according to one item. The random neighborhood sampling is for each unfilled pixel.
Defining a search area closest to the unfilled pixels;
The method of claim 14, comprising selecting features of input pixels in the search region to fill the unfilled pixels.   The image joining method according to claim 15, wherein the image joining method includes enlarging the search area when the search area includes only unfilled pixels.   The method of claim 16, wherein the image joining method includes selecting, for one unfilled pixel, a size of the search area that is smaller than a size of the search area of previously filled unfilled pixels. Image connection method. An image joining device that joins input images that overlap each other in an overlapping region, the image joining device,
An image blending processor;
An image splicing processor;
An image joining device comprising: the image blend processor and an image merger coupled to the image joining processor. The image joining device further includes:
A low pass filter coupled to the image blend processor;
The image stitching device of claim 18, comprising a high pass filter coupled to the image stitching processor.   20. The image splicing device of claim 18 or 19, further comprising an image alignment detector coupled to the image blend processor.   21. The image splicing device according to any one of claims 18 to 20, further comprising an image alignment detector coupled to the image splicing processor.   The image joining device according to any one of claims 18 to 21, further comprising a joining position selection device coupled to the image joining processor.   23. The image splicing device of claim 22, further comprising an edge feature detector coupled to the image splicing processor.   24. The image stitching device according to any one of claims 18 to 23, further comprising a second image blend processor coupled to the image merger.   The image joining device according to any one of claims 18 to 24, wherein the image merger includes an image intensifier.   A computer-readable storage medium storing a program for causing a computer to execute the procedure of the image joining method according to claim 1. On the computer,
A first function for blending images;
A second function for joining the images;
A computer-readable storage medium storing a program for realizing a third function for adding images.

Images