KR101743861B1 - Methods of image fusion for image stabilization - Google Patents

Abstract

Systems, methods, and computer readable media for improving image stabilization operations are described. New approaches are disclosed for fusing non-reference images with previously selected reference frames in a set of commonly captured images. The fusing method may utilize a smooth transition by using a weighted average for the ghost / non-ghost pixels to prevent sharp transitions between neighboring and nearly similar pixels. Also, ghost / non-ghost crystals may be made based on a set of neighboring pixels rather than progressing independently for each pixel. An alternative approach is to perform multiple resolution decomposition of all captured images, using temporal fusion, spatial-temporal fusion, or a combination thereof at each layer, and combining the different layers to produce an output image Step < / RTI >

Description

[0001] METHODS OF IMAGE FUSION FOR IMAGE STABILIZATION FOR IMAGE STABILIZATION [0002]

This disclosure relates generally to the field of digital photography. More particularly, this disclosure relates to still image stabilization techniques, but is not limited thereto. As used herein, image stabilization refers to a collection of techniques for reducing blur caused by motion during image capture operations. Such motion can occur due to motion of the camera, the scene object, or both.

It is not easy to take high-quality pictures in low ambient light conditions, or to take dynamic scenes ( e.g. , motion scenes) due to camera motion and / or motion of objects in the scene during image capture. One way to reduce motion blur without amplifying the noise of an image is to capture and fuse multiple short-exposure images of the scene. Such operations are often referred to as " still image stabilization ". Shortening the image exposure time can reduce motion blur defects, but it results in more noise and / or darkening sacrifices in the image.

Typical solutions for image stabilization include (1) selecting a reference image from a set of multiple short-time exposed images, (2) registering all non-reference images for the reference image as a whole, and (3) And compositing the output image by fusing the images to the reference image. In a manner in which an output image represents a scene, such as when a reference image is captured, non-reference images are used to average / merge multiple observations for each reference pixel across all images to reduce noise in the reference image .

As a general solution, output image synthesis that fuses all registered non-reference images to a reference image is to average the images as is. Averaging as it is can reduce the noise in the static area of the image, but it can also cause ghosting artifacts. Ghosting defects occasionally occur when some of the pixels in the reference image are occluded in some of the non-reference images due to moving objects in the scene. If there is motion between the captured images, serious ghosting defects may appear at the final output when averaging the images as they are. An example of such a ghosting defect effect is shown in Fig. Figure 1 shows an output generated by averaging registered images as a whole. As can be seen in FIG. 1, there is a serious ghost defect when the images are averaged as they are.

One way to avoid ghosting defects is to distinguish shielding from noise in the fusion procedure and to exclude all shielded areas from fusion. Can be accomplished by averaging all non-reference pixels having significantly different values compared to corresponding reference pixels of non-reference pixels. One way to determine the amount of acceptable difference is to calculate it based on the expected noise of a particular pixel. As soon as the acceptance threshold is determined, the non-reference pixels that differ by more than this threshold relative to the corresponding reference pixels of the non-reference pixels may be dispatched from the average.

However, the set threshold for the ghost / non-ghost pixel classification may itself cause image defects and, particularly if there is a significant noise, this may be a typical case for image stabilization. This is because the acceptance threshold is a statistical estimate that can have a certain failure rate. Neighboring pixels can be easily shifted between ghosted / non-ghosted (i.e., more noise / cleaner) pixels because they correspond to either side of the threshold value. Thus, currently used fusion methods can be improved.

In one embodiment, a method of fusing a captured reference image with a captured non-reference image is provided. The method includes the steps of acquiring a first image of the initially captured scene, the image having a plurality of pixels, and acquiring a second image of the scene in a second order, wherein the plurality of pixels of the first image are each a second Having corresponding pixels in the image. The method may then include selecting a first pixel in the first image and determining a non-binary weight value for the corresponding pixel of the first pixel in the second image. The first pixel may then be combined with its corresponding pixel in the second image using a non-binary weight value to obtain a first fused pixel. The process may repeat selection, determination, and combination for each of a plurality of different pixels of the first image to obtain a fused image.

In another embodiment, an alternative method is provided for fusing the captured reference image with the captured non-reference image. The method according to this solution comprises the steps of acquiring a first image of the initially captured scene, the first image having a plurality of pixels, and subsequently acquiring a second image of the scene in a second order, And the plurality of pixels each have a corresponding pixel in the second image. The non-binary weight value for the corresponding pixel of the first pixel in the second image may then be determined. The method may then obtain a first fused pixel by combining the pixels of the first pixel and its corresponding second image if the non-binary weighted value is greater than the specified threshold. If the non-binary weight value is less than or equal to the specified threshold, the pixels of the first pixel and its corresponding second image may not be combined. The process can then repeatedly select, determine, and combine for each of the plurality of different pixels of the first image to obtain a fused image.

In yet another embodiment, the captured reference image may be fused with the captured non-reference image in an alternative manner to obtain the fused image. This solution is obtained by first acquiring a first image of a captured scene, the first image having a plurality of pixels, and subsequently acquiring a second image of the scene captured secondarily, the second image comprising a plurality And the pixels of the second image each have a corresponding pixel in the first image. Layer pyramid representation of the first image, wherein the uppermost layer of the multi-layer pyramid comprises a low-resolution representation of the first image and the lowest layer of the first multi-layer pyramid comprises a high-resolution representation of the first image, The layer between the top layer and the bottom layer includes a high spatial frequency representation of the first image, each corresponding to a resolution of the layer. The method may then produce a multi-layer pyramid representation of the second image, wherein the uppermost layer of the second multi-layer pyramid comprises a low-resolution representation of the second image, and the lowest layer of the second multi-layer pyramid comprises a high- And the layer between the highest layer and the lowest layer each has a corresponding layer of a multilayer pyramid representation of the first image. The method then identifies, for each group of pixels in one layer of the first multi-layer pyramid representation of the scene, a corresponding group of pixels of the second multi-layer pyramid representation of the scene, Layer pyramid representation of the scene for each of the layers of the first and second multi-layer pyramid representations of the scene by merging a group of identified pixels of the scene. Finally, by combining the output multilayer pyramid representation of the scene, an output image representing the scene can be generated and stored in memory.

In yet another embodiment, the captured reference image may be fused with a non-reference image captured by another method for obtaining a fused image. This solution is achieved by first obtaining a first image of the captured scene, the first image having a plurality of pixels, and then performing a multi-resolution decomposition of the first image to generate a first multi-layer pyramid representation of the first image . The second image of the scene may then be captured, the second image captured at a different time than the first image, and each of the plurality of pixels of the first image has a corresponding pixel in the second image. Resolution resolution of the second image to produce a second multi-layer pyramid representation of the second image. The method then selects one or more pixels in the first image, determines a non-binary weight value for one or more pixels of the second image corresponding to the one or more pixels of the first image, and if the non-binary weight value is greater than the specified threshold Combining one or more pixels of the first image and one or more pixels of their corresponding second images to obtain a first fused pixel and if the non-binary weighted value is less than or equal to the specified threshold, Layer pyramid representation of the scene for each layer of the first and second multi-layer pyramid representations of the scene by not combining one or more pixels of their corresponding second images. The process may then repeatedly generate a hierarchy of output multi-layer pyramid representations of the scene for each layer of multi-resolution decomposition of the first image. The different layers of the output multi-layer pyramid representation of the scene can be combined to produce an output image.

Figure 1 shows an example of image fusion operations according to the prior art.
Figure 2 illustrates a convergence operation in the form of a flow diagram according to one embodiment.
FIG. 3 illustrates an exemplary temporal convergence operation in accordance with one embodiment.
4 illustrates an exemplary temporal convergence operation using block pixels in accordance with one embodiment.
5 illustrates an exemplary spatial-temporal convergence operation in accordance with one embodiment.
6 illustrates a multi-resolution fusion operation in the form of a flow diagram in accordance with one embodiment.
Figure 7 illustrates a typical captured image sequence in accordance with an alternative embodiment.
8A and 8B illustrate fused output images generated by fusing the captured image sequence of FIG. 7 according to another embodiment.
FIG. 9 illustrates exemplary images generated by a multi-resolution decomposition according to an embodiment.
10 illustrates a multi-resolution fusion operation in flow diagram form in accordance with one embodiment.
11 shows a block diagram of a multifunctional electronic device according to one embodiment.

The present disclosure applies to systems, methods, and computer-readable media for improving image stabilization operations. In one embodiment, a new approach may be used to fuse registered non-reference images with a reference image in a set of commonly captured images. The fusion method can take advantage of the smooth transition between the pixels by using a weighted average for the ghost / non-ghost pixels to prevent abrupt transition between neighboring similar pixels. In an alternative embodiment, the ghost / non-ghost crystal may be made based on a set of neighboring pixels rather than progressing independently for each pixel. An alternative solution is to perform multi-resolution decomposition of all captured images, use a weighted average at each layer and / or examine a set of neighboring pixels to determine which pixels to fuse at each layer, And combining the different layers to produce an image.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the concepts of the present invention. As part of this description, some of the figures of the present disclosure show structures and devices in block diagram form in order to avoid obscuring the present invention. To be clear, not all features of an actual implementation are described. In addition, the expressions used in this disclosure are in principle selected for readability and educational purposes and are not selected to describe or limit the gist of the present invention, May be required. Reference in the present disclosure to "one embodiment" or "an embodiment " means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention , And numerous references to "one embodiment" or "an embodiment" are not necessarily to be construed as indicating the same embodiment.

In the development of any actual implementation (as in any development project), many decisions must be made to achieve the developer's specific goals (e.g., compliance with system-related and business-related constraints), and these goals may vary from implementation to implementation Will be understood. It will also be appreciated that such development efforts may be complex and time consuming, but nevertheless routine for those of ordinary skill in the art of designing and implementing an image stabilization system with the benefit of the present invention.

One new approach to image stabilization involves generating an output image by temporal fusion of registered non-reference images and reference images. Referring to FIG. 2, in one embodiment according to this approach, image sequence R is received (block 205) and image stabilization operation 200 begins. One of the first steps in a typical image stabilization operation is to select one of the images in the sequence as a reference image (block 210). There are many different methods known in the art for selecting a reference image. Filed concurrently herewith, the subject matter of which is entitled " Reference Frame Selection for Still Image Stabilization ", filed concurrently herewith, the disclosure of which is incorporated herein by reference in its entirety. After the reference image is selected, the remaining images in the sequence may be registered as a whole for the reference image (block 215). One approach to full registration of non-reference images for a reference image is discussed in U.S. Patent Application, entitled " Image Registration Methods for Still Image Stabilization ", filed concurrently with the present application and incorporated herein by reference in its entirety .

As soon as non-reference images are registered as a whole, corresponding pixels of all images in the image sequence may have the same spatial coordinates (x, y). Since images are acquired at different moments of time, each pixel can be represented by a third coordinate, which corresponds to the image index (x, y, t), which represents the time. For example, the pixel (x, y, 3) may represent a pixel located at the spatial coordinate (x, y) in the third image.

Temporal fusing includes fusing pixels along the temporal dimension of the pixels. 3, line 305 represents the reference image and lines 310, 315 and 320 represent one registered non-reference image in the image sequence, respectively. Only one spatial coordinate, shown in s, is shown. The horizontal axis represents time coordinates, and the received frames can be arranged accordingly. Pixel 325 represents a pixel of the reference image that needs to be fused with corresponding pixels of non-reference images. As can be seen, in temporal fusion, the current pixel can be fused with pixels having the same spatial coordinates in all images. Thus, in FIG. 3, pixel 325 may be fused with pixels 330, 335, and 340.

Non-reference image pixels corresponding to a reference image pixel can often be shielded by moving objects in the scene. As discussed above, fusing such pixels with reference pixels can cause ghosting defects. To prevent ghosting defects in the final output image, the temporal convergence operation 200 (see FIG. 2) may determine whether the pixels of the non-reference image are ghosted. (Block 220) by comparing each pixel of the non-reference image with a corresponding pixel of the reference image to determine the similarity between the two pixels.

However, instead of making the ghost / non-ghost determination with difficulty based on pixel similarity, the operation may calculate the weighting function for each non-reference pixel (block 225). In one embodiment, the weighting function has a value between zero and one. Weight 1 corresponds to non-ghost pixels and weight 0 corresponds to ghost pixels.

In one implementation, the weights may be computed by comparing each non-reference pixel with a pixel of the reference image corresponding to it. In an alternative embodiment, the weights may be calculated based on the pixel similarity value and the expected noise content at certain exposure parameters. As is known in the art, many cameras have a known anticipated noise content for each pixel at specific exposure parameters. The expected noise content of a pixel can be used to calculate its weighting function. Can be performed by calculating the weight W (x, y) for the pixel (x, y) based on the noise standard deviation S (x, y) and the pixel similarity value D (x, y). The pixel similarity value may be a pixel difference value between the pixel (x, y) and the corresponding reference pixel. Assuming that the images are displayed in the YUV color space, all pixels may have three pixel value differences (Dy, Du, Dv), and three noise standard deviations (Sy, Su, Sv).

The specific weighting function used can vary, which is a matter of design choice. In one embodiment, the weighting function may be a Gaussian function. In another embodiment, the weighting function may be linear. Equation (1) represents an example weight function.

ω _{t =} ω _{Y *} ω _{U *} ω _V

(One)

Where ω _t denotes a weight component assigned to the non-reference pixel (x, y, t), ω _Y denotes a weight component corresponding to the Y channel, ω _U corresponds to a weight component for the U channel, ω _V Represents a weight component for the V channel. In this embodiment, the calculated weighting function? _T represents the probability that the pixel (x, y, t) is a non-ghost pixel. In an embodiment in which the weights are calculated based on the expected noise content and the difference in pixel values, the weighting parameters may be calculated according to the following equations.

(2)

(3)

(4)

)는 설계 선호도에 따라 값이 설정될 수 있는 상수들일 수 있다.here(

) May be constants whose values can be set according to design preferences.

An alternative approach to determining whether a pixel is a ghost is to compare blocks of pixels with each other. For each pixel (x, y), by analyzing the block of pixels centered at (x, y), instead of analyzing the individual pixels alone. An example where image 400 represents a reference image and images 420 and 450 represent two non-reference images are shown in FIG. If a weight needs to be computed to fuse the pixel 405 of the image 400 with the corresponding pixels 425 and 455 then the block 410 centered on the pixel 405 is the block 430, , Where each block is centered on a corresponding pixel in each non-reference image. Then, instead of calculating the difference between the individual pixels, the difference between the corresponding blocks, for example, block 410 and block 430, or between block 410 and block 460, can be calculated. For example, it can be performed by calculating Mean Absolute Difference (MAD) or Mean Square Difference (MSD) between blocks. MAD can be calculated according to the following equation.

(5)

는 고스트/비-고스트로서의 상태가 결정되는, 비-기준 이미지에 위치한 픽셀의 좌표를 나타내고,

는 기준 이미지에 위치한 대응하는 픽셀의 좌표들을 나타낸다. i 및 j는 각각의 픽셀을 둘러싸는 블록에 걸쳐있는 가산 인덱스(summation indice)들이고, '||'는 절대값 연산자를 나타낸다. 픽셀 차이값이 블록에 대하여 식(5)에 따라 계산되면, 계산된 값은 위에서 논의한 바와 같이 가중치 파라미터들(ω_Y, ω_U, ω_V)을 계산하는 데 사용된다. 선택되는 블록의 크기는 달라질 수 있고 이는 설계 선택의 문제이다. 일 실시예에서, 블록은 3×3 일 수 있다. 다른 구현예에서, 블록은 5×5 일 수 있다.here

Represents the coordinates of the pixel located in the non-reference image, the state of which is determined as ghost / non-ghost,

Represents the coordinates of the corresponding pixel located in the reference image. i and j are summation indices that span blocks surrounding each pixel and '||' represents an absolute value operator. When the pixel difference value is calculated according to equation (5) for the block, the calculated value is used to calculate the weighting parameters (? _Y ,? _U ,? _V ) as discussed above. The size of the selected block can vary, which is a matter of design choice. In one embodiment, the block may be 3x3. In other implementations, the block may be 5x5.

Referring again to FIG. 2, once a weight is calculated for each non-reference pixel, the result value may be used to calculate a value for the corresponding pixel of the output image (block 230). The following equation can be used for this calculation.

(6)

는 최종 출력 픽셀 값을 나타내고,

는 이미지 t의 공간 좌표

에서의 픽셀 값을 나타내고,

는 비-기준 픽셀

에 배정된 가중치이다. 기준 이미지는 t = 0 인 시간 좌표를 갖는 것으로 가정될 수 있다.here

Represents the final output pixel value,

Is the spatial coordinate of the image t

Lt; / RTI > pixel value,

Lt; RTI ID = 0.0 > non-

. The reference image can be assumed to have a time coordinate of t = 0.

By using a weighting function instead of the set threshold, the temporal convergence operation provides a smooth transition between ghost and non-ghost pixels, thereby avoiding abrupt transitions and hence image defects. However, considering only pixels with the same (x, y) spatial coordinates in all images limits the process to achieving good noise cancellation only as many as the number of short-time exposed images for each sequence. For example, if only four images are received in the image sequence, the reference pixel may average a maximum of three unshielded pixels in other images. If one or more of the images is a ghost, the number of pixels that can be fused is further reduced. In such a case, it has been found that the more pixels to choose, the better the quality of the final output image. In one embodiment, this can be done by using an operation referred to herein as spatial-temporal fusion.

Spatial-temporal fusion in one embodiment may include an extension of the temporal convergence approach by merely fusing pixels as well as other possible pixels with the same spatial coordinates. Thus, the pixel (x, y) of the reference image may be matched with pixels having different spatial coordinates in the non-reference images. The pixel 525 of the reference image 505 is shown in FIG. 5 to be fused with multiple pixels of each of the non-reference images 510, 515, 520.

Referring to FIG. 6, in an embodiment consistent with this approach, if an image sequence R is received (block 605), image stabilization operation 600 begins. The reference image may then be selected from the image sequence (block 610) and non-reference images may be registered for the selected reference image (block 615). Each pixel (x, y) of the non-reference images may then be compared to the pixels of the corresponding reference image (block 620) to compute a weighting function for each non-reference pixel (block 625) . Once the weight is calculated, it may be compared to a predetermined threshold to determine the likelihood that the pixel is a ghost (block 630).

The predetermined threshold may be selected from any possible values for the weights. In one embodiment, the threshold may be 10% of the maximum value of the weight. Thus, if the range of weight values is between 0 and 1, then the threshold may be 0.1.

In one embodiment, the determination as to whether a pixel represents a ghost may be accomplished in the same manner as in temporal fusion, by combining a small image block centered at the corresponding pixel of the reference image with a similar block centered on the pixel being analyzed .

If it is determined at block 630 that the weight is greater than the threshold, temporal fusion may be used to calculate the value for the pixel of the corresponding output image (block 635). If the weight is less than the threshold, it indicates that the pixel is likely to be a ghost pixel, then the action 600 may perform a spatial search on the image containing the pixel (x, y) to find a better fusion candidate Block 640). In one embodiment, the spatial search may be performed by considering all the different spatial positions neighboring the pixel (x, y). by considering all the pixels of the non-reference images having a spatial position that is specifically adjacent to (x, y). An alternative embodiment involves using only a subset of non-reference pixels located in a neighborhood specific to the spatial coordinates (x, y). As discussed previously for temporal fusing, an alternative approach may include matching a pixel block centered around a reference pixel with a corresponding pixel block surrounding each of the selected pixel candidate groups in non-reference images. The subset of non-reference pixel candidate sets may also be changed from one reference pixel to another reference pixel. This means that the pattern of non-reference pixel candidates when processing a new reference pixel may differ from the pattern used when processing the previous reference pixel.

Regardless of the approach used for the search, if one or more pixels for fusion with the pixels of the corresponding reference image are found in each non-reference image, the weights of the selected pixels may be calculated in a similar manner as before (Step 645). The calculated weight value may then be used to determine the value for the corresponding output image pixel (block 635).

It has been found that combining temporal and spatial fusing in this manner increases efficiency and also improves the quality of the output image. The reason is that searching for a better pixel is only performed when it is determined that the pixel is likely to be a ghost. This means that the search operation is typically performed on a limited number of pixels, rather than all pixels of the images, thereby significantly improving efficiency.

The temporal and spatial-temporal convergence operations discussed above work well when all images in the image sequence are relatively clear. However, there is no guarantee that all images in the image sequence will be clear. While it is true that the reference image is generally a sharp image, some of the non-reference images may be blurred due to camera or fast object motion. By blending the images as described in the previous operation, the blur present in any non-reference image can be seen in the output image. This is illustrated in Figures 7 and 8.

FIG. 7 shows four short-time exposed images 705, 710, 715, 720 of an image sequence being processed by an image stabilization operation in accordance with the present disclosure for generating an output image. As can be seen, the image 705 of the image sequence exhibits severe motion blur defects. FIG. 8A shows the resulting output image 805 of the fusion of images 705, 710, 715, 720 using temporal or spatial-temporal fusion. As can be seen, the blur present in the input image 705 also exists in the final output image 805. To prevent this problem, a multi-resolution fusing strategy may be used that effectively blurs the images in different frequency bands to effectively remove the blurring areas present in some input images.

Deterioration caused by blurry frames is predominantly at the edges of the image and in the neighborhood of high frequency textures. Conversely, in a smooth image region (e.g., a low spatial frequency band), the distribution of blurred frames can be useful for reducing noise. In one embodiment, the multi-resolution fusion approach utilizes this understanding by using blurred frames to blend low spatial frequency content and excluding them to blend image edges or high frequency textures.

One approach to implementing this approach is to decompose the images of the input image sequence into different spatial frequency bands and fuse each of them with such frequency bands. Multi-resolution image decomposition of different frequency bands is well known in the art and can be accomplished in a variety of ways. There can be one procedure that uses a high-pass pyramid decomposition algorithm. There may be other approaches that utilize wavelet decomposition. Other alternatives are also possible.

In a preferred embodiment, a highpass decomposition algorithm may be used. The algorithm may include generating a sequence of copies of the original image that is reduced to a regular step of the sample density and resolution to produce a plurality of intermediate layer original images. To accomplish this, the image may first be low-pass filtered and then down-sampled by a predetermined factor to obtain the next pyramid layer of the image. The predetermined factor may vary, which is a matter of design choice. In one embodiment, the predetermined factor is four. In an alternative embodiment, the predetermined factor is two. The number of layers of each image generated also depends on the needs and processing capabilities of the devices used. In one embodiment, the number of layers is four. In an alternative embodiment, the number of layers is three.

When all intermediate layers are generated in this manner, each layer is upsampled and low-pass filtered to the same resolution as the previous layer, and the result is subtracted from the previous layer to obtain a high frequency band component corresponding to the resolution in each layer . It should be noted, however, that the higher frequency band of the highest layer can not generally be obtained in this manner. In the generated pyramid, each layer is smaller than the previous layer and includes a higher spatial frequency band at that resolution. The top layer of the pyramid resembles a low resolution version of the original image and contains a low spatial frequency band. An example that includes three layers 905, 910, 915 is shown in FIG. The highest hierarchical image 905 is an image of a lower frequency representation. The layers show progressively higher resolution / frequency as they go from top to bottom. In this way, the original image can be decomposed into different spatial frequency bands, and these bands can be fused together individually.

When multiple layers are created for each of the images of the image sequence (including the reference image), the multi-resolution fusion operation 1000 begins by receiving all the layers for each of the images of the image sequence according to FIG. 10 (block 1005) . The highest layer image to be processed may then be selected (block 1010) and temporal fusion may be performed at the highest layer. By comparing each pixel in the uppermost layer of the non-reference images with the pixels of the corresponding reference image (block 1015) and calculating the weighting function for each non-reference pixel, Same as. After the weighting function is calculated for each pixel, the weighting may be compared to a predetermined threshold to determine if the pixel is a ghost (block 1025). If the calculated weight exceeds a predetermined threshold, the non-reference pixel may be used to calculate a value for the corresponding pixel in the top layer of the output image (block 1045).

If the weights are less than a predetermined threshold, operation 1000 may find a better match using a space-time technique by performing a spatial search in the neighborhood of the selected pixel (block 1030). The relative position or coordinates of the best match may then be stored in the corresponding field of the selected pixel (block 1035). The field refers to the field of corresponding pixels identified. The operation may then calculate a weighting function for the best matching pixel (block 1040) and use this value to determine the value for the corresponding pixel in the top layer of the output image (block 1045).

Once the processing for all the pixels in the top layer is complete, the operation may move to block 1050 to determine if there are other layers to process. If all layers are processed, the values may be available to pixels in all layers of the output image. The layers may then be combined or combined to produce a final output image (block 1060). Starting with the top-level pyramid layer and scaling up (i.e., upsampling and low-pass filtering) the output layer and then adding it to the next output layer. This operation can be repeated until all layers are combined and the output image has the same resolution as the input image. If it is determined that another layer remains in block 1050, then the corresponding field for each best match found in the current layer may be updated (block 1055). This can be done by considering the predetermined factor that each layer is downsampled and scaling up the location information of that field by the same factor to match the resolution of the next layer. The next layer to process can then be selected (block 1065) and the process begins at block 1015 and repeats. However, for this layer, the updated corresponding field may be used as an initial estimate of where to find the corresponding pixel for each reference pixel. Steps 1005-105 may be repeated until all layers have been processed and a final output image is generated according to block 1060. [

In this way, operation 1000 performs fusion at all pyramid layers, starting at the highest layer (low-frequency band) and ending at the highest resolution layer. At all layers, similarity between corresponding non-reference pixels and reference pixels can be used to prevent ghosting defects. At the top level of the pyramid, there may be an assumption that corresponding pixels have the same spatial coordinates. However, because of the moving objects in the scene, the spatial displacement between each non-reference pixel and its corresponding reference pixel for all non-reference layers, because the corresponding pixels do not have the same spatial coordinates in the next layer May be determined. It is determined that the non-reference pixel is likely to be a ghost if the weight is below a certain threshold. In such a case, a local search around its spatial location may be performed to find a better match with the reference pixel. The best match found subsequently can then be used for fusion using the weight associated with it and the spatial displacement of the best match found for the reference pixel coordinates can be maintained. Typically, such a search is only needed for a small percentage of pixels because most scenes are static. This increases efficiency because the process performs the search only when it is needed. The search may be performed by block matching as discussed above. By performing these steps, the approach 1000 utilizes all three fusion techniques discussed (temporal, spatial-temporal, and multi-resolution) to achieve an efficient convergence operation that significantly reduces or eliminates noise and blur To produce a high quality final output image.

Alternatively, the multi-resolution fusion approach may be performed by using only temporal fusion at each layer. Can be performed by fusing in each pyramid layer, using only those pixels with the same spatial coordinates in all images. Thus, a single pixel of each non-reference hierarchy can be fused with a corresponding reference pixel having the same spatial coordinates (at the corresponding reference image hierarchy).

In an alternative embodiment, in order to achieve the advantages of spatial-temporal fusion, a multi-resolution fusion procedure may utilize spatial-temporal fusion at each pyramid layer. In one embodiment, for every pyramid layer, it may be performed by fusing a reference pixel with more pixels of each non-reference layer. The spatial coordinates of the non-reference pixels fused with the reference pixels may be in a particular neighborhood around the reference pixel coordinates.

Other embodiments may perform fusion using a motion field. Starting from the highest layer of the pyramid and estimating the motion field for each non-reference image. In all layers of the pyramid, the motion field can associate the most similar reference pixel with the non-reference pixel for fusion. All reference pixels may then be fused with a single pixel of all non-reference layers, but their spatial coordinates may differ depending on the motion field of the layer.

Another embodiment may be utilized in which the reference pixel is fused with two or more pixels of any non-reference hierarchy. The spatial coordinates of the non-reference pixels may be in a particular neighborhood around the spatial coordinates proposed by the motion field.

As used herein, the term "camera" refers to any electronic device that includes or embeds digital image capture functionality. For example, it includes standalone cameras ( e.g. , digital SLR cameras and 'point-and-click' cameras) as well as other electronic devices with built-in camera functionality. Examples of the latter type include, but are not limited to, mobile telephones, tablets and notebook computer systems, and digital media player devices.

Referring to Fig. 11, a simplified functional block diagram of an exemplary electronic device 1100 is shown in accordance with one embodiment. The electronic device 1100 includes a processor 1105, a display 1110, a user interface 1115, graphics hardware 1120, a device sensor 1125 (e.g., a proximity sensor / ambient light sensor, an accelerometer, and / 1135, a communication circuit 1145, a digital image capture unit 1150, a video codec (s) 1155, a memory (memory) 1130, 1160, a storage device 1165, and a communication bus 1170. For example, the electronic device 1100 can be a digital camera, a personal digital assistant (PDA), a personal music player, a mobile phone, a server, a notebook, a laptop, a desktop, or a tablet computer. More specifically, the disclosed techniques may be implemented on a device that includes some or all of the components of device 1100.

The processor 1105 may execute the instructions necessary to perform or control the operation of many of the functions performed by the device 1100. [ For example, the processor 1105 may drive the display 1110 and receive user input from the user interface 1115. The user interface 1115 may have various forms, such as buttons, keypad, dial, click wheel, keyboard, display screen, touch screen, or a combination thereof. The processor 1105 may also be a system-on-chip such as, for example, provided on a mobile device and may include a dedicated graphics processing unit (GPU). The processor 1105 may be based on a reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architectures or any other suitable architecture, And may include one or more processing cores. Graphics hardware 1120 may be special purpose computing hardware for processing graphics and / or auxiliary processor 1105 processes graphics information. In one embodiment, the graphics hardware 1120 may include a programmable graphics processing unit (GPU).

The sensor and camera circuitry 1150 may be included at least in part within the video codec (s) 1155 and / or the processor 1105 and / or the graphics hardware 1120 and / or the circuitry 1150, Can capture still and video images that can be processed by a dedicated dedicated image processing unit. The images thus captured may be stored in memory 1160 and / or storage device 1165. Memory 1160 may include one or more different types of media used by processor 1105 and graphics hardware 1120 to perform device functions. For example, memory 1160 may include a memory cache, read only memory (ROM), and / or random access memory (RAM). Storage device 1165 may store media (e.g., audio, image and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 1165 can be any type of storage device such as, for example, magnetic disks (fixed, floppy, and removable) and optical media such as tape, CD-ROM, and digital video disk (DVD), and Electrically Programmable Read-Only Memory (EPROM) Electrically erasable programmable read-only memory (EEPROM), and the like. Memory 1160 and storage device 1165 may be used to tangentially store computer program instructions or code organized in one or more modules and written in any desired computer programming language. For example, when executed by processor 1105, such computer program code may implement one or more of the operations described herein.

It is to be understood that the above detailed description is intended to be illustrative, not limiting. Those skilled in the art will be able to make and use the claimed subject matter as described herein and are provided in the context of specific embodiments thereof, (For example, some of the disclosed embodiments may be used in combination with one another). For example, Figures 1-11 are described in the context of processing unprocessed or unprocessed images, but this is not necessary. An image stabilization operation in accordance with the present disclosure may be applied to the processed versions of the captured images ( e.g., edge-maps) or subsampled versions of the captured images. In addition, some of the operations described may have separate steps performed in accordance with or in a different order than the other steps presented herein. More generally, if there is hardware support, some of the operations described in conjunction with FIGS. 1-11 may be performed in parallel.

Many other embodiments will be apparent to those of ordinary skill in the art upon reviewing the above description. The scope of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "in which" are used as the plain English equivalents of their respective terms "comprising" and "wherein".

Claims (20)

A non-transitory program storage device comprising instructions readable and stored by a programmable control device, the instructions causing the programmable control device to:
Acquiring a first image of the initially captured scene, the first image having a plurality of pixels;
Performing a multiple resolution decomposition of the first image to produce a first multi-layer pyramid representation of the first image;
Acquiring a second image of the scene, the second image being captured at a different time than the first image, and each of the plurality of pixels of the first image having a corresponding pixel in the second image;
Performing a multiple resolution decomposition of the second image to produce a second multi-layer pyramid representation of the second image;
Layer pyramid representation of the scene for each layer of the first and second multi-layer pyramid representations of the scene in accordance with instructions, the instructions causing the programmable control device to:
Identifying a group of corresponding pixels in the second multi-layer pyramid representation of the scene for a group of pixels in the hierarchy of the first multi-layer pyramid representation of the scene,
To fuse a group of the identified pixels of the first and second multi-layer pyramid representations of the scene;
Repeating the instructions to cause the programmable control device to generate a layer of the output multi-layer pyramid representation of the scene for each layer of the multi-resolution decomposition of the first image;
Combine the output multilayer pyramid representations of the scene to produce a single output image representative of the scene;
Store the output image in a memory,
Wherein the instructions cause the programmable control device to fuse a group of the identified pixels of the first and second multi-layer pyramid representations of the scene to cause the programmable control device to select one of the first multi- Wherein the spatial coordinates of the two or more pixels are such that the spatial coordinates of the pixels of the second multi-layer pyramid representation identified corresponding to the one pixel In a predetermined neighborhood of the non-temporary program storage device. The method of claim 1, wherein the instructions cause the programmable control device to fuse a group of the identified pixels of the first and second multi-layer pyramid representations of the scene to cause the programmable control device to:
Determining a weight value associated with each group of pixels in the layer of the first multi-layer pyramid representation of the scene using the corresponding group of pixels in the layer of the second multi-layer pyramid representation of the scene ;
If the determined weight value is greater than a specified threshold, fuses a group of the identified pixels of the first and second multi-layer pyramid representations of the scene, and if the determined weight value is less than or equal to the specified threshold, 1 and a group of the identified pixels of the second multi-layer pyramid representation. 3. The method of claim 2, wherein the instructions for causing the programmable control device to determine a weight value cause the programmable control device to:
Compare the similarity between each pixel of the group of pixels in the layer of the first multi-layer pyramid representation of the scene and the corresponding pixel of the corresponding group of pixels of the second multi-layer pyramid representation of the scene ;
Obtain a pixel similarity value based on the comparison;
Obtaining an expected noise content for each pixel of the group of pixels in the layer of the first multi-layer pyramid representation of the scene;
And calculate the weight value based on the pixel similarity value and the expected noise content. 3. The apparatus of claim 2, wherein the instructions for causing the programmable control device to determine a weight value further cause the programmable control device to:
Perform a spatial search of the layer of the second multi-layer pyramid representation of the scene to search for a group of pixels corresponding to a better corresponding value if the weight value is less than or equal to the specified threshold;
Determine a weight value for the group of the better corresponding pixels;
And to cause a group of the identified pixels of the first multi-layer pyramid representation to fuse with the group of the corresponding corresponding pixels of the second multi-layer pyramid representation. The method of claim 1, wherein the instructions cause the programmable control device to fuse a group of the identified pixels of the first and second multi-layer pyramid representations of the scene to cause the programmable control device to:
Estimate a motion field for each of a second multi-layer pyramid representation of the second image, the motion field associating pixels of the first image with pixels of the second image;
And to cause each pixel of the first multi-layer pyramid representation to fuse with a selected pixel in the motion field. delete 2. The method of claim 1, wherein the instructions for generating a first multi-layer pyramid representation of the first image cause the programmable control device to generate a high-pass pyramid decomposition of the first image ≪ / RTI > instructions, and instructions. 2. The method of claim 1, wherein the instructions for generating a first multi-layer pyramid representation of the first image comprise instructions for causing the programmable control device to generate a wavelet decomposition of the first image. Non-transient program storage device. As a system,
Image capture device;
Memory; And
One or more programmable control devices
The one or more programmable control devices interacting with the image capture device and the memory,
Acquiring a first image of the initially captured scene, the first image having a plurality of pixels;
Performing a multiple resolution decomposition of the first image to produce a first multi-layer pyramid representation of the first image;
Acquiring a second image of the scene, the second image being captured at a different time than the first image, and each of the plurality of pixels of the first image having a corresponding pixel in the second image;
Performing a multiple resolution decomposition of the second image to produce a second multi-layer pyramid representation of the second image;
Layer pyramid representation of the scene for each layer of the first and second multi-layer pyramid representations of the scene in accordance with instructions, the instructions causing the programmable control device to:
Identifying a group of corresponding pixels in the second multi-layer pyramid representation of the scene for a group of pixels in the hierarchy of the first multi-layer pyramid representation of the scene,
And fusing a group of the identified pixels of the first and second multi-layer pyramid representations of the scene;
Combine the output multilayer pyramid representations of the scene to produce a single output image representative of the scene;
And storing the output image in a memory,
Wherein fusing the group of identified pixels of the first and second multi-layer pyramid representations of the scene fuses one pixel of the first multi-layer pyramid representation with two or more pixels of the second multi-layer pyramid representation Wherein the spatial coordinates of the two or more pixels are in a predetermined neighborhood of a pixel of the second multi-layer pyramid representation identified corresponding to the one pixel. 10. The method of claim 9, wherein fusing the group of identified pixels of the first and second multi-layer pyramid representations of the scene comprises:
Determining a weight value associated with each group of pixels in the layer of the first multi-layer pyramid representation of the scene using the corresponding group of pixels in the layer of the second multi-layer pyramid representation of the scene ;
If the determined weight value is greater than a specified threshold, fuses a group of the identified pixels of the first and second multi-layer pyramid representations of the scene, and if the determined weight value is less than or equal to the specified threshold, 1 and a group of the identified pixels of the second multi-layer pyramid representation. 11. The method of claim 10, wherein determining the weight value comprises:
Compare the similarity between each pixel of the group of pixels in the layer of the first multi-layer pyramid representation of the scene and the corresponding pixel of the corresponding group of pixels of the second multi-layer pyramid representation of the scene ;
Obtain a pixel similarity value based on the comparison;
Obtain an expected noise content for the group of pixels in the layer of the first multi-layer pyramid representation of the scene;
And calculating the weight value based on the pixel similarity value and the expected noise content. 11. The method of claim 10, wherein determining the weight value further comprises:
Perform a spatial search of the layer of the second multi-layer pyramid representation of the scene to search for a group of pixels corresponding to a better corresponding value if the weight value is less than or equal to the specified threshold;
Determine a weight value for the group of the better corresponding pixels;
And fusing the group of identified pixels of the first multi-layer pyramid representation with the group of the corresponding corresponding pixels of the second multi-layer pyramid representation. 10. The method of claim 9, wherein fusing the group of identified pixels of the first and second multi-layer pyramid representations of the scene comprises:
Estimate a motion field for each of a second multi-layer pyramid representation of the second image, the motion field associating pixels of the first image with pixels of the second image;
And fusing each pixel of the first multi-layer pyramid representation with a selected pixel in the motion field. delete As a method,
Acquiring a first image of an initially captured scene, the first image having a plurality of pixels;
Performing a multiple resolution decomposition of the first image to produce a first multi-layer pyramid representation of the first image;
Obtaining a second image of the scene, wherein the second image is captured at a different time than the first image, and wherein each of the plurality of pixels of the first image has a corresponding pixel in the second image;
Performing multiple resolution decomposition of the second image to generate a second multi-layer pyramid representation of the second image;
Generating a hierarchy of output multilayer pyramid representations of the scene for each layer of the first and second multilayer pyramid representations of the scene,
Identifying a group of corresponding pixels in the second multi-layer pyramid representation of the scene for a group of pixels in the hierarchy of the first multi-layer pyramid representation of the scene; And
Fusing the identified group of pixels of the first and second multi-layer pyramid representations of the scene;
Combining the output multilayer pyramid representations of the scene to produce a single output image representative of the scene; And
Storing the output image in a memory,
Wherein merging a group of the identified pixels of the first and second multi-layer pyramid representations of the scene comprises merging one pixel of the first multi-hierarchical pyramid representations with two or more pixels of the second multi- Wherein the spatial coordinates of the at least two pixels are in a predetermined neighborhood of a pixel of the second multi-layer pyramid representation correspondingly identified for the one pixel. 16. The method of claim 15, wherein fusing the group of identified pixels of the first and second multi-layer pyramid representations of the scene comprises:
Determining a weight value associated with each group of pixels in the layer of the first multi-layer pyramid representation of the scene using the corresponding group of pixels in the layer of the second multi-layer pyramid representation of the scene step; And
If the determined weight value is greater than a specified threshold, fuses a group of the identified pixels of the first and second multi-layer pyramid representations of the scene, and if the determined weight value is less than or equal to the specified threshold, 1 and a group of the identified pixels of the second multi-layer pyramid representation. 17. The method of claim 16, wherein determining a weight value comprises:
Comparing the similarity between each pixel of the group of pixels in the layer of the first multi-layer pyramid representation of the scene and the corresponding pixel of the corresponding group of pixels of the second multi-layer pyramid representation of the scene step;
Obtaining a pixel similarity value based on the comparison;
Obtaining an expected noise content for the group of pixels in the layer of the first multi-layer pyramid representation of the scene; And
Calculating the weight value based on the pixel similarity value and the expected noise content. 17. The method of claim 16, wherein determining the weight value further comprises:
Performing a spatial search of the layer of the second multi-layer pyramid representation of the scene to find a group of pixels corresponding to a better corresponding value if the weight value is less than or equal to the specified threshold;
Determining a weight value for the group of the corresponding corresponding pixels; And
And fusing the group of identified pixels of the first multi-layer pyramid representation with the group of the corresponding corresponding pixels of the second multi-layer pyramid representation. 16. The method of claim 15, wherein fusing the group of identified pixels of the first and second multi-layer pyramid representations of the scene comprises:
Estimating a motion field for each of a second multi-layer pyramid representation of the second image, the motion field associating pixels of the first image with pixels of the second image; And
Fusing each pixel of the first multi-layer pyramid representation with a selected pixel in the motion field. delete

Images