TW202338734A - Method and image processor unit for processing image data - Google Patents

Abstract

The invention refers to a method for processing image data (IMGRAW) of an image sensor (4), wherein the image data comprising a burst of frames captured by the image sensor (4) per image, each frame comprising a raw matrix of pixels. The method comprises the steps of: (A) determining motion vectors (MV1...N). wherein each forward motion vector in forward direction represents the displacement of a block in a selected anchor frame of the burst of frames to the best-matching block in an respective alternate frame of the burst of frames and each backward motion vector in backward direction represents the displacement of a block in a respective alternate frame of the burst of frames to the best-matching block in an selected anchor frame of the burst of frames; (B) determining reliability factors for the motion vectors determined in step (A) to assign the reliability of the respective motion vector for a given block by use of the difference between the forward motion vector and the related backward motion vector, wherein the reliability increases with decreasing difference; (C) aligning the frames of the burst of frames for an image, wherein the motion is compensated by use of weighted motion vectors, wherein the motion vectors are weighted with the respective reliability factor determined in step (B).

Description

Method and image processor unit for processing image data

The present invention relates to a method for processing image data of an image sensor, wherein the image data includes the original pixel matrix of each image, that is, the original image data.

The invention further relates to an image processor unit for processing raw image data provided by an image sensor, the image sensor comprising a sensor array providing a matrix of raw pixels per image.

Furthermore, the invention relates to a computer program configured to perform the steps of the aforementioned method.

Digital imagers are widely used in everyday consumer products such as smartphones, tablets, laptops, cameras, cars and wearable devices. The use of small imaging sensors is becoming a trend to maintain small, lightweight product form factors and reduce product costs. Even when imaging sensors with high numbers of megapixels are used, color filter arrays (CFA), such as the common Bayer color filter array, are often used to reduce costs. The use of color filter arrays limits or reduces spatial resolution because full-color images are produced through the interpolation (demosaicing) of undersampled color channels.

Resolution/dynamic range/noise limitations drive engineers to develop multi-frame computational photography image processing pipelines, also known as burst image signal processors (BISPs). To solve resolution limitations, dynamic range limitations and noise limitations. In BISP, frame bursts are captured preferably through predefined (eg, programmable) settings and the frames are blended together to achieve multiple goals.

"Burst photography for high dynamic range and low-light imaging on mobile cameras" by S. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy (ACM Transactions on Graphics, Volume 35, Number 6, November 2016, SIGGRAPH Asia 2016) describes multi-frame technology designed to reduce noise and increase dynamic range. A short pulse of underexposed frames is captured, aligned and combined to produce a single intermediate image with high bit depth, and this image is tone mapped to produce a high-resolution photograph. The merging method operates in the spatial frequency domain on image tiles that overlap by half at each spatial dimension. By blending smoothly between overlapping tiles, visually objectionable discontinuities at tile boundaries are avoided. Additionally, windowing functions must be applied to the tiles to avoid edge artifacts when operating in the DFT domain.

In the original image alignment solution present in this reference, each 2×2 block (quad) in the RGGB Bayer raw data is averaged to produce a reduced-size grayscale image (1/4 resolution of the original CFA data Spend). Multi-scale (cone) motion estimation is performed on downsized grayscale images. Pursuing block matching is used for motion estimation at each scale, where the L2 cost function is minimized at all levels of the image cone except at the finest scale (reduced grayscale image), where the L1 cost function is minimized minimize. Subpixel-level accuracy is searched during motion estimation at all scales of the cone except at the finest scale where pixel-level accuracy is searched. This strategy effectively limits the pixel shift between original frames to a multiple of 2. This constraint is considered sufficient for applying multi-frame denoising and high-dynamic-range (HDR) fusion.

"Handheld Multi-Frame Super-Resolution" by B. Wronski, I. Garcia-Dorado, M. Ernst, D. Kelly, M. Krainin, C. K. Liang, M. Levoy, and P. Milanfar (ACM Transactions on Graphics, pp. 38 Volume 4, Article 28, July 2019, SIGGRAPH 2019) reveals multi-frame super-resolution MFSR, in which a short pulse of raw frames shifted in sub-pixel dimensions by normal hand shake or hand tremor is fused to produce higher resolution frames. The raw (Bayer CFA) image of the captured short pulse is input to the algorithm. Each frame is locally aligned with a single basic frame. The contribution of each frame at each pixel is estimated via kernel regression, and these contributions are accumulated separately per color channel. The kernel shape is adjusted based on the estimated local gradient, and the sample contributions are weighted based on the robustness model, which is calculated per pixel for each frame using the alignment field and local statistics collected from the neighborhood around each pixel. weight. The final merged RGB image is obtained by normalizing the accumulation results of each channel. This process of incorporating a short pulse frame has the effect of increasing the perceived resolution or simply enabling applications such as being able to select the best photo or to zero shutter lag usage. Propose three iterative additional steps of Lukas-Kanade optical flow image warping, which achieve sub-pixel level accuracy because pixel-level accuracy is not sufficient for multi-frame super The purpose of the fusion algorithm.

The most modern digital single lens cameras (SLR) support continuous or short-pulse shooting. Burst mode is supported to enable selecting the best photo, or performing complex multi-frame processing, such as the multi-frame super-resolution feature. The burst rate (that is, the number of frames obtained in rapid alternation) varies, but has increased as camera technology has evolved.

An essential step in the development of many multi-frame (short pulse) processing solutions is image alignment. In this step, motion between captured frames or motion of a selected frame with respect to an anchored (reference) frame is estimated, and the frames are then aligned or buffered to compensate for global motion and/or local motion. . The aligned frames can then be blended in a variety of ways depending on the desired features. Image alignment can occur in the native color filter array (CFA) domain or the full color (e.g., RGB) domain, can be designed for pixel-level accuracy or sub-pixel level accuracy, and can Customized to fit a variety of motion models, such as translation, similarity, affine, and projective motion models.

Since the goal of multi-frame processing is usually to overcome imaging sensor limitations and improve final image quality, the alignment (and fusion) of frames in the original CFA domain is optimal. However, image alignment in the raw domain poses challenges primarily due to insufficient color sampling (50% green, 25% green, and 25% red) typical of standard RGGB Bayer sensors such as in CFA. Additionally, it is necessary for the alignment algorithm to achieve arbitrary precision/accuracy to support solutions requiring pixel-level accuracy or sub-pixel-level accuracy. Furthermore, in real-time resource constrained environments (as in many consumer camera products), it is desirable that image alignment will be computationally inexpensive due to speed, power, and memory constraints.

Image alignment involves motion estimation in short burst frames. Motion estimation techniques that support global and local motion estimation and multiple motion models ranging from pixel-based solutions to feature-based solutions are well known in the art.

"A Study on Block Matching Algorithms for Motion Estimation in Video Coding" by L. C. Manikandan and R. K. Selvakumar (International Journal of Scientific & Engineering Research, Volume 5, Issue 7, July 2014) describes the different methods used in motion estimation. Block matching algorithm. Block matching via exhaustive search (brute force search) is computationally quite expensive. Significant speedup of searches may be achieved, for example, by diamond searches or other fast search algorithms.

Evaluation of R. Yaakob, A. Aryanfar, A. A. Halin and N. Sulaiman's "A Comparison of Different Block Matching Algorithms for Motion Estimation" (The 4th International Conference on Electrical Engineering and Informatics (ICEEI 2013)) for motion estimation Part 4 Different block matching algorithms, namely Exhaustive Search (ES), Three-Step Search (NTSS), Simple and Efficient Search (SES) and Adaptive Cross Pattern Search (Adaptive Rood Pattern Search, ARPS).

R. Hartley and A. Zisserman's "Multiple View Geometry in Computer Vision" (University Press, Cambridge, 2004) and G. H. Golub and C. F. Van Loan's Matrix Computations (2nd edition, 1989) each describe in detail the methods used to process image sense. Algorithm for detector data.

The object of the present invention is to provide an improved method and image processor unit for processing image data of an image sensor.

The object is achieved by a method having the features of claim 1 , an image processor unit having the features of claim 18 and a computer program having the features of claim 21 . Preferred embodiments are described in the accompanying claims.

In order to achieve improved alignment of the frame or parts thereof (e.g., blocks within the frame), the method includes the following steps: A) Determine motion vectors, where each forward motion vector in the forward direction represents a block in the selected anchor frame of one of the burst frames to a respective alternative frame of the burst frame The shift of the best matching block, and each backward motion vector in the backward direction represents one of the short pulse frames respectively replacing a block in the frame to one of the short pulse frames selected The displacement of the best matching block in the anchor frame; B) Determine reliability factors for the motion vectors determined in step A) to assign the respective motion vectors to a given block using the difference between the forward motion vector and the associated backward motion vector The reliability of , where the reliability increases as the difference decreases; C) the frames aligned with the burst frame of an image, wherein the motion is compensated using weighted motion vectors, wherein the motion vectors are weighted by the respective reliability factors determined in step B) .

Alignment can be used to temporarily store a frame or portion thereof to a respective image position within a related frame or portion thereof. A frame refers to a complete image captured by an image sensor, or a portion of a complete image, that is, a frame with a specific full size or reduced size. The frame may be aligned, for example, by aligning the complete frame or the complete image using global motion vectors of the complete frame or image, or by aligning multiple blocks of the frame using respective local motion vectors of the associated blocks. to execute.

For example, the frames considered for steps A) to C) may be a selected region of interest (one of a larger frame captured or capable of being captured by the image sensor). ROI).

Using weighted motion vectors has the effect of improving the quality of the motion vectors by over-weighting more reliable motion vectors and lightly weighting less reliable motion vectors. Weighting in this sense also includes using only two weighting factors zero and one to take into account motion vectors with reliability above a given threshold and completely ignoring others with reliability below the given threshold. motion vector.

Depending on the definition of the threshold value, "above" or "exceeds" may also be understood as "equal to and above" or alternatively "below" may be understood as "equal to or below".

The method may be performed by estimating motion directly from raw color filter array (CFA) image data of the image sensor. Bidirectional motion estimation allows adaptive selection of anchor (reference) frames. Motion estimation results via explicit identification of unreliable motion estimates, leading to improved robustness.

Neither iteration/optimization nor multi-scale/pyramidal representation is required, allowing the method to be implemented in a digital signal processor or image data processing software with reduced computational effort and hardware expense. The method allows implementation with low computational complexity. It is possible to implement the method with a complexity that is linear in several pixels.

Motion can be estimated with arbitrary accuracy, such as pixel-level and/or sub-pixel-level accuracy. Subpixel-level motion estimation with arbitrary accuracy is achievable without using any iteration or multi-scale procedures. The estimated sub-pixel level motion vectors can be used to derive parameters of various motion models to support global and/or local motion.

The method supports global motion and local motion, and different motion models.

Methods may be implemented to perform true and direct image staging (ie, alignment) in the original domain for the selected color channel with the highest sampling/strongest response. Since the offsets of the other color channels in the CFA with respect to the selected color channel are known and fixed, the temporary storage of their color channels is performed implicitly without any additional computational effort.

The method explicitly detects unreliable motion vectors by means of reliability criteria. Due to the availability of forward and backward motion vectors, there is the possibility of adaptive/variable selection of reference frames.

A plurality of different motion vector reliability rules may be applied to obtain at least one reliability factor for a respective frame or part thereof, such as a set of reliability factors for a block, where each reliability factor of the set is represented by the associated Reliability rules are obtained such that the set includes reliability factors obtained by different predefined reliability rules.

Using a first reliability rule, if the difference between the forward motion vector and the associated backward motion vector exceeds a predefined threshold, the resulting reliability factor of zero (0) may be assigned to a A set of forward motion vectors and associated backward motion vectors, eg, to motion vectors determined for respective blocks or frames and used for further processing, such that these motion vectors are ignored in the further processing. The motion vectors determined for individual blocks or frames and used for further processing may be, for example, only forward motion vectors.

Using a first reliability rule, if the difference between the forward motion vector and the associated backward motion vector is below a predefined motion consistency threshold, a reliability factor of one (1) may be assigned To a set of forward motion vectors and associated backward motion vectors, eg to motion vectors determined for respective blocks or frames and used for further processing, such that these motion vectors are fully taken into account in the further processing. The motion vectors determined for individual blocks or frames and used for further processing may be, for example, only forward motion vectors.

The difference between the forward motion vector and the associated backward motion vector can be determined as the L1 sum of absolute difference (SAD) calculated from the minimum absolute deviation by the illustrative formula: , in represents the forward motion vector, and Represents the backward motion vector, where i = 1,2..., M, and j = 1,2,..., N, where M and N are the sizes of the M × N frame.

Using a second reliability rule, a motion vector may be assigned a value of zero (0) if the magnitude of at least one of the x component in the x direction or the y component in the y direction is equal to the block matching search size. Reliability factor to this motion vector. Therefore, if the magnitude of the x and/or y components of a given block's motion vector is trimmed to the block match search size, the motion vector is likely to be untrustworthy. Otherwise, the reliability factor associated with this second rule may be set to one (1).

Using the second reliability rule, a reliability factor can be assigned to a motion vector of a given block using the difference between the respective motion vector and the motion vector of the adjacent block. This situation imposes a smoothness constraint on the frame's motion vector field.

This situation can be achieved by: if at least the magnitude of the x-component of the motion vector in the x-direction is consistent with the motion of a set of blocks located in the direct and/or indirect neighborhood of the selected block The difference between the average magnitude of the x-component of the vector in the x-direction exceeds a predefined threshold, or at least the magnitude of the y-component of the motion vector in the y-direction is located between the selected block The third rule of assigning zero (0) if the difference between the average magnitudes of the y-components in the y-direction of the motion vectors of a set of blocks in the direct and/or indirect neighborhood exceeds a predefined threshold Reliability factor to the motion vector of a given block.

During block matching in step a), the sum of absolute differences can be calculated. The fourth rule can be implemented for the purpose of identifying only high-quality motion vectors. A fourth rule reliability factor may be assigned to the motion vector of a given block corresponding to the motion vector of the best matching block determined using a sum of absolute differences (SAD). Therefore, after estimating the absolute difference and SAD during block matching for a given block, the SAD value corresponding to the best matching motion vector is compared with a predefined threshold value, and if the SAD value is higher than the threshold value, Then the corresponding motion vector is considered unreliable, that is, has a reliability factor of zero. Otherwise, the motion vector is considered reliable, so that the associated reliability factor can be set to one.

If the sum of absolute differences (SAD) of the motion vector corresponding to the best matching block exceeds a predefined threshold, a reliability factor of zero (0) may be assigned. In this case, motion vectors are completely ignored for further processing of the image.

The overall reliability factor can be calculated by combining a set of reliability factors assigned to the motion vectors of a given block. This situation can be calculated simply by multiplying the set of reliability factors, which are determined using different rules for the associated motion vectors.

The motion vectors may be weighted, for example, by respective overall reliability factors in step C).

The method may be performed using any applicable strategy for evaluating matching of portions of an image, in particular block matching.

The best matching block can be determined, for example, by calculating the sum of absolute differences (SAD) between the selected block in a first frame and the searched block in a second frame. Searching The plurality of blocks have different relative positions from each other in the second frame; and the best matching block is determined to be the block with the smallest sum of absolute differences (SAD).

A global motion vector can be determined by calculating an average motion vector from the set of weighted motion vectors of the frame, each by a respective total reliability factor or for a given block of the frame. The motion vector determination is weighted by at least one of the reliability factors of the motion vector determination.

Estimating motion with sub-pixel level accuracy can be achieved by following the following steps: - Align the alternative frame and a selected anchor frame using the respective global motion vectors of the alternative frame; - Determine the sub-pixel level forward motion vector for each pair of blocks in the alternative frame and anchor frame respectively; - determine a global sub-pixel level motion vector by calculating an average motion vector from the set of sub-pixel level forward motion vectors determined for the block of the frame; and - Align the alternative frame and the anchor frame using the global sub-pixel level motion vector determined for the alternative frame.

For each block of the frame, a separate local sub-pixel level motion vector can be determined. At least one reliability factor can be determined for each local sub-pixel level motion vector, and the blocks in the replacement frame can be compared with the anchor map using the local sub-pixel level motion vectors determined for respective blocks of the replacement frame. Frames are aligned pixel by pixel (i.e., segmented tile by tile for each tile of the frame).

If information from some motion-related auxiliary sensors such as accelerometers/gyros and depth sensors is available, the motion estimation procedure can be supported by this information and the motion vector quality can be further improved. Preferably, the sensor signal may be used for each of the anchor frame and the replacement frame. This situation can be used to solve problems where, for example, the motion model is valid for full range motion and may not be sufficiently accurate for objects that are not at the same distance from the imaging system.

Image sensors capture image data to provide multiple color channels. This may also include white or gray.

If each frame (ie, image) contains multiple color channels, then the estimation of motion vectors is performed on the selected channel with the highest information density, ie the highest sampled color channel. In the case of a Bayer color filter array, this color channel is the green channel due to the pattern R-G-G-B. The method therefore includes the following steps: selecting the color channel with the highest information density compared to the information density of the other channels of the plurality of channels; aligning the frame of the selected color channel by inserting missing pixels in the selected color channel Original pixel matrix; and proceed with steps A) to C) for the inserted pixels of this selected color channel.

The goal is also achieved by an image processor unit for processing raw image data provided by the image sensor. The image sensor includes a sensor array that provides frame-by-frame short pulses captured by the image sensor. Each frame includes an original pixel matrix for each image, an original pixel matrix.

According to the invention, the image processor unit is configured to: A) Determine motion vectors, where each forward motion vector in the forward direction represents a block in the selected anchor frame of one of the burst frames to a respective alternative frame of the burst frame The shift of the best matching block, and each backward motion vector in the backward direction represents one of the short pulse frames respectively replacing a block in the frame to one of the short pulse frames selected The displacement of the best matching block in the anchor frame; B) Determine reliability factors for the motion vectors determined in step A) to assign the respective motion vectors to a given block using the difference between the forward motion vector and the associated backward motion vector The reliability of , where the reliability increases as the difference decreases; and C) the frames aligned with the burst frame of an image, wherein the motion is compensated using weighted motion vectors, wherein the motion vectors are weighted by the respective reliability factors determined in step B) .

The image processor unit is configured to process image data by performing the aforementioned method steps.

The object is therefore further solved by an image processor unit comprising the features of one of claims 18, 19 or 20. The object is further solved by a computer program containing instructions which, when executed by a processing unit, cause the processing unit to perform the steps of the aforementioned method.

Figure 1 is an exemplary block diagram of an electronic device 1 including a camera 2 and an image processor unit 3. The electronic device 1 is used to process raw image data IMG _RAW provided by the image sensor 4 of the camera 2.

The image sensor 4 includes a pixel array, so that the original image IMG _RAW is a data set in the original pixel matrix of each image. In order to capture the colors in the image, a color filter array CFA is provided in the optical path in front of the image sensor 4 . The camera contains an opto-mechanical lens system 5, such as a fixed uncontrolled lens.

The electronic device 1 further includes a mechanical actuator 6 . Mechanical actuators 6 are provided in electronic devices primarily for other purposes, such as signaling to the user. This is a familiar feature of smart phones used to send incoming messages or calls.

In this regard, the electronic device 1 may be a handheld device, such as a smartphone, a tablet, a wearable device, a camera, or the like.

The image processor unit 3 is configured to process image data IMG _RAW from the image sensor 4 capturing image bursts/frames F _1...N of an image. The image processor unit 3 is configured to process the estimated motion vector MV and align the pixel matrix of several bursts of the captured image with a specific alignment, and combine the image bursts to utilize the original matrix available for each pixel location and a plurality of pixels of the matrix of the resulting image to achieve the resulting image IMG _FIN .

Figure 2 illustrates a flow chart of a method for processing raw image data IMG _RAW of the image sensor 4. The original image data IMG _RAW contains image short pulses, or frame short pulses F _REF , F _{ALT_1} , F _{ALT_2} , ... , F _{ALT_N} captured for an image. The raw image data IMG _RAW is processed by automatically performing at least steps a) to h) on a separately configured digital signal processor or by software executing on the image processor. Steps a) to h) include a similar routine for dividing the steps so that they are performed on separate frames (i.e., steps a) = {a0), a1), ...aN)}; step b) = {b0) , b1), …, bN)}) options.

The basic assumption in the flow chart of Figure 2 is that the goal is to align the frame F _REF , F _{ALT_1} , F _{ALT_2} , ... , F _{ALT_N} and therefore pursue global motion estimation. However, the proposed method can be generalized to local motion estimation, for example by incorporating a motion segmentation step using the estimated local motion, or performing motion estimation for predefined regions in the frame. a) Color Channel Interpolation (CCI)

The first step a) or a set of steps a0), a1), a2), ... , aN) of each frame of the frame pulse of the image is designed for color channel insertion CCI.

In the original CFA data, colors are undersampled in standard RGGB Bayer sensors, such as 50% green, 25% blue, and 25% red. It is proposed to perform alignment on the color channel in the CFA that has the highest sampling and/or the channel that gives more detail/higher response. Examples of this channel are the green channel in a standard RGGB Bayer CFA or the white channel in an RGBW Bayer CFA. However, as mentioned, the color channels in CFA are often undersampled. Therefore, the first step in the image alignment pipeline is to insert the selected color channels to fill the gaps due to missing samples. Interpolation is performed only where samples are lost, resulting in full resolution color channel data. Detail-preserving insertions such as bicubic/bilateral insertions can be pursued. However, if computational complexity is a concern, simpler interpolations such as bilinear interpolation can be performed.

Motion estimation will then be performed on the interpolated full resolution color channels. Since the positions of the other color channels in the CFA are known with respect to the selected channel, motion estimation is performed implicitly for those channels without additional computational effort. For simplicity, it is mentioned that the full-resolution color channel selected for insertion is C. The anchor (reference) is represented by C _anchor , and the frame region of interest (ROI) of the alternative frame/frame is represented by C _alternative . b) Color Channel Smoothing (CCS)

Then in step b), or in the division of the routine into steps b0), b1), b2), ..., bN) for respective frames, color channel smoothing CCS is performed.

This step b) is designed to be a low-pass filter both C _-anchored and C- _substituted for the selected color channel. This operation is performed to harden the motion estimation operation against noise in the original data. Low-pass filtering is achieved by convolving C- _anchoring and C- _substitution using a smoothing filter, such as a two-dimensional Gaussian filter. Smooth image by and to express, among which And F is a 2-dimensional smoothing filter, and ⨂ represents a 2-dimensional convolution operation. c) Fast forward and backward block matching (Forward and Backward Block Matching, FB-ME)

The following step c) (reflected by step A) in the patentable scope) is designed to estimate alternative frames (including the option of alternative ROIs of the frame) with respect to the anchor (reference) frame (or the ROI of the anchor frame) ) motion vectors in the forward and backward directions.

Block matching (BM) is well-known and by far the most popular motion estimation method used in various image/video processing applications. The image can be divided into, for _example , M × N rectangular blocks, each of size _L It is executed in a message window to find the best matching block, which minimizes a predefined block distortion measure (for example, the so-called Displaced Frame Difference (DFD)).

Match searches can be performed within ±P pixels, that is, up to P pixels in the ± x and ± y directions are supported, as depicted in Figure 3.

Block matching BM via exhaustive search (brute force search) is computationally quite expensive. Significant acceleration of searches may be achieved, for example, via diamond searches or other fast search algorithms known in the art.

The motion vector MV is calculated by finding the best match, which achieves the minimum block distortion measure (±P pixels) in the search window; the motion vector MV of a given block is calculated as the reference frame block to the substitute frame minimum Shift of the best matching block. Of course, the implicit underlying assumption in block matching BM is that within small blocks, motion can be modeled as translational. Furthermore, the brightness is assumed to be constant. Although this latter assumption may not hold for example in frames acquired at different exposure times, a preprocessing photometer alignment can be performed before block matching BM in order to find the motion vector MV under the assumption of constant brightness.

The third step c) in the motion estimation pipeline is designed to perform block matching BM to estimate the smoothed color channel and Movement between image data. Block matching is in the forward direction (toward the match ) and backward direction (towards match ) are executed on both.

Bidirectional block matching BM for motion reliability estimation allows estimation of reliability factors for individual motion vectors, as explained later with respect to step d). It is also worth mentioning that having both forward and backward motion estimation results available can facilitate the adaptive selection of anchor (reference) frames for subsequent multi-frame fusion operations.

Fast block matching searches can be achieved by using a diamond search strategy. At this stage, only pixel-level accuracy is considered. Due to low computational complexity, the block matching/distortion criterion used is the sum of absolute differences (SAD) between the block in the anchor frame and the block under search of the alternative frame (in forward mode and vice versa in backward mode). In order to speed up the determination of the motion vector MV for each block, only selected pixels of the block to be matched can be used to calculate the shifted frame difference DFD, for example by subsampling the block.

For example, skipping every other pixel in the x and y directions will reduce the computational effort by a factor of four. The larger the block size, the more accurate the comparison, but at the same time the resolution of the vector field is lower, which results in inaccurate vectors, especially at the edges of objects. In order to achieve high accuracy and resolution, a larger area than the actual block size is usually used to calculate the shifted frame difference DFD.

For this purpose, additional _pixels may be included around the boundaries of the block and form an enlarged area called a wide block with size _WLx × _WLy , where _WLx≥Lx and _{WLy≥Ly .}

If the image is divided into M × N rectangular blocks, then the motion vector MV of block (i,j) is represented by [ u(i,j),v(i,j) ], where i = 1,2, ……, M and j = 1, 2, ……, N.

Collection of MVs Represents the motion vector field of the alternative frame (in forward mode), i = 1,2,… ,M , and j = 1,2,…, N . d) Motion Vectors Reliability Calculation (MV-R)

Many factors can lead to unreliable estimates of motion vectors MV, such as occlusions, motion blur, reflections (to name just a few). Furthermore, if the x and/or y absolute components of the true motion vector MV exceed the search size P of the matching BM, the estimated motion vector MV will not be correct and the magnitude of the excess components will be pruned to the search size P. Therefore, in the fourth step d) (reflected by step B) of the proposed motion estimation pipeline), it is necessary to be able to detect the unreliable motion vector MV such that the value of the unreliable motion vector MV is within Subsequent analyzes used to find global motion parameters of the frame (or local motion, if necessary) will be lightly weighted, such as completely excluded.

Preferably, a plurality of different motion reliability rules are applied to estimate respective reliability factors assigned to motion vectors for a given frame or part thereof (eg, a block). The reliability factors in the set of reliability factors assigned to a common motion vector for a given frame or a part thereof may be combined to achieve a motion vector assigned to a given frame, ie, a complete frame, a block, or the like. the overall reliability factor. Sports Reliability Rule #1:

As mentioned earlier, in the third step c), forward and backward motion estimation are performed. Ideally, after block matching BM, the forward motion vector MV should be opposite to the backward motion vector MV; that is, the respective x and y components of the forward and backward motion vectors should be zero. Assume the following fact: unreliable motion estimation (due to occlusions, saturated pixels, motion blur, shadows, reflections and other local changes in the scene) can be detected by calculating the difference between the forward motion vector MV and the backward motion vector MV If the difference is neither zero nor a minimum value, the motion vector MV of the given block is declared unreliable.

Mathematically, this situation can be expressed as follows. - For a given block (i,j), let [ u _forward (i, j), v _forward (i, j)] represent forward MV, and let [ u _backward (i, j), v _Backward (i,j)] represents backward MV. -The L1 distance between their two MVs is calculated and compared with the predefined threshold value _TMV , and if the L1 distance exceeds the threshold value, the block The binary reliability factor is set to zero (0), otherwise it is set to one (1). In the case of pixel-level accuracy and to ensure high quality motion vectors, T _MV can be set to zero (0). Otherwise ( i, j ) = 1 The first rule reliability factor of block (i, j) is represented by R ₁ (i, j) . Sports Reliability Rule #2:

As introduced earlier, for a given block, if the magnitude of the x and/or y components of the motion vector MV is equal to (pruned to) the block matching BM search size P, then it is most likely that the given block is not reliable. Therefore, the additional motion vector MV binary reliability factor R ₂ (i,j) is calculated as follows. like or → (i,j) = 0 otherwise (i,j) = 1 The second rule reliability factor of block (i, j) is given by (i,j) to represent. Sports Reliability Rule #3:

The two reliability measures presented above are further supplemented by a third reliability rule, which implicitly imposes a smoothness constraint on the motion vector field of the frame. For a given block (i, j), if the corresponding motion vector MV is significantly different from the motion vector MV of the adjacent block in the small window, the motion vector MV is considered unreliable.

Mathematically, this situation is expressed as follows: or Otherwise (i,j) = 1 where umean _and vmean _are , for example, the average x and y components of the motion vectors MV of adjacent blocks in the 3×3 block neighborhood of the block under consideration. And T _u and T _v are tunable parameters that control the strength of the smoothness constraint of the motion vector MV. Sports Reliability Rule #4:

The aforementioned three reliability measures are further supplemented by a fourth reliability rule whose purpose is to identify only high-quality motion vector MV estimates as reliable. To achieve this, after estimating the absolute difference and SAD during block matching for a given block, the absolute difference and SAD values corresponding to the best matching motion vector MV are compared with a predefined threshold value, and if the SAD If the value is higher than the threshold value, the corresponding motion vector MV is considered unreliable, that is, the fourth rule reliability factor R ₄ (i,j) = 0. Otherwise, the corresponding motion vector MV is considered reliable, that is, the fourth rule reliability factor ( i,j ) = 1. Combined to form the total reliability factor:

For a given block (i,j), the total motion vector MV binary reliability is then calculated as follows: where R _MV (i,j) represents the total reliability factor of the motion vector assigned to a given block. If the reliability factors R ₁ (i,j), R ₂ (i,j), R ₃ (i,j), R ₄ (i,j) are set to the on-off state by digital zero or one value, then once they are reliable At least one of the factors R ₁ (i,j), R ₂ (i,j), R ₃ (i,j), R ₄ (i,j) is zero, the total reliability factor R _MV (i,j) is zero. This situation can simply be implemented using a logical AND gate. e) Pixel-level global motion parameters

In the following, the options for calculating global motion parameters at the pixel level are explained. In detail, two important motion estimates related to selection are highlighted. 1) Motion model

Although the translational motion vector MV is calculated by the block matching BM strategy, this does not mean that the global motion of the frame needs to be modeled as a translational motion. For example, an affine motion model can be assumed for the entire frame, and its parameters can be calculated from reliable translational motion vectors. Therefore, depending on the chosen motion model, the deviation of the global motion parameters will be as follows, as briefly described. 2) Movement accuracy

Depending on the target application, motion estimation can be performed taking into account pixel-level accuracy or sub-pixel-level accuracy. The block matching BM can be computationally more complex if it will be performed with sub-pixel accuracy. Therefore, block matching BM with pixel-level accuracy is preferably performed, followed by a non-recurrent non-interpolation step of motion estimation with arbitrary sub-pixel accuracy. That is, the motion vector MV is calculated with arbitrary sub-pixel accuracy while still keeping the overall computational complexity low.

Since the block matching BM has been performed to calculate the pixel-level motion vector MV of the anchor frame and the replacement frame, and has been calculated again in the fifth step e) of the proposed motion estimation pipeline illustrated in Figure 2 Corresponding reliability, the global motion parameters of the surrogate frame with respect to the anchor frame (or vice versa, if necessary) are thus calculated.

Let the set of reliable motion vectors be represented by Ω _MV , where ,in

The global robust translation/shift between the anchor frame and the alternative frame is calculated from the reliable motion vector MV in Ω _MV . One possible robust estimate is the average of all their reliable motion vectors MV, eg the median, as shown below. The result of this operation is the global motion vector MV represented by [ u _global , v _global ]. uuniversal ₌ median value and vuniversal ₌ median value f) Alternative frame alignment including calculation of global motion parameters at sub-pixel level

Subpixel accuracy is critical for various subframe processing features, such as multiframe super-resolution operations. Any accuracy is essential to achieve high-quality image staging and thereby high-quality subsequent multi-frame processing. To achieve this goal, in the sixth step f) (reflected by step C) in the proposed motion estimation pipeline patent application), a non-iterative non-instrumentation method is similar to that of S. Chan referred to in the above description of the prior art. The method described in "Subpixel Motion Estimation Without Interpolation" by et al. was used. Alignment is based on Taylor approximation.

First, the replacement frame is aligned with the reference frame based on the estimated global pixel-level motion vector MV. Let B _substitute ( i,j ) and B _anchor ( i,j ) represent the (i, j)th block in the replacement frame and anchor frame after global alignment respectively, and (i,j ) _ΩMV . The sub-pixel motion estimate of the aligned replacement frame can then be calculated as follows: 1. For each B _replacement (i,j) and B _anchor ( i,j ) block pair, the pixel-level forward motion vector [u _{sub Pixel} ( i,j ), v _sub-pixel (i,j)] can be calculated as the solution of the following system of equations 2. In order to derive the global sub-pixel level motion estimate of the replacement frame, the global robust sub-pixel level motion vector MV can be calculated from the sub-pixel level motion vector MV in Ω _MV . One possible robust estimate is the median of all sub-pixel level motion vectors MV, as shown below. The result of this operation is the global sub-pixel level motion vector MV represented by [ u _{global , subpixel} , v _{global , subpixel} ]. u _{global , subpixel =} middle value v _{global , subpixel =} middle value

The total global sub-pixel level motion vector ( u _{global , total} , v _{global , total} ) can then be calculated as follows ( u _{global , total} , v _{global , total} ) = ( u _{global ,} v _global ) + ( u _{global , sub-pixel} , v _{Global , sub-pixel} )

After the global sub-pixel level motion vector MV has been estimated, the replacement frame _{FALT_k} may be aligned with the anchor frame F _REF . The alignment error signal expected to be at the motion vector MV is not reliable (i.e., (i,j ) = 1) and is expected to be high in the region where it is extremely useful information (coupled with R _MV ) for subsequent multi-frame processing.

At this point, for the purpose of illustrating the steps of the motion estimation pipeline, one example of the estimated motion vector field is for a surrogate map with ground truth global displacements of ΔX = -0.25 and ΔY = 4 with respect to the anchor frame. The frames are presented in Figures 4 to 6.

Local changes occur because, for example, objects in the captured image, that is, a car moving on the right side of the scene, along with its shadows and movement of leaves and other reflections and small changes, are present in the scene.

Figure 4 presents the estimated motion vector field with pixel-level accuracy in the horizontal X-direction and vertical Y-direction of the frame without any reliability checks. This is the result of block motion estimation for L _x = L _y = WL _x = WL _y = 32 and search size P = 6 pixels. In some areas, such as in the X,Y segment from about (40, 5) to (55, 15), there is high irregularity in the motion vectors.

Figure 5 presents the estimated weighted motion vector field with pixel-level accuracy components for the horizontal X-direction and vertical Y-direction of the frame. Ignore unreliable motion vectors. Shown is the reliable local motion vector MV after excluding the motion vector MV with an overall reliability factor of zero. The calculation results of the global pixel level motion vector MV of the entire frame are ΔX = 0 and ΔY = 4.

Figure 6 presents the estimated weighted motion vector field with sub-pixel level accuracy in the horizontal X-direction and vertical Y-direction of the frame. Furthermore, unreliable motion vectors are ignored. The sub-pixel level motion vector MV is added to the pixel level motion vector MV in Figure 5 to obtain the total sub-pixel level motion vector MV. The calculation results of the global sub-pixel level motion vector MV of the frame are ΔX = -0.25 and ΔY = 4.

The aforementioned exemplary embodiments are based on a translational motion model. However, the method is not limited to translational motion models and can be applied to other higher order models, in particular non-translational motion models.

To elaborate on this aspect, we assume that an affine model is chosen for global motion. The affine model can then be expressed as

The parameters t _x and t _y capture the two-dimensional translation in the x and y directions respectively. Rotation is captured by parameters a ₁ , a ₂ , a ₃ and a ₄ , where scaling is captured by scaling parameters a ₁ and a ₄ , and where shear is captured by shear parameters a ₂ and a ₃ . There are various ways, one way is to derive the six parameters of the affine motion model from the calculated translational motion vectors. An exemplary approach is explained below. 1. Instead of global alignment of the replacement frame based on the calculated global pixel-level motion vector MV, block-by-block alignment is pursued based on the local pixel-level motion vector MV, followed by block-by-block sub-pixel level estimation of the motion vector MV. This situation induces a new sub-pixel level motion vector field. 2. The reliability of the calculated motion vectors is calculated in a similar manner (using appropriate thresholds for rules #1 to #3) in order to identify reliable motion vectors in the motion vector field achieved in step 1 mentioned above MV. 3. Let multiple reliable motion vectors MV be Q, one of which is assigned to each block, and vice versa, one block is exactly related to one motion vector. For those Q blocks, this equation system can be formed: where X = , and A is a stack of Q 2×6 matrices, each of which has entries . b is a stack of Q 2×1 matrices, and each of these matrices has entries . By computing the pseudoinverse of A to solve for X , we obtain the global affine motion model parameters.

It is worth saying that at this point the motion model is valid for full range motion and is not sufficiently accurate for objects that are not at the same distance from the imaging system. If information from some motion-related auxiliary sensors such as accelerometers/gyros and depth sensors is available for each of the anchor frame and the replacement frame, the motion estimation procedure may be supported by this information, and Motion vector quality can be further improved.

Figure 7 shows a schematic diagram of a method suitable for higher order motion models that support non-translational transformations such as similarity, affine and projective (2D homography) transformations, which transformations are described in Section 1 of the description of the prior art. Reference is made in part to "Multiple View Geometry in Computer Vision" by R. Hartley and A. Zisserman (Cambridge University Press, 2004). It should be noted that the method illustrated in Figure 7 also supports translational motion models, as will be described later.

As depicted in Figure 7, after estimating the pixel-level motion vector field and the reliability map of the motion vector MV, sub-pixel refinement is performed between the reference frame/frame ROI and the anchor frame/frame ROI. Performed on a block basis without any explicit intermediate alignment. The result is a sub-pixel level motion vector MV field which is then transformed into a 2D correspondence list. The correspondence is then used to estimate the global motion model parameters. Once the motion parameters are calculated, frame alignment can be performed.

Steps a) to d) correspond to the exemplary method described above with reference to Figure 2. The previous step e) is modified, denoted as step i), and is designed to make use of the reliability factors derived in step d) and the alternative frames or alternatives derived in steps b1), b2), ... bN) Block-based sub-pixel level refinement BB-ME is performed on the inserted and smoothed low-pass filtered raw CFA data of the ROI of the frame.

This step is followed by step j) designed for two-dimensional correspondence and step k) designed for transformation matrix calculation. The result of step k) is the estimated motion parameters.

As described earlier, the output of step a), that is, the dividing steps a0), a1), a2, ..., aN) are C _r and C _a for each frame or region of interest ROI of the frame. Step b), that is, the output of dividing steps b0), b1), b2), ..., bN) is the output of each relevant C _r and C _a and . The output of step c) is the forward motion vector field , i = 1,2 …, M and j= 1,2 ,…, N ; and backward motion vector field , i = 1,2 …,M and j = 1,2 ,…,N . The output from step d) is an M × N binary motion vector reliability map .

In this step i), block-based sub-pixel refinement is pursued only for reliable motion vectors. Perform non-explicit global intermediate alignment. Therefore, for each block B _r in the reference frame, the matching block B _a in the replacement frame is based on the motion vector MV to identify, . Then, for each block pair, the sub-pixel motion vector is calculated by solving the system of equations in Section (3.7). The result is a sub-pixel level forward motion vector field . Let the motion vector field and Add up to obtain the total sub-pixel level motion vector field ,in .

As mentioned earlier, Ω _R represents the set of reliable motion vectors. Let the number of reliable motion vectors in Ω _R be represented by Q. For reliable motion vectors , Q pair corresponds to It is constructed as follows, where k =1, 2,… , Q : where each source point [ S _x , S _y ] is defined as And the corresponding destination point [ S _x , S _y ] is defined as

Of course the implicit assumption here is that the reference frame/frame ROI matches towards the alternative (current) frame/frame ROI.

2D correspondence set This is then used to estimate the parameters of the underlying global motion model, as described below.

Assume that the constructed number pairs of the reference frame/frame ROI and the alternative frame/frame ROI correspond , the motion model parameters are estimated via direct linear transform (DLT). Direct linear transformations are described in more detail in GH Golub and CF Van Loan, Matrix Computations, 2nd ed. 1989, and R. Hartley and A. Zisserman, "Multiple View Geometry in Computer Vision" (Cambridge University Press, 2004) . Pursue the random sampling consensus (RANdom SAmple Consensus, RANSAC) algorithm, which is described in "Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography," (Communications of the ACM, 24(6):381-395, 1981), where in each RANSAC iteration, the DLT algebraic system is formed from eight (8) randomly selected pairs of correspondences, and the transformation of the captured motion model parameters is decomposed by a single value ( singular value decomposition, SVD) to estimate. This is described in more detail in GH Golub and CF Van Loan, Matrix Computations (2nd ed. 1989).

The 2D transformation of homogeneous image point coordinates from source to destination is defined as follows = T _movement Among them, the 3×3 transformation matrix T _motion captures the motion model parameters, and T _motion = , where x ₉ =1.

In the following, different motion models are described. 1. Pure translational motion model:

For a given pair of correspondences, the translational motion model can be expressed by = T _movement Among them, t _x and t _y represent the translation in the x and y directions respectively. The translation model parameters ( t _x , t _y ) can be estimated from the set of Q corresponding pairs as

An averaging operation is pursued here since only reliable motion vectors are included in the construction of reliable correspondence pairs. The average value in the example is the mean, and specifically the arithmetic mean. However, medians, squared or cubed means, logarithmic means, trimmed or weighted means, and the like are applicable. 2. Similarity transformation motion model: (translation + rotation + scaling)

For a given pair of correspondences, the similarity transformation motion model can be expressed by = T _movement where θ is the rotation angle, and s is the scaling factor for the x/y coordinates. The corresponding DLT system of the equation can be defined as It is written based on multiple pairs of correspondences and motion model parameters as

The unknown motion model parameter vector X _4×1 can be estimated by using SVD to analyze the DLT system: 3. Rigid transformation motion model: (translation + rotation)

The estimation is almost identical to the procedure in the similarity transformation, with the only difference being the additional constraint for transformation scaling invariance: 4. Affine transformation motion model: (transformation + rotation + scaling + shearing)

For a given pair of correspondences, the affine transformation motion model can be expressed by = T _movement Among them, t _x and t _y represent the translation in the x and y directions respectively. And parameters a , b , c and d are combinations of scaling, rotation and shearing. The corresponding DLT system of the equation can be defined as It is written based on multiple pairs of correspondences and motion model parameters as in , the unknown motion model parameter vector X _6×1 can be estimated by using SVD to solve the DLT system: 5. Projection (2D homography) transformation motion model:

For a given pair of correspondences, the projection (2D homography) transformation motion model can be expressed by The DLT system of equations can be defined as It is written based on multiple pairs of correspondences and motion model parameters as , The unknown motion model parameter vector X _9×l can be estimated by using SVD to solve the DLT system, where the last row of V represents X _9×l as shown below. [U,S,V]=SVD (A _2Q×9 ) X _9×l = last row of V

1: Electronic device 2: Camera 3: Image processor unit 4: Image sensor 5: Optical mechanical lens system 6: Mechanical actuator c)~h): Steps a1)~a2): Steps b1)~b2) :Step CFA:Color filter array IMG _RAW :Original image data IMG _FIN :Resulted image F _REF :Frame F _{ALT_1} :Frame F _{ALT_2} :Frame F _{ALT_N} :Frame

In the following, the invention is explained by means of exemplary embodiments by means of the following figures. It shows:

Figure 1 - Block diagram of an electronic device including a camera, image processor unit and mechanical actuator;

Figure 2 - Flowchart of a method for processing image data from an image sensor;

Figure 3 - Schematic diagram of block matching for giant block shifting within the search window of the reference (anchored) frame and the substitute frame;

Figure 4 - Illustrative estimated motion vector field for a frame with pixel-level accuracy;

Figure 5 - An exemplary estimated motion vector field according to the frame of Figure 4 with weighted motion vectors ignoring unreliable motion vectors;

Figure 6 - Illustrative estimated motion vector field from the frame of Figure 5 with weighted motion vectors with sub-pixel level accuracy

Figure 7 - Schematic representation of methods suitable for higher order motion models.

Domestic storage information (please note in order of storage institution, date and number) without Overseas storage information (please note in order of storage country, institution, date, and number) without

c)~h): steps

a1)~a2): steps

b1)~b2): steps

F _REF : picture frame

F _{ALT_1} : Picture frame

F _{ALT_2} : Picture frame

F _{ALT_N} : Picture frame

Claims (22)

A method for processing image data (IMG _RAW ) of an image sensor (4), wherein the image data includes a short pulse frame captured by the image sensor (4), each frame Contains an original pixel matrix; where: A) Determine motion vectors (MV _1...N ), where each forward motion vector in the forward direction represents an area in the anchor frame selected by one of the short pulse frames block to the best matching block in one of the burst frames' respective replacement frames, and each backward motion vector in the backward direction represents one of the burst frame's respective replacement frames The shift from a block to the best matching block in one of the selected anchor frames in the short pulse frame; B) Determine the reliability factors for the motion vectors determined in step A) and Utilizing a difference between the forward motion vector and the associated backward motion vector to assign a reliability to the respective motion vector for a given block, wherein the reliability increases as the difference decreases; C) Alignment The frames of the burst frame of an image, wherein the motion is compensated using weighted motion vectors, wherein the motion vectors are weighted by the respective reliability factors determined in step B). The method of claim 1, wherein if the difference between the forward motion vector and the associated backward motion vector exceeds a predefined threshold, assigning a reliability factor of zero (0) to a A set of forward motion vectors and associated backward motion vectors such that these motion vectors are ignored in further processing. The method of claim 1 or 2, wherein if the difference between the forward motion vector and the associated backward motion vector is lower than a predefined motion consistency threshold (T _MC ), then assigning a (1) a reliability factor to a set of forward motion vectors and associated backward motion vectors such that these motion vectors are taken into account in the further processing. The method of any one of claims 1 to 3, wherein the difference between the forward motion vector and the associated backward motion vector is determined by the minimum absolute deviation Calculated L1 sum of standard absolute differences (SAD), where represents the forward motion vector and represents the backward motion vector, where i = 1,2..., M and j = 1,2,...N, where M and N are the sizes of the M × N frame. The method according to any one of the preceding claims, wherein if the magnitude of at least one of the x component in the x direction or the y component in the y direction of a motion vector is equal to the block matching search size (P) , then assign a reliability factor of zero (0) to the motion vector. A method as claimed in any one of the preceding claims, wherein the difference between the respective motion vector and the motion vectors of adjacent blocks is used to assign a reliability factor to a motion of a given block vector. The method of claim 6, wherein if at least the magnitude of the x component in the x direction of the motion vector is consistent with the magnitude of a set of blocks located in the direct and/or indirect neighborhood of the selected block The difference between the average magnitudes of the x-components in the x-direction of the motion vector exceeds a predefined threshold value ( _TU ), or at least the magnitude of the y-component in the y-direction of the motion vector is the same as that in the y-direction of the motion vector. The difference between the average magnitudes of the y-components in the y-direction of the motion vectors of a set of blocks in the direct and/or indirect neighborhood of the selected block exceeds a predefined threshold value (T _V ), assign the reliability factor of zero (0) to the motion vector of a given block. The method according to any one of the preceding claims, wherein the sum of absolute differences is calculated during the block matching in step A), and the sum of absolute differences corresponding to the motion vector of the best matching block is used (SAD) to assign a reliability factor to the motion vector of a given block. The method of claim 8, wherein if the sum of absolute differences (SAD) of the motion vector corresponding to the best matching block exceeds a predefined threshold, a reliability of zero (0) is assigned factor. A method as claimed in any one of the preceding claims, wherein an overall reliability factor is obtained by combining a set of reliability factors assigned to a motion vector of a given block, for example by Calculated by multiplying the set of reliability factors. The method of claim 8, wherein the motion vectors are weighted by the respective overall reliability factors (R _MV (i, j)) in step C). The method according to any one of the preceding claims, wherein the frames considered for steps a) to c) are captured by the image sensor (4) or can be captured by the image sensor (4) Capture a selected region of interest (ROI) within a larger frame. The method according to any one of the preceding claims, wherein the best matching block is determined by calculating the selected block in a first frame and the searched block in a second frame. The sum of absolute differences (SAD) between the blocks and the best matching block is determined to be the block with the smallest sum of absolute differences (SAD), and the multiple blocks under search have in the second frame different relative positions to each other. A method as claimed in any one of the preceding claims, wherein a global motion vector of a frame is determined by calculating an average motion vector from the set of weighted motion vectors of the frame, the weighted motion vectors being Weighted by at least one of the respective overall reliability factors or the reliability factors of the motion vector decisions for a given block of the frame. A method as described in claim 14, wherein - aligning a substitute frame with a selected anchor frame using the respective global motion vectors of the substitute frame, and - Determine the sub-pixel level forward motion vector for each pair of blocks in the replacement frame and anchor frame respectively, - determine a global sub-pixel level motion vector by calculating an average motion vector from the set of sub-pixel level forward motion vectors determined for the blocks of the frame, and - Align the alternative frame and the anchor frame using the global sub-pixel level motion vector determined for the alternative frame. A method as described in any of the preceding claims, wherein - Determine a respective local sub-pixel level motion vector for each block of a frame, and determine at least one reliability factor for each local sub-pixel level motion vector, and - Aligning the blocks in the substitute frame to the anchor frame on a pixel-by-pixel basis using the local sub-pixel level motion vectors determined for the respective blocks of the substitute frame. The method according to any one of the preceding claims, wherein the sensor signal of the auxiliary sensor is used, preferably the image sensor measured by an accelerometer, a gyroscope and/or an optical depth sensor. The motion of the block is estimated using the acceleration of the detector (4). The method as claimed in any one of the preceding claims, wherein each frame includes a plurality of color channels, wherein those having the highest information density compared to the information density of the other channels of the plurality of channels are selected. the color channel, that is, the highest sampled color channel; the original pixel matrix of the frame for the selected color channel aligned by inserting missing pixels in the selected color channel; and the original pixel matrix for the selected color channel Inserting pixels continues with steps A) to C). An image processor unit (3) for processing raw image data (IMG _RAW ) provided by an image sensor (4). The image sensor (4) includes an image provided by the image sensor (4). 4) A sensor array capturing a short pulse frame, each frame containing a raw pixel matrix of each image, wherein the image processor unit (3) is configured to: A) determine motion Vector (MV _1...N ), where each forward motion vector in the forward direction represents a respective replacement of a block in the anchor frame selected by one of the short pulse frames to one of the short pulse frames The displacement of the best matching block in the frame, and each backward motion vector in the backward direction represents that one of the short pulse frames respectively replaces a block in the frame to the short pulse frame a shift of the best matching block in a selected anchor frame; B) determining reliability factors for the motion vectors determined in step A) to utilize the forward motion vector with the associated backward motion The difference between vectors to assign a reliability to the respective motion vector for a given block, where the reliability increases as the difference decreases; and C) the burst frames aligned to an image Frame, wherein the motion is compensated using weighted motion vectors, wherein the motion vectors are weighted by the respective reliability factors determined in step B). The image processor unit (3) of claim 19, wherein the image processor unit (3) includes or is connected to at least one auxiliary sensor, preferably an accelerometer, a gyroscope and/or an optical depth sensor. sensor, wherein the image processor unit (3) is adapted to estimate the motion of the block using the signal of the at least one auxiliary sensor. The image processor unit (3) of claim 19 or 20, wherein the image processor unit (3) is configured to process image data by performing the step of any one of claims 1 to 17 . A computer program comprising instructions which, when executed by a processing unit, cause the processing unit to perform the steps of the method of one of claims 1 to 18.

Images