Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
  • G06T7/269—Analysis of motion using gradient-based methods

Definitions

• the present invention relates to a method and a device for simultaneous motion estimation and segmentation of digitized images, in particular moving pictures.
• BM Block Matching
• a block moves with translational motion in the image plane.
• Block Matching technique There are several problems with the Block Matching technique and its variants: i) They give an erroneous estimate if a block covers several moving objects or uncovered background. ii) They are sensitive to noise in the input frames. iii) The accuracy of the estimate is generally poor, especially if no sub-pixel matching is performed. If subpixel matching is performed the technique is computationally expensive and real-time implementation is difficult to obtain. iv) The estimate is poor for blocks moving with non-translatory motion. v) The estimates are sensitive to illumination changes.
• Another possible technique is to perform motion estimation based on an image which has been segmented into regions.
• the image can for example be segmented into regions of slowly varying intensity.
• This method performs simultaneous motion estimation and segmentation by iteratively maximising the support defined by a sum of errors weighted by a robust kernel between two regions: one in a reference frame and the other in the consecutive frame.
• the position and size of the reference block are arbitrary, whereas the position of the consecutive region is determined by a geometric transformation of the reference block.
• This prior art technique is also illustrated by the flow diagram in fig. 8, which shows how input frames are low-pass filtered in a block 801. Then a coarse resolution is set in a block 803, and motion parameters are initialised in a block 805. Thereafter derivatives are calculated in a block 807, which are used for updating the estimate in a block 809. Based on the estimate a new scale is computed in a block 811.
• the technique described in the above cited prior art papers solved some of the problems associated with the block matching and correlation-based techniques.
• the technique can cope with non-translatory motions, e.g. affine models, and with multiple moving objects within a block.
• the technique still exhibits several deficiencies, such as failing to converge for regions with complicated motions and still being computationally expensive.
• the invention aims at solving the problem with the convergence of the prior art technique as described in the above cited papers by M. Bober and J. Kittler, and to increase the computational efficiency thereof.
• a modified gradient-based search technique is used and the centre of the coordinate system is placed in the centre of the region, image or block, where the estimation is applied.
• the use of such modified gradients, i.e. gradients scaled by different factors reduce the number of iterations needed, and thus also the complexity of the technique. It also improves the convergence, i.e. the modified search is more likely to converge to the true motion parameters.
• the median absolute deviation is used in the prior art technique as scale estimate. This has turned out to cause problems with convergence and accuracy. When there is only one moving object within a block, during iterations on finer resolutions, the values of scale become very small and many pixels, which are not outliers, are removed from the estimation process. To overcome this deficiency, a small constant is added to the MAD (Median Absolute Deviation) estimate of scale.
• MAD Median Absolute Deviation
• the scale of residual errors has to be recomputed after each update of motion parameters. This has been found to be very inefficient, and the efficiency of the method has been found to be significantly improved by introducing the following modification.
• the scale is updated after each step.
• finer resolutions i.e. subpixel resolution in case of a translational motion model
• the scale is only updated every k steps, k being an integer greater than one.
• the optimisation on the discrete grid does not specify how to select a good starting point for the iterations after a change of resolution in the parameter space.
• the description only defines the condition for change of the resolution in the parameter space when the path retraces. It has now been found that if during iterations, a record of the point in the parameter space corresponding to the lowest value of the error function is saved and used as the starting point at finer resolution, a significant improvement of the computational efficiency is obtained.
• a hybrid scheme where the initial estimate at coarse resolution is determined by a non-iterative technique, such as the phase-based technique, can be used. That is, first at the low resolution stage a fast non-iterative technique is used. Then, at the medium or fine resolution stages the modified Hough Transform using Robust Statistical Kernels is used.
• the prior art technique is only applied to a block, i.e. regions of rectangular size, and the centre of the coordinate system is placed in the corner of the image.
• the framework is not limited to block shaped regions, but can be extended to regions of any shape. In the latter case, the placement of the centre of the coordinate system however becomes crucial for the performance of the technique. It has been found that by using the centre of gravity of the reference region as the origin of the coordinate system, a good performance is obtained.
• the accuracy of the prior art technique has been found to be improved by prefiltering the initial image with a low-pass filter, preferably Gaussian shaped, and then storing the resulting image in an array with increased dynamic range. All successive processing is performed on the images stored with increased dynamic range.
• a low-pass filter preferably Gaussian shaped
• the technique can employ any parametric motion model. It is particularly suited to be used in:
• a ( a ⁇ -.a 6 ) are the motion parameters.
• - Fig. 1 is a general block diagram.
• FIG. 1 shows the steps involved in the preprocessing block of fig. 1.
• FIG. 3 shows the Robust Hough Transform Block of Fig. 1 in more detail.
• - Fig. 4a and 4b show different segmentations of an image and the position of the coordinate system.
• Fig. 5a - 5d show the operation of a parameter space memory, and the decision mechanism.
• - Fig. 6 shows an example how the resolutions in the image and parameter space are changed.
• - Fig. 7 is an example of the values of k's for various combinations of resolutions.
• - Fig. 8 is a general flow diagram of the prior art technique.
• Figure 1 depicts a general block diagram of the technique. It consists of three main processing blocks to which consecutive images are fed:
• RHTB Robust Hough Transform Block
• FIG. 2 illustrates the steps carried out in the preprocessing block 101.
• LPM low-pass filtering module
• the LPM 203 performs low-pass filtering (smoothing) of the images.
• the images are convolved with two separable 3 tap filters having the coefficients ⁇ 1/4, 1/2, 1/4 ⁇ in the x and y directions.
• the number of times the image passes the filter depends on the image resolution, for example 2 passes for QCIF and three passes for a CIF resolution can be used. If the input image is integer valued and represented by 8 bpp, the output image from the low-pass filter is float valued, due to the fact that the filter coefficients are float valued.
• the input image grey-levels are stored as 8 bits as in the case shown by Fig. 2, i.e. dynamic range 0 - 255, then the low-pass filtered image is stored as a 12 bits image, i.e. an image having 12 bpp, in a memory 207, whereby the information of the float values of the float valued output image can be used in the further processing.
• a Gaussian pyramid formed by images subsampled to different spatial resolutions (Srn, .., Sr2, Sri) is constructed, where Sri is the original spatial resolution.
• This is performed by recursive filtering in the LPM module 203 and subsa pling of each of the images in the subsampling module (SSM) 205.
• the results of the subsampling (se, sf) i.e. the subsampled versions at different spatial resolutions (Sr2,..., Srn) of the original images are stored in corresponding memories.
• the low-pass filtered image (sc) is subsampled in the SSM module 205.
• the result (se) is stored in a memory 209.
• the subsampled image (sd) is also fed back to the LPM module 203 and low-pass filtered again, subsampled and stored in a memory 211. The process can then continue in this manner until a desired number of resolutions is obtained.
• a Gaussian pyramid is constructed by recursively subsampling the low-pass filtered image by a factor of 1, both horizontally and vertically, i.e. the value of a certain pixel within a 1 x 1 block is selected to represent the entire l x l block.
• the process of filtering and subsampling is performed (n - 1) times, where n is the number of resolutions in the pyramid. Any pixel from the l x l block can be chosen to represent the subimage at different degrees of resolution. However, the choice should be consistent, e.g. the upper left pixel can be chosen.
• CIF Common Interchange Format
• the Robust Hough Transform Block 103 comprises a Hough Transform Module (HTM) 301, a Decision Module (DM) 303, a Control Module (CM) 305 and a Scale Estimation Module (SEM) 311.
• HTM Hough Transform Module
• DM Decision Module
• CM Control Module
• SEM Scale Estimation Module
• the control module 305 has an overall control of the estimation process and performs the following tasks:
• the initial spatial resolution is set by means of the switch swl controlled by the CM module 305 via a signal on the line (sw) .
• the images corresponding to that particular spatial resolution (Sr) are fed to the HTM module 301 on the line (sg) .
• the initialisation value for motion parameters is sent to the DM 303 on a line (sp) .
• the CM module 305 counts the number of iterations, each iteration is signalled via line (so) from the DM module 303, and compares the count with a threshold, which is depending on both the current spatial resolution (Sr) and the current parameter resolution (Pr) .
• the decision at which spatial resolution the estimation should start depends on whether any a priori information on motion of the region is available.
• the estimation usually starts at a coarse or medium spatial resolution, i.e. at resolution Srn, where n>l, unless a motion parameter initialisation value (sr) is available.
• the initialisation value sr may be based on the estimate from a neighbouring block or region, or other motion estimator may be used to obtain a coarse estimate of motion.
• the estimation may start at the coarsest (Srn) , medium (Sr (n-l) , ... , Sr2) or finest spatial resolution (Sri).
• the DM module 303 performs two tasks:
• the HTM module 301 computes the value of support (sk) as described in M. Bober and J. Kittler, "Robust Motion Analysis", in the CVPR Conference proceedings, 1994, pages 947-952, and the derivatives (si) of the support surface (Hough surface) for a given value of motion parameters (sm) , which are provided by the Decision Module (DM) 303. Values of support (sk) and derivatives (si) , which preferably are scaled according to the paragraphs below, are passed back to the Decision Module DM 303 for analysis. The DM module updates the estimate of value of the motion parameters (sm) and detects the termination condition which is signalled on the line (so) to the CM module 305.
• the motion analysis usually starts on a coarse resolution in the parameter space (Pr) .
• the motion parameters (sm) are initially set to either no-motion case or to some initialisation value (sp) .
• the DM module 303 decides on the update of the motion parameters, i.e. the next position in the parameter space as illustrated by Fig. 5c. This decision is based on the value of the partial derivatives dH ⁇ on line (si) calculated by the HTM module 301 as:
• each partial derivative dH ⁇ is scaled by means of factors, which are dependent on the spatial extension of a current region, and in a preferred embodiment factors, t ⁇ , t 2 , t 3 , t 4 and t 5 , computed as follows are used in the different models as described below.
• the scaled derivatives dHN ⁇ are then computed as follows:
• each partial derivative dH ⁇ is scaled by a factor corresponding to the current resolution in the parameter space, in particular a multiplication with the current resolution r ⁇ in the parameter space for the corresponding parameter a ⁇ .
• vd (dHN lf ..., dHN-j) in the parameter space.
• a set of basis vectors ⁇ b , ..., b s ⁇ , where s 3N - 1, is constructed, each vector being associated with one possible direction.
• Each vector b 1 is then normalised so that the norm equals 1.
• Fig. 5d shows all vectors b 1 for the translational motion model.
• the HTM module 301 also provides an Outlier Map 307 and a Transformed Frame Difference Map (TFD) 309.
• the values of the TFD Map 309 (si) are passed to the SEM module 311, which calculates a scale of residual errors (su) , if requested by the CM module 305.
• the scale of the residual errors does not have to be recomputed after each iteration step.
• the value of the parameter k depends on the resolution in the parameter space and suitable values for k are shown on Fig. 7, and will be further discussed below.
• a hybrid scheme where the initial estimate at coarse resolution is determined by a non-iterative technique, such as the phase-based technique, can be used. That is, first at the low, coarse resolution stage a fast non-iterative technique is used. Then, at the medium or finest resolution stages the modified Hough Transform using Robust Statistical Kernels is used.
• the Segmentation Map i.e. the Outlier Map 307 is a binary image with a symbol, indicating if the pixel is considered to be an inlier or outlier, attached to each pixel.
• the TFD map is a float valued image with the transformed frame difference for each pixel calculated for the current value of motion parameters (sm) .
• a Parameter Space Memory which is a part of the DM module, records all positions visited in the parameter space and the corresponding support. Each new position in the parameter space is checked against this list, shown in Fig. 5b. If this position has already been visited, i.e. the positions for which calculations have been made, see positions S3, s7 in Fig. 5a, this fact is detected by the decision module, and a change of the resolution in parameter space follows, i.e. to a finer resolution.
• CM control module 305
• the values of the thresholds can be selected as: Iteration threshold 200 iterations Scale threshold 0.35 grey-level
• motion parameters from the coarser spatial resolution are transformed to the finer resolution and used as initialisation parameters (sp) .
• the optimisation search for the maximum in the parameter space is performed iteratively, and during the optimisation process motion parameters can only take discrete values.
• Such an optimisation process is herein referred to as an optimisation on a discrete grid.
• the coarseness of the discrete grid i.e. the resolution in the parameter space, can be different for different motion parameters.
• the maximum displacement of an object, object size and the resolution in the parameter space should preferably be related as explained in the following paragraph.
• d [pixels] is the displacement, i.e. the maximum translation for any pixel from the region, or maximum expected pixel displacement, for example most coding techniques limit such displacement to 16 pixels for CIF images
• M [pixels] is the average pixel distance from the centre of the coordinate system calculated for all pixels within a region.
• Prl 1
• Pr2 1
• Pr3 0.05
• Pr4 0.01.
• the estimation starts at coarse resolutions both in the image space (Sr2) and in the parameter Space (Prl) , shown at position 1 on the graph.
• the resolution in the parameter space is increased to 0.25 (Pr2) , shown at position 2. Since this is the finest resolution in the parameter space for this spatial resolution (Sr2) , the estimation at this spatial resolution is completed. This fact is signalled on line (so) to the Control Module CM 305 which decides whether to continue the estimation on a finer resolution in the image space.
• the estimation process continues at spatial resolution Sri, positions 3, 4 and 5, after the CM module 305 has signalled to the switch swl via the line (sw) that the spatial resolution Sri now is to be fed to the HTM module 301.
• the SEM module 311 recomputes the scale of residual errors at each request on the request line (st) from the Decision Module 303.
• the scale can be updated after every iteration or each k iterations, where k can vary depending on the spatial and parameter resolution.
• the value of the constant C is preferably 0.3, which by means of experiments has been found to be a suitable value.
• the maximum value of the scale estimate is usually around 10, but can theoretically be more, when the estimation starts and has a lower limit at C, which in this example is equal to 0.3.
• Figure 7 illustrates an example of the values of k's for various combinations of the resolutions in the image space Sr and in the parameter space Pr .
• the estimate of scale is returned to the HTM module, through the signal (su) .
• the estimator in the HTM module 301 outputs the motion parameters (sh) and the Outlier map (sj).
• the outlier map is analyzed for large spatially coherent region (s) of outliers. The presence of such regions signals multiple motions. If motion of such remaining regions is of interest, the estimation process can be restarted solely for these remaining region (s) .
• the analysis for large spatially coherent region (s) is performed by means of majority filtering, which works in the following manner:
• a window size is chosen, in the preferred case a window size of 3 x 3 or 5 x 5 is chosen.
• this window is centred around each pixel in the image and the number of outliers within the window is counted.
• the considered pixel is determined to be an outlier and is labelled accordingly.
• the inlier region is usually the largest one and the estimated motion parameters correspond to its motion.
• the framework as described herein is not limited to block shaped regions, but can be extended to regions of any shape.
• the placement of the centre of the coordinate system has turned out to be crucial for the performance of the technique. It has been found that by using the centre of gravity of the reference region as the origin of the coordinate system, good performance is obtained.
• the optimisation procedure becomes very fast. This is due to the fact that an exhaustive search, which is used for instance by Block Matching techniques, is avoided.
• the proposed modifications improve speed and convergence of the technique.
• the accuracy of the final estimate is also improved.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Image Analysis (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (2)

Family

ID=20403433

Family Applications (1)

Country Status (5)

Families Citing this family (35)

Family Cites Families (5)

• 1996
  • 1996-07-19 SE SE9602820A patent/SE510310C2/sv not_active IP Right Cessation
• 1997
  • 1997-07-18 AU AU37141/97A patent/AU3714197A/en not_active Abandoned
  • 1997-07-18 EP EP97933970A patent/EP0978098A2/en not_active Withdrawn
  • 1997-07-18 WO PCT/SE1997/001293 patent/WO1998003939A2/en not_active Application Discontinuation
• 1999
  • 1999-01-08 US US09/227,346 patent/US6356647B1/en not_active Expired - Fee Related

Non-Patent Citations (5)

Also Published As

Similar Documents

Legal Events