EP0780795A1 - Bewegungsschätzungsverfahren - Google Patents

Bewegungsschätzungsverfahren Download PDF

Info

Publication number
EP0780795A1
EP0780795A1 EP96402778A EP96402778A EP0780795A1 EP 0780795 A1 EP0780795 A1 EP 0780795A1 EP 96402778 A EP96402778 A EP 96402778A EP 96402778 A EP96402778 A EP 96402778A EP 0780795 A1 EP0780795 A1 EP 0780795A1
Authority
EP
European Patent Office
Prior art keywords
image
vector
field
vectors
frame
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP96402778A
Other languages
English (en)
French (fr)
Inventor
Catherine Heimburger
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Vantiva SA
Original Assignee
Thomson Multimedia SA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Thomson Multimedia SA filed Critical Thomson Multimedia SA
Publication of EP0780795A1 publication Critical patent/EP0780795A1/de
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/223—Analysis of motion using block-matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Definitions

  • the invention relates to a motion estimation method.
  • This method is particularly suitable for image conversion systems comprising a change in frame frequency, in particular for conversion from an interlaced format 50 Hz to a progressive format 75 Hz.
  • the invention applies in any field using sequences images, especially in the television sector.
  • the block matching method which consists in comparing a current block to all the blocks of the same size in a reference image. For each comparison, the distortion between the blocks is determined, and it is generally the comparison giving the lowest distortion which determines the motion vector. It can easily be seen that the number of calculations to be performed increases proportionally with the area of the reference image. To limit the number of computations, it is known to allow block comparisons only within a search window of reduced size. One thus limits the maximum amplitude of the vectors of movement, as well as the number of computations to be carried out. Statistically, the number of large amplitude vectors (generated by rapid movements in the image sequence) is indeed small.
  • Patent application EP-A-0648052 (D93 / 062l) also describes methods and devices for locating motion vectors.
  • the first vector field can be calculated using an exhaustive method of "block matching”("blockmatching” in English terminology). As mentioned above, this method requires many calculations, it is not used for all images, which reduces the memory and computing power requirements accordingly.
  • image here also denotes an interlaced image ("frame” in English terminology) composed of two frames, a progressive image, an interlaced frame (“field”) etc ...
  • the determination of the first vector field is carried out for blocks of a first size, while the determination of the second vector field is carried out for blocks of a second size less than or equal to the first cut.
  • the criterion for choosing a vector from among the candidate vectors is the criterion of the smallest distortion.
  • the evaluation of the distortion associated with a vector and with a given block of this fourth image is carried out by projecting said block along said vector in the second and third images and by evaluating the difference between these two projections.
  • motion estimation is used to improve the interpolation of the output images. These do not necessarily coincide temporally with the input images: to improve the estimation, we take this fact into account by carrying out temporal distortion or error evaluations at the level of the output frames.
  • the candidate vectors for a current block of said fourth image are derived from the vector of the second field of vectors corresponding to the block resulting from the orthogonal projection of the current block on the second image, and / or from the vectors of the blocks adjacent to this projection.
  • the blocks corresponding to the second field are respectively vertical horizontal bands of the blocks corresponding to the first field and the candidate vectors for the second field comprise the vector of said projection whose horizontal component respectively vertical has been modified by a quantification step.
  • said method is implemented in a system for converting image frequencies from 50 Hz to 75 Hz, the first vector field being determined between a first input image and a second image. input, the second vector field being determined between the second input image and a third input image, a first output image being temporally merged with the first input image, a second output image being temporally located between the first and second input image, a third output image being temporally located between the second and third input image, a third vector field being determined between the second and third input images, the candidate vectors for the vectors of this third field being derived from the vectors of the second field, the blocks corresponding to this third field having a size smaller than the blocks corresponding to the second field.
  • a vector field is calculated between the first and the second input image.
  • it is refined twice, an intermediate refinement being carried out to refresh the first field of vectors which, remember, was calculated from the first two input images and not the second and the third.
  • the size of the blocks for the second vector field is intermediate between the size of the blocks of the first field and that of the third field, which will have the density desired for the interpolation.
  • the size of the second blocks, and therefore the density of the second field is chosen so that the mass of calculations to be performed is within the capacities of the device implementing the method.
  • the calculation of the second and third fields must indeed be carried out during a half-period of the process.
  • the estimation of the noise in the image is: where a and b respectively represent the number of blocks per image width and height in the first vector field.
  • the invention will be explained in relation to a method for converting the frequency of television images from 50 Hz interlaced to 75 Hz progressive.
  • a period is defined as the time interval used to display an output frame, ie 1/75 seconds.
  • a cycle is defined as the time interval necessary for a complete cycle of the algorithm, either three periods or 1/25 seconds.
  • the process repeats identical to itself every three output frames or every two input frames.
  • the first output frame is number 1
  • the second frame is number 2, and so on.
  • the frames 1, 4, 7 ... will be generated identically; they will be called type 1 frames.
  • frames 2, 5, 8 ..., respectively frames 3, 6, 9 ... will be generated identically and respectively called frames of type 2 and 3.
  • the input frames will be numbered by reference to the position they occupy with respect to the output frames.
  • an input frame corresponding temporally to an output frame will have the same number as this output frame.
  • An input frame located between two output frames will carry the numbers of these two frames: for example, an input frame located between output frames 2 and 3 will be called frame 2/3.
  • Figure 1 illustrates the respective positions of the input and output frames.
  • the input frame number 1 comprises the first line displayed on the screen of a television; it has been subscripted line O.
  • This frame therefore the odd input frame, is formed by lines 0, 2, 4, ..., 624.
  • a 2/3 frame, even, will include lines 1, 3, 5 , ..., 623.
  • the variable there will represent the line numbers, increasing from the top to the bottom of the screen, in the scanning direction.
  • the variable x will be used to designate the abscissa of a pixel and increases from left to right also in the direction of scanning.
  • t will represent time, normalized so that a time unit represents a cycle.
  • Frame 1 will be located at time 0, frame 2/3 at time 1/2 and frame 4 at time 1.
  • the output frames will have all line numbers, since they are progressive frames.
  • the luminance signal at the position of a given pixel will be represented by the variable P (x, y, t).
  • the described conversion method and the implementation device require only a single frame memory. This constraint is explained by the fact that the method is intended to be implemented in consumer devices manufactured in large series. Reducing the cost of implementation from the hardware point of view is therefore an important factor.
  • FIG. 2 is a block diagram of the device for implementing the invention. Each element will be seen in detail later.
  • the device comprises a random access memory 1 organized in FIFO and intended to increase the frequency of the frames from 50 to 75 Hz, by repeating in reading every other frame.
  • the memory 1 therefore receives as input frames at the frequency of 50 Hz and supplies frames at the frequency of 75 Hz.
  • the device also comprises a noise reduction circuit 2, comprising two filters: a spatial filter intended mainly to reduce impulse-type noise, and a time filter recursive.
  • the spatial filter receives the frames from memory 1, then transmits the filtered frames to the time filter.
  • the time filter also receives a frame called projected frame, composed of information of frames previously processed. The operation of the time filter will be seen in more detail later.
  • a temporally filtered frame is stored in a frame memory 3 and transmitted to an interpolator 4.
  • the interpolator 4 performs the interpolation between two input frames to provide the output frames of the device.
  • the interpolator 4 receives motion vectors from a motion estimation circuit 5.
  • the motion estimation is performed between the frame stored in the frame memory 3 and the "current" frame read in the memory 1. A certain amount of information relating to motion estimation is used when reducing noise.
  • FIG. 3 comprises the chronograms relating to the frames processed by the various elements of the device of FIG. 5.
  • the letters identifying each chronogram correspond to the letters identifying the connections between the elements of this latter figure.
  • a frame is represented in the form of a sawtooth corresponding to the scanning of this frame.
  • the first timing diagram (I) corresponds to the input frames of memory 1, that is to say the interlaced frames at 50 Hz. Frame 1 is odd, frame 2/3 is even etc ...
  • the second timing diagram (A) corresponds to the output of memory 1: the frames are re-read at a speed of 75 Hz. Frame 1 (and frames 4, 7 ...) is read once, the reading starting even before that writing is not completed. Frame 2/3 (and frames 5/6, 8/9 ...) is read twice.
  • the third timing diagram (G) represents the output of the frame memory 3.
  • This memory maintains a time difference of a frame between its input frame and that at its output, so that the interpolation can be done correctly. For this purpose, memory does not behave simply as a delay of a frame. If this were the case, the 2/3 filtered frame would be both present at the entry of the memory and upon its release. It is therefore the filtered frame 1 which is repeated twice at the output of the frame memory.
  • the fourth timing diagram (B) represents the periods of calculation of the information provided by the motion estimator to the rest of the device.
  • the motion estimator behaves differently at each period of a given cycle.
  • coarse motion vectors are calculated (reference "MB") for large blocks (called main blocks), while finer motion vectors are determined for the other two frames (reference "SB" ) for sub-blocks of the main blocks.
  • the calculation of the vectors for the sub-blocks is based on the coarse vectors of the main blocks.
  • the reference "MB1 // 2/3" of the timing diagram indicates, for example, the period of the "rough" motion estimate between frame 1 and frame 2/3.
  • the fifth timing diagram (C) relates to the output of the spatial filter from the noise reduction circuit. This filtering is done directly on the frame read in memory 1.
  • the sixth timing diagram (D) represents the projected frame compared by the noise reduction circuit with the spatially filtered frame.
  • the seventh timing diagram (E) indicates the output of the time filter and therefore the input of the interpolator 4 and of the frame memory 3.
  • the last timing diagram indicates the output of the interpolator and therefore the output of the device itself.
  • the motion estimator works according to the hierarchical block comparison process. This process takes place in two stages. We start by dividing the image into coarse blocks or main blocks of 16 * 32 pixels whose movement is determined, then we divide these main blocks into sub-blocks to refine the vector field.
  • the estimation is made for the luminance signal, which generally contains sufficient information to describe the movement of objects on the screen.
  • the chrominance information it is also possible to use the chrominance information or to combine the chrominance and luminance information for the estimation.
  • a low-pass filter is applied to the frame intended to suppress the high spatial frequencies, sources of spectral superposition.
  • horizontally we use for example a triangular window with seven coefficients:
  • filtering is also carried out. As we only keep the odd lines, this corresponds to vertically interpolated line numbers of the input frame 1 and to the numbers of the input lines of the frame 2/3.
  • a low-pass filtering of the 2/3 input frame is carried out:
  • Psous - sampled (x, y, 0) and Psus-sampled (x, y, 1 2 ) are only defined for x even and odd y.
  • the sub-sampled frames will be denoted P '(x, y, t).
  • the filters used are shown in Figure 7.
  • the image is first cut into main blocks of 32 * 16 pixels. Since the size of an active image is 720 * 576 pixels for the standard sampling frequency, there are thus 23 main blocks per line and 36 per column.
  • the components of the motion vectors are even and between respectively -16 and +16, and -4 and +4. We therefore compare the main block considered with blocks of a search window of a previous frame, this search window being located around the position of the main block related to this previous frame.
  • the vector retained among the set of vectors is noted (u ', v') and corresponds to the minimum of the error function.
  • the vector field of the entire image is calculated by scanning, block by block, and applying this search to each main block.
  • Errmin represents an estimate of the sum of two noises: the noise inherent in the luminance signal, and the quantization noise of the motion vectors. In fact, if the motion vector that has been calculated corresponds well to the movement actually present in the image, Errmin will be the sum of the differences in brightness existing in two identical surfaces the size of a main block: these differences will then be caused only by image noise.
  • a second estimator is therefore defined, namely Errmax-Errmin. It is also a function of the image quantization noise. On a uniform area, the difference between the maximum error and the minimum error comes only from the noise of the image. If the quantization noise is loud, then Errmin and Errmax are both affected; we can hope that this estimator is less sensitive to the quantization noise, eliminated by differentiation.
  • a first correction is made when the image includes uniform areas.
  • the error matrix is approximately constant. Fluctuations are due to the presence of noise in the image.
  • the vector calculated by means of the method according to the invention is therefore unpredictable, and it can result in a strong heterogeneity of the vector field, which is detrimental as a result of the method. In the presence of such heterogeneities, it is decided to force the vector corresponding to zero.
  • the threshold_x and threshold_y values are for example equal to multiples of 2: 2, 4, 8 ...
  • the criterion for determining whether a horizontal component is to be forced to zero is the following:
  • This criterion takes into account the variations of the difference of errors on a whole line of the matrix of errors, and not only the difference with the only vector whose component is possibly to be forced to zero.
  • Detecting an ambiguity on a component of a motion vector means detecting that a priori none of the possible values for this component has a reason to be chosen more than the others. It is therefore necessary to examine the errors linked to each of these values, which amounts to examining the entire line of the error matrix. Uncertainty is determined at near noise.
  • the criterion used is as follows: the average of the differences with respect to the error of the vector whose component is forced to zero is less than a certain noise-dependent threshold. It is in fact a criterion for detecting uniformity of the line of the error matrix.
  • the criterion for a vertical component is similar:
  • a correction is also made when the image contains periodic structures.
  • the shaded block in FIG. 4 represents the block whose movement is to be determined, while the two dotted blocks represent two blocks in the reference image giving rise to a minimum of error in the error matrix. An incorrect vector will generate very noticeable faults when interpolating the output frames.
  • the detection of this type of fault is carried out from the Err error matrix. Indeed, the line containing the minimum of the elements of the matrix will also contain one or more secondary minima as explained above.
  • Usec_min be its index in the Row (u) row.
  • Usec_min located between the umin and usec_min positions. It is characterized as being a local maximum: Row (u-1) ⁇ Row (u) and Row (u + 1) ⁇ Row (u)
  • the histogram in FIG. 5 represents a characteristic example of such a situation.
  • a main block can be astride the edge of the zebra, one half belonging to the animal, the other to the savannah. This block can give a minimum of error, which will however be higher than the minima linked to a periodicity. One thus also avoids considering as periodic a noisy error matrix.
  • An error line verifying all the conditions will come from a matrix associated with a block probably containing a periodic structure.
  • the next step is to correct the vector associated with this main block.
  • the detection of a periodic main block will signal that an entire area of the image containing a periodic structure has been found. We will not immediately retain the motion vector of this main block; on the other hand, the error matrix corresponding to this block will be stored.
  • This matrix is not yet sufficient to determine the correct motion vector. You have to consider the blocks at the beginning and at the end of the periodic structure. Indeed, these blocks generally contain the edge of the periodic zone in motion and therefore make it possible to remove the ambiguity of the information contained in the error matrix.
  • the vector selected will thus be the vector used for all the blocks of the periodic structure.
  • This phase of the process is used twice in succession, during periods 2 and 3.
  • the sub-blocks which, according to the present example have a size of 4 * 4 pixels. This will allow better matching of the vector field to the edges of moving objects.
  • the set of errors associated with the entire range of possible motion vectors is not recalculated as has been done for the main blocks.
  • the candidate vectors for a sub-block are the vectors of the four main blocks closest to this sub-block.
  • the vector generating the lowest error will be assigned to the sub-block.
  • the candidate vectors Once the candidate vectors have been chosen, they must be compared with one another. As indicated above, the sub-block is part of the displayed frame. The vector of the main block in question is therefore scaled. We start by calculating 2/3, and round the result. The movement vector between frame 1 and frame 2 is thus obtained. The remaining vector corresponds to the movement between frame 2 and frame 2/3. If we call these two vectors respectively rear vector and front vector, we have:
  • the vector of the sub-block will be the vector giving a minimum error.
  • the components of the vectors of this field are subjected to a median filtering to eliminate the outliers: indeed, a small error on a sub-block is less visible than a local heterogeneity of the vector field.
  • the median filtering is therefore carried out, for a given sub-block, taking into account the vector previously calculated for this sub-block and the vectors of the horizontally adjacent sub-blocks.
  • the error associated with the chosen vector is also defined for each sub-block. This error is a measure of the confidence placed in each motion vector. The error is written:
  • the vectors of the sub-blocks for frame 3 will be calculated from the vector field previously calculated for the main blocks between frames 1 and 2/3. This poses problems in the event of rapid movements, since the vectors of the sub-blocks will then be calculated from vectors no longer reflecting the real movement.
  • the information concerning the main blocks is refreshed by creating a vector field for so-called intermediate block blocks, the size of which is, precisely, intermediate between that of a main block and that of a sub- block.
  • the intermediate blocks have a size of 32 pixels by 4 lines: they have the width of a main block and the height of a sub-block.
  • the main block which corresponds to it is determined: it is the main block which, projected orthogonally on frame 3, contains this intermediate block.
  • the second and third vectors have the same vertical component as the first vector, the horizontal components being modified.
  • the fifth and sixth vectors have the same vertical component as the fourth vector, the horizontal components being modified. This choice is due to the size of the intermediate blocks, which have only two lines of height. If we had chosen intermediate blocks of larger vertical size, it would have been possible to carry out a correction of the vertical component. But here, with only two lines per intermediate block, it is difficult to reliably determine a vertical component and therefore a fortiori to correct it.
  • the error associated with each of the six vectors is then calculated for each intermediate block.
  • the vector retained for an intermediate block among the six candidate vectors is that giving the minimum error.
  • the correction on the horizontal components is eliminated. This makes it possible to avoid generating too much strong discontinuities in the vector field in the event of noisy images.
  • the last step of the motion estimation of frame 3 consists in determining a sub-block level vector field from the vector field with intermediate blocks.
  • a sub-block will be assigned to the intermediate block of which it is a part.
  • two candidate vectors are determined for each sub-block: the first candidate vector is that of the intermediate block of which the sub-block is a part, while the second vector is that of the adjacent intermediate block as defined below .
  • the first step uses the information contained in a single frame, while the second step uses the information contained in several, in this case two, successive frames.
  • the spatial filtering of the chrominance is not considered to be necessary, the chrominance samples being too far apart from each other for the median filter of directional type to have any meaning. It is obvious that the chrominance can also be filtered, in particular if the chrominance information is more dense, in particular if it had a resolution identical to that of the luminance.
  • the temporal filtering of the chrominance is deduced directly from that of the luminance by considering input frames whose size is divided by the factor of sub-sampling of the chrominance, (either by 2 in the direction of the x-coordinates for a format input 4: 2: 2). We must also divide the x component of the motion vectors by this same factor.
  • Spatial noise reduction is intended to reduce impulse noise in the input frames.
  • the spatial filter is a directional median filter, which has the advantage of not degrading the edges and the fine textures of an image. Simple linear or median filtering would not have this advantage.
  • the directional median filtering is carried out here on three pixels: the "current" pixel, as well as two adjacent pixels located on the same straight line passing through the current pixel. Given these restrictions, four filtering directions are possible: horizontal, vertical and two diagonals.
  • the choice of direction to be considered is made according to the correlation existing between the three pixels of the same direction.
  • the following table represents the current pixel (X22) surrounded by its adjacent pixels: x11 x12 x13 x21 x22 x23 x31 x32 x33
  • the horizontal direction (called d1) passes through the pixels X21, X22 and X23.
  • the vertical direction (called d2) passes through the pixels X12, X22 and X32.
  • the first diagonal direction (d3) passes through the pixels X31, X22 and X13.
  • the second diagonal direction (d4) passes through the pixels X11, X22 and X23.
  • filtering can be carried out on more than three pixels or in more than three directions, if the window used allows it.
  • a significant advantage of motion estimation is that it allows the application of motion-compensated noise reduction methods.
  • a known filtering method consists in performing a pixel-to-pixel recurrent digital filtering. This method comprises memorizing, for each pixel, a certain number of samples, and applying a recurrence relation between these stored values and a current pixel. It is thus possible to determine a filtered value for this current pixel, as well as to update the stored values. The choice of the recurrence relation and the number of terms memorized determine the efficiency of the filter.
  • this effect is attenuated by using the motion vectors already calculated during the implementation of the recurrence relation.
  • the recurrence will be performed not on pixels with fixed coordinates, but on pixels translated by the motion vectors. In other words, we follow the movement of objects to match image pixels of the same point on the moving object.
  • the temporal noise reduction is achieved by means of a single frame memory on the input frames (spatially filtered).
  • the vector field thus obtained will be used to reduce noise in the input frame.
  • Interpolation will be performed between the input frame thus processed and between the stored frame: the two frames used for the interpolation will therefore have undergone noise reduction.
  • Pixel mapping is performed sub-block by sub-block of the input frame.
  • the pixels of a sub-block of the input frame are matched with the pixels of the sub-block designated by the corresponding motion vector in the stored frame.
  • a projection of the stored frame is therefore carried out according to the field of motion vectors.
  • the recurrence relation used is therefore: where f is a non-linear function adapted to noise, resulting from the non-linear function NL represented in FIG. 10 (a), with the diagram of the recurrent filter used (FIG. 10 (b)).
  • This function has a linear part at the start of its curve: for small differences between the image received and its projection, the correction is maximum and the output pixel is linearly interpolated, with a factor 3 ⁇ 4 for the projection and 1 ⁇ 4 for the pixel received. If the difference increases, the probability of erasing an actual transition in the image also increases, and we choose to favor the received pixel, until we no longer take any account of the estimate if the difference is too large.
  • this function is adapted to the variance of the noise estimated at the time of the calculation of the motion vectors: we thus manage to differentiate between the noise to be reduced and the information to be saved.
  • f is the odd function of R in R defined on R + by: where s is the standard deviation of the noise measured by the algorithm and x M the local maximum of the NL function
  • the temporal noise reduction is not carried out when the confidence in a motion vector is too low compared to the noise of the image.
  • a spatial filtering of the stored frame is carried out before the projection according to the field of motion vectors. Indeed, given the sub-sampling carried out before the motion estimation, the precision of the motion vectors, and consequently of the projection, is two pixels. Finer details are eliminated through a linear low pass filter. Measurements have shown that low-pass filtering lowers the noise level by an additional 2 dB.
  • Frame 1 is a special case, in the sense that its even lines are used as is. However, the odd lines must be reconstructed.
  • P 1 (x, y) with (0 ⁇ x ⁇ 719; 0 ⁇ y ⁇ 575) the luminance of a pixel in frame 1.
  • the choice of the type of interpolation is made main block by main block according to criteria related to the vector and to the error associated with the block.
  • the vector associated with a main block of frame 1 is that associated with the main block of the same position in frame 2/3.
  • the motion estimation is not carried out for the blocks of frame 1, but for the blocks of frame 2/3, these vectors are nevertheless taken into consideration. Indeed, if we move away from the edges of moving objects and if the movement is regular, the vectors obtained by direct estimation on the blocks of frame 1 would be similar to the vectors calculated for frame 2/3.
  • the vertical averaging has the disadvantage of showing very visible "stairs" on the diagonal lines. In addition, it gives a mediocre definition, at most equal to that of a conventional television. Finally, it necessarily creates a line flicker since we cannot interpolate frames 2 and 3 in this way. We will therefore try to avoid vertical averaging if possible.
  • This pure spatial filtering is implemented inside the spatio-temporal filter described below.
  • a so-called pure temporal filtering is carried out, in the sense that the missing lines of frame 1 are chosen in another frame.
  • the criterion chosen to determine the static character of a main block is the comparison of the error of the motion estimate with respect to a threshold. If the error is less than this threshold, then the block is considered static. Indeed, if only one part of the block is static, the other part will nevertheless generate a significant error.
  • This type of interpolation allows, compared to the vertical type interpolation by average, to allow a double vertical definition and to suppress line flickering.
  • a spatio-temporal filtering is used for the interpolation. In the context of this example, this is a median type filtering.
  • a median type filter allows you to keep a good definition and to respect the image fronts, unlike purely linear filters.
  • Figure 11 illustrates this type of interpolation.
  • an "X" denotes the pixel to be interpolated
  • a circle denotes a pixel of the frame 1
  • a triangle denotes a pixel of the frame 2/3.
  • F1 averages the values of the pixel (b1) located above the pixel to be interpolated and the pixel (b2) located below the pixel to be interpolated,
  • F2 gives the median value between b1, b2 and the value of the pixel (b) of the frame 2/3 corresponding to the translation by the appropriate motion vector of the pixel to be interpolated from the frame 1,
  • MB (i, j) be the vector of the main block of which the pixel which we seek to interpolate is a part.
  • frame 2 The interpolation of frame 2 is carried out from the vector field at the level of the sub-blocks. All the pixels of the frame must be interpolated, since it is not temporally confused with an input frame, as is the case for frame 1.
  • interpolation mechanisms used for frame 2 have characteristics similar to those already used for frame 1.
  • the interpolation is carried out with the space-time filter already used for frame 1.
  • the frame 2 being located temporally between frame 1 and frame 2/3, the motion vector between frames 1 and 2/3 is separated into two vectors, one (Front vector) indicating the movement between frame 1 and the frame 2, the other indicating the movement between weft 2 and weft 2/3.
  • Each of these two vectors is rounded to an integer value. It will be recalled that an identical decomposition into two vectors of a motion vector was used when calculating the motion vectors associated with the sub-blocks of frame 2. Rounding will be carried out in the same way in both cases, for avoid adding errors.
  • Three preliminary values are then determined (a, b, c). The calculation differs depending on whether the pixel to be interpolated is located in an even or odd line.
  • P 2 (x, y) is the value of the interpolated pixel and corresponds to the median value among three values: the results of two linear filters and a median filter from the values of pixels a, b and c.
  • the two linear filters perform a temporal interpolation with weighting coefficients corresponding to the relative positions of the pixel to be interpolated with respect to the pixels a, b, c. Note that these are two pure time filters, i.e. using only one single pixel in each of the input frames. This avoids a loss of vertical resolution.
  • the median filter also allows the generated frame to maintain good spatial resolution. But it also makes it possible to keep a certain homogeneity between the different interpolation mechanisms of the three frames.
  • the interpolation will be, as in the case of frame 1, of pure temporal interpolation type: we interleave the lines of frames 1 and 2/3:
  • Fallback 2 (x, y) corresponds to a linear filtering severely cutting the high spatial frequencies on the two frames concerned:
  • the value P 2 (x, y) is equal to the value of the pixel compensated in motion but framed so as not to be able to deviate from the value of the fallback pixel by a value greater than the confidence divided by a corrective factor .
  • Frame 3 is interpolated from frames 2/3 and 4 (ie frame 1 of the following cycle) and the vectors of the sub-blocks SB 3 ( i, j ). Only the interpolation coefficients and the parity aspects of the lines will change.
  • FIG. 14 illustrates this filtering.
  • the interpolation of chrominance is subject to much less severe constraints than that of luminance; indeed, the definition of the eye is much weaker.
  • the standard used for transmission on the radio network known as 4: 1: 1 format, includes a sub-sampling resulting in transmitting chrominance information for four luminance information, with a repetition of the chrominance information identical to itself on two lines. The tolerances on the filters used are therefore much greater. Therefore, we will be content with simplified filters.
  • the chrominance interpolation algorithms are therefore directly extrapolated from those of luminance interpolation, by considering frames containing half as many pixels horizontally, and therefore dividing the horizontal component of the motion vectors by two.
  • the filter with 7 input pixels used in luminance is limited to 3 input pixels:
  • film mode is that of video sequences which are generated, not in 625/50/2 format, but in 625/25/1. These sequences are therefore in a progressive format at the input, which means that each input image contains 2 frames corresponding to the same instant.
  • the main motion estimation (the phase of the main blocks) will have to be done between a 1 even frame and an odd 2/3 frame and no longer between a 1 odd frame and a 2/3 even frame.
  • the timing diagrams given in Figure 15 illustrate this particular mode of operation.
  • the film mode can be processed without modification of the filters and motion estimation mechanisms: it will suffice to check, when the presence of the film mode has been detected, the phase of the processing, the interpolation coefficients of the filter of the frames 2 as well as the vectors and trusts of frames 3.
  • the reduction of saccade thus obtained will therefore be inexpensive to implement.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Television Systems (AREA)
  • Image Analysis (AREA)
  • Complex Calculations (AREA)
  • Color Television Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
EP96402778A 1995-12-22 1996-12-18 Bewegungsschätzungsverfahren Ceased EP0780795A1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FR9515403A FR2742902B1 (fr) 1995-12-22 1995-12-22 Procede d'estimation de mouvement
FR9515403 1995-12-22

Publications (1)

Publication Number Publication Date
EP0780795A1 true EP0780795A1 (de) 1997-06-25

Family

ID=9485890

Family Applications (1)

Application Number Title Priority Date Filing Date
EP96402778A Ceased EP0780795A1 (de) 1995-12-22 1996-12-18 Bewegungsschätzungsverfahren

Country Status (4)

Country Link
EP (1) EP0780795A1 (de)
JP (1) JPH09233435A (de)
CN (1) CN1168062A (de)
FR (1) FR2742902B1 (de)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0936575A1 (de) * 1998-02-17 1999-08-18 Koninklijke Philips Electronics N.V. Bildbearbeitungsverfahren zur Bewegungsschätzung in einer Bildsequenz, Verfahren zum Filtern von Rauschen, und medizinische Abbildungsvorrichtung diese Verfahren benutzend

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4553332B2 (ja) * 2000-04-12 2010-09-29 キヤノン株式会社 領域分割方法及び装置並びに記憶媒体
US8265158B2 (en) * 2007-12-20 2012-09-11 Qualcomm Incorporated Motion estimation with an adaptive search range
CN101953167B (zh) 2007-12-20 2013-03-27 高通股份有限公司 减少光晕的图像插值
JP4374048B2 (ja) * 2007-12-26 2009-12-02 株式会社東芝 補間フレーム生成装置、補間フレーム生成方法及び放送受信装置
US8537283B2 (en) 2010-04-15 2013-09-17 Qualcomm Incorporated High definition frame rate conversion
KR101394468B1 (ko) * 2013-11-12 2014-05-14 주식회사 아나패스 블록의 불연속도 연산방법 및 이를 이용한 폴백 수행 방법
TWI700922B (zh) 2018-04-02 2020-08-01 聯發科技股份有限公司 用於視訊編解碼系統中的子塊運動補償的視訊處理方法和裝置

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0648052A1 (de) 1993-09-08 1995-04-12 THOMSON multimedia Verfahren und Vorrichtung zur Bewegungsauswertung mit Blockübereinstimmung

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0648052A1 (de) 1993-09-08 1995-04-12 THOMSON multimedia Verfahren und Vorrichtung zur Bewegungsauswertung mit Blockübereinstimmung

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
SOHAIL ZAFAR ET AL.: "Predictive Block-matching motion estimation schemes for Video compression (Digest)", PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS, JUNE 5-7 ,1991 ,ROSEMONT (US); IEEE PRESS ,NEW YORK (US), 1991, pages 300 - 301, XP000289044 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0936575A1 (de) * 1998-02-17 1999-08-18 Koninklijke Philips Electronics N.V. Bildbearbeitungsverfahren zur Bewegungsschätzung in einer Bildsequenz, Verfahren zum Filtern von Rauschen, und medizinische Abbildungsvorrichtung diese Verfahren benutzend
US6154519A (en) * 1998-02-17 2000-11-28 U.S. Philips Corporation Image processing method for motion estimation in a sequence of images, noise filtering method and medical imaging apparatus utilizing such methods

Also Published As

Publication number Publication date
FR2742902B1 (fr) 1998-01-23
CN1168062A (zh) 1997-12-17
FR2742902A1 (fr) 1997-06-27
JPH09233435A (ja) 1997-09-05

Similar Documents

Publication Publication Date Title
EP0781041B1 (de) Verfahren zur Interpolation von Vollbilden
EP0780794B1 (de) Verfahren zur Korrektur von Bewegungsschätzung in Bildern mit periodischen Strukturen
EP0390660B1 (de) Verfahren und Einrichtung zur temporalen Interpolation von Bildern mit korrigierter Bewegungskompensation
EP1908297B1 (de) Vorrichtung zur erzeugung eines interpolierten bildes
FR2766946A1 (fr) Procede et dispositif de pretraitement pour estimation de mouvement
WO2010007285A1 (fr) Procede et dispositif de remplissage des zones d'occultation d'une carte de profondeur ou de disparites estimee a partir d'au moins deux images
FR2623040A1 (fr) Procede et dispositif de traitement de signaux d'image a balayage de trame entrelace
EP3114831B1 (de) Optimierte videorauschunterdrückung für heterogenes multisensorsystem
EP0677228A1 (de) Verfahren zum entschalteln von halbbildern in einer sich bewegenden bildfolge
FR2958824A1 (fr) Procede de traitement d'images stereoscopiques et dispositif correspondant
FR2594281A1 (fr) Appareil d'estimation du bruit dans des signaux ayant des intervalles redondants
KR101125977B1 (ko) 화상 처리 장치 및 방법, 기록 매체
FR2582891A1 (fr) Processeur a balayage progressif.
FR2599204A1 (fr) Processeur de signaux video a balayage progressif
Bareja et al. An effective iterative back projection based single image super resolution approach
EP0780795A1 (de) Bewegungsschätzungsverfahren
EP0337565B1 (de) Einrichtung zur Kodierung von Signalen, die eine Folge von Bildern darstellen und Übertragungssystem von Fernsehbildern mit hoher Auflösung mit einer solchen Einrichtung
EP0365090B1 (de) Einrichtung zum Verdoppeln der Rate von Fernsehbild-Signalen, und Fernsehbild-Dekodierer mit einer solchen Einrichtung
EP3594895A1 (de) Räumlich-zeitlicher rauschfilter für videodaten, der auf konfidenzindexen basiert
FR2968108A1 (fr) Procede de reduction de la taille d’une image stereoscopique
Sapkal et al. Image inpainting by Kriging interpolation technique for mask removal
FR2736741A1 (fr) Procede de traitement d'une sequence source d'images numeriques bruitees
Keller et al. Temporal super resolution using variational methods
FR2536941A1 (fr) Procede pour le filtrage en peigne, d'une image a l'autre, d'un signal composite de television et filtre en peigne a y utiliser
EP2761876A1 (de) Verfahren und vorrichtung zum filtern einer disparitätskarte

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): DE FR GB IT

17P Request for examination filed

Effective date: 19971208

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: THOMSON MULTIMEDIA

17Q First examination report despatched

Effective date: 20001102

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 20020927