FR2651405A1 - METHOD AND DEVICE FOR PROCESSING IMAGES WITH IMPROVED MOTION ESTIMATION. - Google Patents

Abstract

The image is divided into several parts and each image part is examined to determine therein, in the presence of a series of images 20, 21, 22 succeeding one another in time, a motion vector V. When one wishes to create intermediate images 23, 24 between the images initially available, there may be several different vectors all giving equivalent results for certain parts of the image. To resolve the ambiguity, we determine for each equivalent motion vector 16 capable of being used to create an intermediate image 24, the position 17 where the part of the image concerned 25 would be in a previous image 23 if we applied to this image. part of the image said motion vector, and the difference between the motion vector used to arrive at said position 17 and the motion vector which had been determined for this position in the previous image 23 is calculated each time. the chosen movement is then one of those for which this difference is less than a determined threshold. Application: high definition or ordinary television transmission-reception systems, in image recording systems, for example those using a laser reading disc, and in general in digital image processing systems.

Description

"Method and apparatus for image processing with motion estimation

improved "

  The present invention relates to a method of treating

  television image, in which the image is divided into several parts and each image part is examined

  to determine, in the presence of a series of images succeeds

  in time, a vector-movement.

  It also relates to a treatment device

  of images comprising a processor for, in the presence of a

  a succession of images succeeding each other in time,

  motion-factors associated with image parts.

  Finally, it concerns a coding system for televi-

  if we. The present invention can be used in high definition or ordinary television transmission-reception systems, in image recording systems, for example those using a laser-readable disk, and in

  general in digital image processing systems.

  He is known from the conference entitled "Motion compensated interpolation applied to HD." MAC pictures encoding and decoding "by F. Fonsalas and MR Haghiri, presented at the 2nd International Workshop on Signal Processing of HDTV, L'Aquila, 29 Feb. March 3, 1988, to use such a method in a television transmission-reception system, in which complementary data of the video data and in particular the motion vectors are transmitted digitally, called DATV (Digitally Assisted Television, in French). In the system proposed by this document, coding is used to reduce the sampling frequency, with

  the lowest possible associated digital data rate

  corn. For this purpose the image is divided into several parts, to

  16X16 pixels squares, each of which

  treatment can be chosen from several, one of which makes use of vector-motion. For displaying images

  In reception, motion vectors are used to

  construct an intermediate image not transmitted between two transmitted images. In this case, motion vectors have been

  determined at the time of coding, ie when we have

  core at its disposal all the source images and in particular those which will be deleted by the coding, which makes it possible to carry out all the desirable tests to check the validity of the motion vectors, in particular by comparing a reconstructed image on the basis of the vectorsmovement with

the corresponding source image.

  A particular problem arises when it comes to creating intermediate images at a time when the source image is not available for comparison. This is for example

  the case when one wants to introduce each time an image sup-

  between two images to double the display rate.

  to reduce the impression of flicker. When

  an image has a periodic structure, for example when

  that it represents at least locally a grid or

  horizontal or vertical, there may be more than one

  different types give equivalent results for certain

  some parts of the image, that is to say that erroneous vectors can nevertheless give correct results (no visible defect) for certain parts of the image (this will be explained in detail later). But if the chosen vector is erroneous for certain parts of the image, while it is correct for other parts, there are visible defects in the transition between the parts that have a correct vector and those that have

an erroneous vector.

  It is known from the cited document to use a cost function, calculated on the 256 pixels of a portion of an image, which makes it possible to determine which is the most appropriate among several motion vectors, but the estimate thus produced does not does not remove the ambiguity when several

  vectors-movement may be suitable for both

  for the same part of an image, that is to say when

  several motion vectors for which the aforementioned

  cost ratio gives equivalent results. Such vectors

  will be called equivalent vectors.

  The invention is based on the observation that this problem is solved when a criterion is used

  geometric in addition to the use of a cost function.

  The method of the invention is particularly notable in that, the vector-motion determination method

  being able to provide for the same part of an image

  for each equivalent vector-motion, the position o the image part

  concerned would be in an earlier image if we apply

  this image part said motion vector, and the difference between the motion vector used to arrive at said position and the motion vector is calculated each time.

  which had been determined for this position in the previous image

  higher, and finally the chosen motion vector is one of those for which the difference is less than a predetermined threshold.

  The device according to the invention is in particular

  markable in that, said processor being capable of

  provide for the same image part several vector-movements

  Equivalently, the device is further provided with means for calculating, for each equivalent motion vector, the position where the relevant image portion would be in an earlier image if said vector-motion was applied to this image part, means to calculate each time the difference between the motion vector used to arrive at

  said position and the motion vector which had been determined

  born for this position in the previous image, and means for selecting the (or) motion vector for which

  this difference is less than a predetermined threshold.

  The description that follows, with regard to the

  Annexes describing non-limitative examples will do well

  understand how the invention can be realized.

  Figures 1. to 3. represent video signals

  to explain the problem that the invention solves.

  Figure 4 shows schematically a device

  in which the process of the invention is carried out.

  Figure 5. represents groups of parts of

  successive generations in order to illustrate the process

of the invention.

  In this example, it is assumed that each image is divided into image portions that are 16X16 pixel squares, or 256 pixels per image portion. The process starts with determining the motion for each part of the image. This determination is of course based on a

  examined at two different times to measure the displacements

  events that occurred between the two moments. It can be done by any known method, for example a method based on space-time differences, or using Fourier transform techniques, or based on comparisons with application of a cost function. These three methods are described, for example, in a report by the British Broadcasting Corporation, BBC RD 1987/11, entitled "Television Motion Measurement for DATV".

  other applications ", by G. A. Thomas.

  The process that uses a cost function and which is also described in the document mentioned first, will be

now briefly recalled.

  The motion vectors are estimated by selecting from among a set of equivalent vectors the one that minimizes a function of cost expressed, on the 256 pixels of a part of image, by the following expression: 16,16 7Z Io (XrY) - _ (Ii (X-Vx, Y-Vy) + Ili (x + Vx, y + Vy)) x, y = 1.1 O Io and Il are the intensities of a pixel in temporal index images respectively 0 and 1 for the positions indicated in parentheses (according to orthogonal coordinates peculiar to the square considered), Vx and Vy being

  the orthogonal components of the motion vector considered.

  After the estimation of the movement thus realized, it is important to check if the chosen vector-movement allows

  to get the best picture. Indeed, other modes of

  can be considered which could possibly give a better image, and in particular a mode in which an image would be described more quickly but with a definition

  lower spatial For this verification, a decoding

  The inverse of the coding is performed in each of the

  and in each mode the decoded picture portion and the source picture portion are compared, to determine which mode gives the best picture for the part in question (currently all source pictures are available). ). Suppose now that the superiority of the mode

  with motion vector has been observed, for a square determined

  mine. Therefore, there is no question of using another mode.

  Video signals are shown in FIG. 1 at times referenced T1, T2, T3. These signals correspond by

  example in 15 in the image of a clear vertical grid (in terms of

  positive modulation) with on the right in the figure a clear part of greater width whose right limit is not represented. If one applies the method explained above to these signals one notes that for the image of the grid there are two vectors V1 and V2 which are equivalent (they give

  the same value for the cost function and furthermore they give

  the same result when checking the best

  their mode). Either to create intermediate images to reduce

  the visual impression of flickering, from these successive images at times T1, T2, T3. The images recreated at

  from the vector Vl are shown in Figure 2 and this

  The recreated from vector V2 are shown in Figure 3. The time between T1 and T2 is called T12 and the time

  between T2 and T3 is called T23. In Figure 2, the images

  Terminiaries T12 and T23 are correct. In Figure 3, there is

  has areas indicated by hatching at 45 o there is uncertainty

  titude: it is obvious that an automatic process is impot-

  to repair information gaps.

  On the coding device of FIG. 4, images to be coded are introduced at 1. They can then be applied in two different ways: The top channel in the figure is intended for coding with

  vector-motion estimation in which a complete image

  is described in 40 mS: the images pass through a filter 3 and the motion vectors are determined in a module 8 according to the method described above (cost function). The module 8 needs to know the images at two successive epochs, images which are supplied to it simultaneously by the outputs of two delay circuits 13 in cascade, each of which brings a delay of 20 mS. The filtered image with its motion vectors is supplied to a module 9 which calculates the value of the aforesaid cost function, in comparison with the source image 1 which is also supplied thereto through a delay circuit 14 whose delay is equal to that provided by the circuits 3 and 8. The bottom channel in the figure is intended for coding without

  vector-motion estimation and in which a composite image

  plete is described in 20 mS: the images pass through a filter 4 and are supplied to a module 6 which calculates the value of the aforementioned cost function, in comparison with the source image 1 which is supplied to it through a circuit of delay 5

  the delay is equal to that provided by the filter 4.

  The two costs calculated respectively in the

  6 and 9 are provided to an evaluation module.

  the delay brought by elements 3.8 is greater than that

  brought by the elements 4,6, an additional delay is introduced between the modules 6 and 10 so that the information

  modules 6 and 9 arrive together at module 10.

  The module 10 determines which is the best way, ie the best coding mode, by choosing the one of the two channels for which the cost function is the most favorable. Alternatively, it can also choose a vector-motion path if the evaluated cost for this path

  is less than a predetermined value (so without

  keep the calculated cost for the other route). Nevertheless,

  some vector-movements can always be equivalent, vectors between which we do not know how to choose. This is why the module 10 is followed by a module 11 in which is calculated a geometric distortion value and a module 19 in which a criterion based on this value makes it possible to choose the

better vector-movement.

  If the vector-motion mode has been chosen, the motion vectors are supplied by the module 10 to the module 11. In order to be able to compare the vectors of an image with those of a previous image, the module 11 is provided with a loopback circuit with a delay of 40 mS which 'brings back'

  the vectors of the previous image to be compared.

  The comparison process implemented in the

  dule 11 is illustrated in FIG. 5. The three images 20, 21, 22 are transmitted images, and the images 23 and 24 are images to be created by means of motion vectors. In the figure, only a fraction of an image containing nine image parts is represented each time, which fraction will nevertheless be named image to simplify the terminology. Or to check the validity of a vector-motion of the image 24 while the

  process is now complete for images 20,21,23.

  For each motion vector to be tested, we apply to the

  the image concerned a displacement corresponding to a backward turn in time of 40 mS in the direction and with the velocity indicated by said motion vector, which brings said image portion of the image 24 into one of the parts of the image 23. For example, the motion vector symbolized by the arrow 16, which brings the central point of the image 24 to the point 17 in the image 23. This point is located in the square 18,

  for which a motion vector had been determined in the

  mage 23. The distortion between this prior vector and the

  The current tester tested for image 24 is calculated and stored.

  Then the operation is repeated for all equivalent vectors.

  slow for the current image portion of image 24.

  If, for example, motion vectors have

  limited (for reasons of transmission rate) to 16 pixels in each coordinate axis,

  can be outside the four squares of the center on the bottom

  gure (each of them has a dimension of 16X16 pixels). For all motion vectors to be tested for a part

  24, there is a maximum of four image parts

  born in picture 23, so the comparison process is

  simple and requires very little storage capacity.

  The module 19 following the module 11 finally determines the motion vector that will be used. The modules 8 and 9 of FIG. 4 are equipped with processor means to possibly determine several equivalent motion vectors and calculate for each its cost function. This data is supplied to module 19. The process in module 19 is then as follows: it contains a register in which it lists the list of equivalent motion vectors for a part

  image, and this by classifying them in order of merit decreases

  from the point of view of the cost function calculated in module 9. If there is a set of motion vectors for which the comparison values provided by the module 11

  are themselves equivalent, that is to say that the distor-

  are all less than a predetermined value, the motion vector chosen is the one that, among the elements of this set, is closest to the beginning of the list. If, for example, the set contains only one motion vector, it will be chosen, even if it is not at the top of the list. If, on the other hand, no comparison value is suitable, it is because the movement has not been regular between the

  compared two images, and therefore the method is not suitable

  and the motion vector at the top of the list is chosen.

Claims (5)

  An image processing method, wherein the image is divided into several parts and each image portion is examined to determine, in the presence of a series of images succeeding one another in time, a motion vector, characterized in that, the motion vector determination method   being able to provide for the same part of an image   all appropriate vectors-movement called vectors-   equivalent movement, we calculate, for each vector-movement   equivalently, the position where the relevant image part would be in an earlier image if this vector-motion is applied to this image part, and it is calculated at each   difference between the motion vector used to   river to said position and the motion vector that had been   determined for this position in the previous image, and   nally the chosen motion vector is one of those for   said difference is less than a predetermined threshold mine.   2. A method of image processing according to the claim   tion 1, using a method for determining vector-   based on the calculation of a function expressing the validity   vector of motion, characterized in that if there are several motion vectors for which said difference is less than said predetermined threshold, one chooses among them   the one for which the said function has the most favorable value corn.   3. Image processing method according to one of the   claims 1 or 2, using a motion vector determination method based on the calculation of a function expressing the validity of a motion vector, characterized in that if there is no motion vector for which said difference is lower than said threshold predetermined, one chooses among the equivalent vectors the one for which the said function has the most favorable value.   4. An image processing device comprising a processor for, in the presence of a series of images succeeding one another in time, determining motion vectors associated with image parts, characterized in that, said processor   being able to provide for the same part of an image   equivalent motion vectors, the device is provided with   moreover, means for calculating, for each vector-movement   equivalent position, the position where the part of the image concerned would be in an earlier image if applied to that   part of the image, said motion vector, means for calculating   every time the difference between the motion vector used to arrive at said position and the motion vector   which had been determined for this position in the previous image   and means to select the vector (s) -   movement for which (or which) this difference is less than above a predetermined threshold.   An image processing device according to the   cation 4, of which said processor comprises means for calculating   assigning a cost function to decide whether each of said motion vectors is acceptable, characterized in that it is provided with means for, if there are several motion vectors for which the aforesaid difference is lower than the aforesaid predetermined threshold, choose among them the one for which the   cost function has the most favorable value.   6. An image processing device according to one of claims 4 or 5, wherein said processor is provided with means for calculating a cost function to decide whether each of said motion vectors is acceptable, characterized in that   that it is provided with means for, if there is no   motion for which the above difference is lower than the aforesaid predetermined threshold, choose the motion vector for   which cost function has the most favorable value.   7 High definition television coding system   comprising a device according to one of claims 4 to 6.