MXPA97004867A - Process to interpolate frameworks for compatibility of pelic mode - Google Patents

Abstract

The invention relates to a process for interpolating frames of digital video data received in the form of frames at the frame rate in the movie mode or in the normal mode, including a frame storage, estimating production motion vectors of motion and an interpolation between the stored frame and a real frame, characterized in that the movie mode storage is performed at half the frequency of that in the camera mode, such that only the second of the two frames of input corresponding to the same image of the film, is stored in the frame memory to be able to read again twice in succession, in order to be compared with the next two input frames to calculate the motion vectors. Application to the frequency conversion of marc

Description

PROCESS TO INTERPOLATE FRAMES FOR COMPATIBILITY OF MOVIE MODE DESCRIPTION OF THE INVENTION The invention relates to a process for interpolating frames with frame frequency conversion from video signals in the camera mode or in the film mode and to allow a reduction in vibration in relation to the use of the film mode with interlaced exploration. The frequency of the conversion using motion compensation, for example the transfer of 50 Hz interlaced to 100 Hz interlaced, actually gives rise to a particular problem when sequences are used in the movie mode. The exploration in the movie mode is a 25 Hz scan in progressive mode (the transfer of 24 images per second to 25 Hz being made by artificially doubling an image every 24 images). In order to be transformed into the normal 50 Hz interlaced television format, the film image is divided into two intermediate frames, which correspond to the same image - that is, to the same snapshot - but which are displayed in two different instants Vision at 50 Hz interlaced, therefore gives rise to a vibrational effect on moving objects on stage, - since their movement is no longer faithfully reproduced, but undergoes a kind of temporary separation. This vibration effect persists after the frequency conversion of the frame - at 75 Hz or 100 Hz, for example - if a conversion algorithm is used, which uses just a single frame memory, even if the interpolation is compensated motion . Figure 1 describes the alterations in a pixel belonging to an object that has a vertical uniform rectilinear motion, in the case of a frequency conversion from 50 Hz to 100 Hz, when the image frequency input is in the movie mode . The axis of the abscissa is graduated in image numbers (n-1, n, n + 1, ...), each number corresponding to an instant t divided by the image of period t of image. The axis of the ordinate corresponds to the vertical and pixel change in the image divided by the product of the uniform velocity of change of the pixel and the frame T of frame period. The two frames of the same image input to the frequency converter are marked with 1 and 3, the intermediate frames reconstructed by the converter are marked with 2 and 4, this corresponds to four cycles in the output for a given image. The input pixels represented by a cross originate from frames in the movie mode and therefore, are placed in the same place for two successive frames 1 and 3 of an image n, corresponding to cycles 1 and 3. The conversion of frequency generates two new frames 2 and 4 interlaced between these frames for the same image n. The estimation of movement between the two successive input frames 1 and 3 of an image, and then 3 of an image and 1 of the following image could give the pixels represented by a circle in Figure 1: - between frames 1 and 3 , the calculated motion vector is zero and the intermediate frame pixel has the same position as those corresponding to nearby frames; - between the frames 3 and 1, the calculated motion vector V makes it possible to place the intermediate frame pixel at the distance V / 2. The effect of vibration, that is to say the uneven aspect in the movement, gives rise here to the appearance of "staircase" in the figure and generated by the input pixels replicated in the output and by the interpolated pixels with respect to the reconstructed frames.

A known process for reducing vibration is described with the help of figure 2. The calculation of the movement is carried out between frame 1 and 3 of an image (actually the motion vector is zero) and between the frame 3 of an image and the frame 1 of the next image. The motion vector V calculated between the input vector 3 of the image n and the frame 1 of the following image n + 1, is used here to place the pixels through interpolation in the output frames 3 and 4 of the real image n (here, the pixel of the frame 3 is not replicated in the output): - The output pixel of the frame 3 is changed by a V / 2 value with respect to the input pixel of the same frame. - The output pixel of the frame 4 is changed by a value 3V / 4 with respect to the input pixel of the frame 3. The movement vector V calculated between the input frame 3 of the previous image and the frame 1 of the real image n is used to place the pixels in the output frames of the real image: - The output pixel of frame 2 changes the input pixel of frame 1 to a value of V / 4. - The output pixel of frame 1 is changed by a value of zero with respect to the input pixel of this same frame (replicated pixel).

An extrapolation of the calculated movement vector of a frame of the previous image and a frame of the real image and two interpolations of the calculated movement vector of a frame of the real image and a frame of the next image, therefore, are made here. The pixels in the output are, as shown in the figures, aligned, causing this vibration effect to disappear. However, this process is very expensive. The estimation of the motion vectors, which produces it, requires a large number of delay lines, especially if the restriction in the implementation of the hardware (program) is imposed, so that the movement is estimated and the output pixels of the Output frames 3 are interpolated in the interval that separates the two output frames. Using a vector projection (extrapolation), assuming, therefore, that the estimated movement between the two frames continues to the next frame, provides a damaged image quality, particularly when the movements are not uniform, except for the vibration phenomenon. Processes based on motion interpolation calculated between, for example, two non-successive frames, may also exist, but then they will require more than one frame memory and therefore will produce a high construction expense. The purpose of the invention is to overcome the aforementioned disadvantages. The present invention relates to a process for interpolating frames of digital video data received in the form of frames at a frame rate in a movie mode or in a normal mode, including frame storage, motion estimation and interpolation. between the stored frame and a real frame, characterized in that the storage of the movie mode is performed at half the frequency of that in the camera mode, such that only the second of the two input frames corresponding to the The same movie image is stored in the frame memory, in order to be read again twice in succession to be compared with the next two input frames to calculate the motion vectors. The invention will be better understood with the help of the following figures, which represent: Figure 1, the alterations in a pixel belonging to an object in a vertical uniform rectilinear movement, this pixel originating from an image in the film mode and exhibiting the phenomenon of vibration; - Figure 2, the alterations in the same pixel on the output of a vibration reduction device according to the prior art; - Figure 3, a motion interpolation and an estimation device that allows the reduction of vibration and implements the process according to the invention; - Figure 4, a timing control diagram of the signal outputs through the various modules of the device according to the invention; - Figure 5, the alterations in a pixel at the output of the device according to the invention; - Figure 6, two filtration modes to carry out the interpolation. By virtue of the invention, the construction expense is reduced, limiting the requirements for the frame memory and the delay lines. Using an individual frame memory it is possible to calculate intermediate frames between two frames of the same image in the movie mode through interpolation. The calculated field of motion vectors is of better quality, with no extrapolation required. The movement exhibited is thus improved, limiting block effects and giving better quality motion vectors, thus contributing to the suppression of the vibration effect.

Figure 3 represents a device according to the invention. The input of the device is the input of a double output frequency memory 1 called as an acceleration memory, and very commonly as a "boot memory". The output of the memory 1 is linked in parallel to the input of a frame memory 2, to the input of a motion estimator 3 and to the input of an interpolator 4. The output of the frame memory 2 is linked to a second input of the motion estimator and to a second input of the interpolator. The motion estimator transmits the motion vectors to the interpolator through a link. The output of interpolator 4 is the output of the device. The frame memory 2 also has a control input, which receives a signal corresponding to the camera or film mode of the images entering the device. The digital signals received at the input of this boot memory are in a film mode, ie progressively scanned at the image frequency of 25 Hz. These signals are, as explained above, transported at the frequency of 50 Hz in the linked mode before being transmitted to the input of the device. The operation of the device will be better understood with the help of the curves in Figure 4, which represent three markers with line scan and as ordered and time t as abscissa. F1n and F ^ n are the odd and even frames that correspond to the image n. Referring to Figure 2, the labels F ^ n or F ^ n are cycles 1 or 3 of image n. In the movie mode, the two frames that belong to the same image, for example F ^ n and F ^, actually correspond to the same image. The curve 5 corresponds to the signals y- ^ of input to the start memory, the curve 6 to the output signals y2 through the memory and the curve 7 to the output signals y3 through the frame memory 2 The role of the boot memory is the output, at a double frequency, to the frame stored in the input, in a first period during the storage of the second half of the frame, and a second period during the storage of the first half of the frame. next frame. From here on, the period or frequency of the starting frames of the boot memory will be referred to as the frame period or frequency. The frame memory 2 stores, among the data originating from the double output frequency memory, those corresponding to the second separation frame only of the even frames, ie the second of the frames F2? ' etc. Frame memory transmits over its output, the frame stored after its storage and repeats it four times before moving to the next frame, as shown by curve 7. The role of this memory is, therefore, to select frames from those that leave the boot memory and delay these frames selected by the duration of a frame to four frames, presenting them four times in succession on their output, at frame frequency. The movement estimator 3 compares the frames received on each of its outputs to determine the field of motion vectors. Comparing the curves 6 and 7 corresponding to the signals present on each of the outputs of the movement estimator, it is observed that the movement taken into account for the calculation is that between the pair frame of image n and the odd frame of the following image n + 1 during two frame periods and then between the even frame of the image n and the even frame of the image n + 1 during the next two frame periods. The two compared frames correspond, in the movie mode, to two different instants, since they are always different picture frames. The operation of the motion estimator and the interpolator is specified here below. The circuits must be able to handle both the case where the frames available on their inputs are of similar parity and the case where they are of opposite parity.

Thus, the movement estimator performs vertical filtering of the frames at the entrance allowing the calculation of the motion vectors whether the frames compared are of similar parity or of opposite parity. During a first frame period corresponding to the first presentation of frames as input to the movement estimator, a first estimation of the motion vectors is made by, for example, making a coincidental appearance of main blocks in which the image is divided (MB for "main blocks"), through correlation, a process that is more commonly known as "block match". During the "second step" (the same frames are represented in a second frame period), a refinement of this calculation is performed, calculating the movement vectors of sub-blocks (SB for "sub-blocks") developing the main blocks . This method of calculation in two stages (or 2 cycles) makes it possible to limit the number of delay lines required for these calculations: in this way, since the calculation of a SB vector requires that the closest MB vectors are known, during the calculation of the SB in the cycle after the MB calculation cycle, thus avoiding the introduction of additional delay lines; On the other hand, however, SB vectors are not available until the next cycle. This last point is not problematic since it is not required no interpolation compensated with movement, during the cycle where only MB vectors are available. The table, presented below, represents the various phases for calculating the movement estimate when handling images in movie mode and camera mode. Movie mode: The outputs in double frequency are marked to indicate that this refers to the cycle number i of the image n. The outputs C ^ and C3n thus correspond to the inputs F1n and F2n. The motion estimator calculates the field of motion vectors for the main blocks, for example, for C1 ^ of the frames F ^ n and F2n_1.

The signals of the frames Y2 and Y3 are those from which the motion vectors and the interpolated output frames are calculated, using the appropriate coefficients. In the next cycle C ^ n, the frames presented at the two inputs of the motion estimator are again F ^ _ and This second cycle in this way will allow the motion estimator to refine the calculation of the motion vectors acting on the sub-blocks. During the third and fourth cycles corresponding to the outputs C ^ n and C * n, the frames present are F2n and F2 ^ • The calculation of the motion vectors in this way is also done at a level of the sub-block, the vectors of movement calculated for the main blocks MB during the first cycle being used here since they are still valid. Thus, only the frame changes between the first and the following cycles, the comparison always being made between the image n-1 and n with respect to the calculation of these vectors, and it is known that in the movie mode, the even and odd frames correspond to the same image, that is, to the same moment of the source image. The calculation is repeated for the fourth cycle, since it avoids the storage of the previous vector field and therefore the saturation of the memory. Camera mode: In camera mode, two frames F ^ n and F2 of the same image correspond to two different instants of the source image. Thus, during the first cycle corresponding to C1n, a calculation of the movement vectors on the main macroblocks MB based on the frames F2n_1 and 1n and during the second cycle on the sub-blocks for these same frames is performed. For the third cycle, the present frames are F ^ and F n, from which, therefore, the new motion vectors can be calculated on the MB blocks, the fourth cycle allowing refining in the sub-blocks. The difference between the two modes refers to the frame memory. A signal corresponding to the mode involved is sent to a control input of the frame memory, this input allowing or denying the storage of the frame presented in this entry.

In curve 7 of Figure 4, it can be seen that the y2 signal available in the-frame memory input is sampled in writing at the speed of every fourth cycle in the movie mode, for example F2n_? P2n ' F2? +1 'etc- E ^ stored frame after it is read four times on four successive cycles (or four periods of successive frames). While in camera mode, sampling is performed at a double frequency and the frame stored in this way is repeated over the two successive cycles. The motion vector field calculated for a given frame is transmitted to the interpolator 4. It performs an interpolation as a function of the frame received in its first input and that received in its second input. The coefficients applied to the motion vectors are 0 and 1/2 in the normal mode. In the movie mode, Figure 5 provides a better understanding of the chosen values. A motion vector VI is calculated between the frame F2n_? and * "!, in order to produce the intermediate frame pixel at a distance of 1/4 x VI from the pixel of the first frame and a motion vector V2 is calculated between the frames F n-1 and F n to produce a pixel of the intermediate frame at a distance 3/4 x V2 from the pixel of the first frame The frames F ^ and F n correspond to the same movie image and to the vectors VI and V2, and therefore are identical, this being the why that the calculations are made in the sub-blocks during the third and fourth cycles, while taking into account the movement vectors calculated on the macroblocks during the first cycle. The effect of vibration, as shown by the alignment of the pixels represented by circles (pixels of outputs), is suppressed. A conventional interpolation filter operates with two frames of opposite parities. It is possible to reduce the use of such a filter through the vertical average over one of the input frames. Filtering through the average of one of the two input frames makes it possible to generate an opposite parity field. The interpolation filtering can then be performed in a conventional manner, while averaging over three pixels, mainly two pixels below the other of a frame and that on the intermediate line of the next reconstructed frame of opposite parity. This solution can, however, produce a loss of resolution in the vertical direction. Another solution is to perform filtering based on the input of two even frames to the interpolator in order to obtain an odd frame. The filtering is carried out, for example, on four pixels or two pixels, using a linear / medium filter or simply a linear filter.

An example is presented in Figure 6, which represents the pixels & output (circles) obtained from such filtering of the input pixels (junctions). The filtering choice depends on the motion vector located in the intermediate frame pixel and the parity of the line on which the pixel lies interpolated. The output pixel Y, which has a motion vector of zero, can be generated from the values (luminances) of the input pixels A, C of one frame and the input pixels B, D of the other frame of similar parity with a linear / media hybrid filter.

Y = MEDIA MEDIA (A.B.C.D), '- J.

The average of the operator with an odd number of variables eliminates the extreme values and selects only the central value, the average of the operator with an even number of variables selects the two central values and calculates its average value. The output pixel Z, whose motion vector is +2, can be generated from the pixels E of one frame and D from the other frame of similar parity, changed by a line with a simple linear filter.

The invention is not limited to the described examples, and can also be applied to any type of frame scan converter that requires interpolation. In particular, depending on the relationship between the input frame frequency and the output frame frequency, it is not essential to use a "staircase" type memory in order to implement the invention.

Claims (7)

1. A process for interpolating frames of digital video data received in the form of frames to a frame frequency in a movie mode or in a normal mode, including a frame storage, a motion estimate and an interpolation between the stored frame and a real frame, characterized in that the storage of the movie mode is performed at half the frequency of that in the camera mode, such that only the second of the two input frames corresponds to the same movie image it is stored in the frame memory, in order to be read again twice in succession to be compared with the next two input frames to calculate the motion vectors.

2. A process for converting the digital video data frame frequency including frame storage, movement estimation production motion vectors and an interpolation between the stored frame and a real frame according to claim 1, characterized in that the frame real is stored a first time so that it can be read twice in succession at the double frequency of the input frame frequency in order to produce y2, since y2 is stored every second frame in the normal mode and fourth frame in movie mode, each time, - the last of these frames, in order to subsequently read the double frequency of the input frame frequency to produce y3, and the motion estimation and interpolation is perform between the frames y2 and y3 that are read at twice the frequency of the frame frequency input.

3. The process according to claim 1 or 2, characterized in that the interpolation is taken in advance, when the two received frames are of similar parity, a prefiltration through the vertical average of one of the two frames y2 in order to produce a opposite parity frame for y34.

The process according to claim 1 or 2, characterized in that the interpolation is taken beforehand, when the two received frames are of similar parity, a linear / average filtering on four pixels that coincide by the motion vector of the pixel to be interpolated , the first two pixels belonging to the two consecutive lines, and the other two to the corresponding consecutive lines in the other frame, the horizontal and vertical positions of these pixels being defined by the position of the pixel that is going to be interpolated and by the motion vector located in this pixel.

5. The process according to claim 1 or 2, characterized in that the interpolation is taken beforehand when the two received frames are of similar parity, a linear filtering in the two pixels coincides through the motion vector of the pixel to be interpolated, a pixel of a frame and the corresponding pixel of the other frame, the horizontal and vertical positions of these pixels being defined by the position of the pixel to be interpolated and by the vector of motion distributed to this pixel.

6. The process in accordance with the claim 4 or 5, characterized in that the choice of filtering depends both on the vector of motion distributed to the pixel of the interpolated frame and on the parity of the line to which this pixel belongs.

7. A device for interpolating a frame of digital video data received in the form of frames in the movie mode or in the normal mode, which includes a frame memory that receives digital video data, a motion estimator and an interpolator, each one having two entries, its first inputs receiving the same digital video data as the frame memory input, its second inputs receiving the output from the frame memory, characterized in that the digital video data passes in advance through a memory type of " "staircase", producing the video data in the output to a double frequency by repeating the frames twice, the frame memory writes c gives the second frame and then reads this frame twice twice the frequency of the input frame frequency in the camera mode, and writes the fourth frame and then reads this same frame four times twice the frequency of the frame frequency in the movie mode, the frame stored by the frame memory being each time the second of the two frames that arise from the same image.