GB2162018A - Moving picture frame rate conversion system - Google Patents

Description

1 GB 2 162 018 A 1

SPECIFICATION

Moving picture frame rate conversion system This invention relatesto a pictureframe rate 70 conversion system fortelevision signals andthe like. It is particularly suitable for moving pictureframe rate conversions, for example mutual conversions be tween twoTVstandard systems, and is effective in recovering atime-axis compressed moving picture signal.

Moving pictures in general, includingTV pictures, are notgiven by a continuous signal in time. This is becausethe basic principle of a moving picture isto displaya certain fixed number (orframe rate) of quiescent pictures one after another per second.

Regarding TV systems, for example, Japan's TV system has aframe rateof 30frames/second, while theTVsystem adopted in European countries includ ing the U.K. and France has aframe rate of 25 f ram es/seco nd. Therefore, it is essential to have an appropriate frame rate conversion techniquefor enabling the exchange of TV broadcast programmes between countries having different frame rates.

The conventional technique used in converting frame rate will now be explained. A colourTV signal, in general, is composed of three components known as R, G, and B or Y, 1, and Q respectively. In converting the frame rate of such a signal, the signal is first decomposed to its three components, and then each component is subjected to a desired f rame rate conversion bythe same method. Therefore, the conventional frame rate conversion of one of the three components is explained below. This is equivalent to that of a monochrome TV signal.

Fig. 1 and Fig. 2 of the accompanying drawings showthe basic principle of frame rate conversion from a frame rate of 25 frames/second to a frame rate of 30 frames/second orvice versa. That is, conversion from 25 frames/second ((a)25, (b)25. -) into 30 frames/ 105 second ((a)30, (b)30, -) is shown in Fig. 1 and the inverse conversion is shown in Fig. 2. In both cases, each of the vertical solid lines and dotted lines represents one frame of a moving picture signal which extends two-dimensional ly.

Asthe ratio oftheframe rates in Fig. land Figs. 2 is 25130 = 516, if atan instant in time the position of a frame of 25frames/sec. and that of a frame of 30 frames/see. are synchronised, synchronisation of both frame groups takes place at every 5 frame intervals of one or 6 frame intervals of the other. For example, in Fig. 1, starting with coincidence of (a)25 and (8)30, the position Of M25 coincides with that of (9)30. In Fig. 2, in the same condition the position of (9)30 coincides with that Of M25. Generally, in f rame rate conversion, the process is repeated with a repetition period determined by the ratio of the f rame rates (in the above examples, 5 f rame intervals of one group and 6 frame intervals of the other group), and it is sufficient to explain the process for one such period. 125 In Fig. 1, the converted frames (a)30, (b)30,---,(9)30 are obtained as follows.

f ram e (a)30 --- frame (a)25 itself, frame (b)30 --- synthesised with frame (a)25 and frame (b)25.

frame M30 --- synthesised with frame (b)25 and frame M25.

frame (d)30 --- sunthesised with f rame (025 and frame (d)25.

frame (e)30 --- sunthesised with frame (d)25 and frame (e)25.

f ram e M30 --- synthesised with frame (e)25 and frame M25.

f ram e (9)30 --- frame M25 itself.

An interpolation process is carried out in synthesising the converted frames. A device which performs the interpolation process is called an interpolation filter and, in the explanation to follow,the simplest interpolation filterwhich can be realised in hardware, the linear interpolation filter, is considered as an example.

Fig. 3 shows an interpolation process which provides an interpolated frame 1 from given consecutive frames, i.e. frame A and f rame B. That is, each picture element of the interpolated frame 1 (i.e. a sampled value of the sampled part of a picture signal) Yij (where i is the scanning line number and j is the column number counted f rom the furthest left) is produced from a picture element W1,)of frame A and a picture element X(2i,)of f rame B, each of which has the same position coordinates as those of the picture element yij. The interpolation rule is given by the following expression.

yiI = b W1)jj + a X(2),j --- ------------------------------------ (1) where a and bare proportional to the distance in time axis between the frame A and the frame[ and that between the frame Band the frame 1 respectively, and are normalised asfollows.

a + b = 1.0 (a, b->0) (2) Forexample, in Fig. 1,when theframe (030iS synthesisedfrom theframe (b)25 andtheframe (025, a and b are given as follows.

a = 213, b = 113 In the example shown in Fig. 3, if the original picture signal is a quiescent picture signal,the signal value of the frame A isthe same as that of the frame B ignoring possible noise superposed, and the signal value of the interpolated frame 1 is equal to that of theframe A or that of the frame B. Therefore, no deterioration arises in picture resolution. If the original signal is a moving picture signal, however, two picture elements X(!,,), XRol) shown in Fig. 3, rarely have the same signal value, and an extra noise component, known as area error is included in the corresponding interpolated picture element. As a result, the conventional technique has the following drawbacks.

(i) The signal of the interpolated frame 1 is badly blurred.

(ii) The movement of the converted moving picture is somewhat unnatural (this is referred to as jerkiness).

It will be clearthatthese drawbacks become more striking with progress in the picture resolution of the TV camera and with increase in velocity of movement of the picture. (see "Aframe rate conversion for moving picture signals- an analysis of the conversion characteristic with thetime axis direction process." (in Japanese) Trans. of the Institute of Electronics and Communication Engineers of Japan,'8412, Vol. J 67-13, No. 2J.

The present invention is aimed at providing a frame 2 GB 2 162 018 A 2 rate conversion techniquewhich alleviatesthe drawbacks of the conventional time axis direction interpolation process, and provides a converted moving picture signal with little deterioration in picture resolution even when the original signal is in quick motion.

According to the invention there is provided a picture frame rate conversion system which converts a first picture signal having a firstframe rate into a second picture signal having a second frame rate which is differentfrom thefirstframe rate, by generating an interpolation frame between two consecutive frames of the first picture signal; wherein the interpolation frame is generated by using a first picture block in a firstframe and a second picture block 80 in a second frame of thefirst picture signal,the position of the selected second picture block in its frame being differentfrom the position of the selected first picture block in its frame.

An embodiment of the invention will now be 85 described, by way of example, with reference to the accompanying drawings, in which Fig. 1, Fig. 2 and Fig. 3 show examples of conven tional frame rate conversion systems, as mentioned above, Fig. 4 shows an example of the time axis interpolation of the present invention.

Fig. 5 shows the interpolation of a picture element of a converted frame, Fig. 6 is a block diagram of a frame rate conversion circuit according to the present invention, Fig. 7 is an operational time chartfor use in a frame rate conversion according to the present invention, Fig. 8 is a block diagram of a calculation circuit of Fig. 6, Fig ure 9 is a block diag ram of an interpolation circuit for a frame rate converted picture signal, and Fig. 10 shows the relative deviation of scanning line position between an interpolated frame and an original frame, when the scanning line number ratio is 105 given by 615.

Fig. 4 of the drawings shows the principle of operation of the present invention. Firstly, fora picture elementyij of an interpolated frame 1, a move-quantity forthis picture element, ora block including this 110 picture element, is obtained, the quantity being denoted by (v,,, v.) (picture eiements/frame).

Then, a picture element of a frame A, Xli-)avj-av.), is chosen, the position of which is deviated from that of the picture elementto be synthesised by an amount 115 related to the move-quantity. In the same way, a picture element of the frame B, X121 +bv, j+bv) is chosen.

The picture elemeritto the synthesised, yi,j, is given by the following interpolation process.

X(2 yi,j=bX(,,,lvx,j--avy+a i+3bv,j+bv,,j+bv, -------------------- (3) 120 wherethe parameters a and bare defined as the relative distances between the frame land the frame A ortheframe B, respectively, and are normalised by expression (2).

There is a problem is executing the interpolation 125 process of the equation (3). For example, the calcu lated position of the frame A, i.e. (i-avx,j-av,), does not always coincide with the position of a picture element of frame A, because the picture elements of frame A are defined by discrete points regularly sampled along 130 its scanning lines. Therefore, when a calculated position is notthe position of a picture element of frame A, a linear interpolation with four picture elements of frame A surrounding the calculated position to give an approximated picture element value is found to be effective to avoid picture blur. (1) In this case, the values of picture element Xl-av.,I-av,, and picture element X(12)bv.,l+bv aregiveribythe + v following two expressions respectively.

W'1)av., j-avy 0C2[P2Xi',j'+P1Xi',j+11+al [P2XV+1j' +Pixil+1j1+11 (4) where V= [i- av.] ff j is the symbol of Gauss) j'= [j- av.] ff 1 is the symbol of Gauss) oc, = 11 - avx - avj<'I (X2 = 1 - 0(1 P, = 1 -av, - [ av,11<1 02 = 1 _P1 X(2) i+bv.,j+bv v =Y2152Xi".j"+ 61 XV'j-+1 1 + V1 162 XV'+ 1j" + 61 xl,'+lX,+11 (5) where V' = [i + bvxl ff 1 is the symbol of Gauss) j11 = [j + bv,l ff 1 is the sumbol of Gauss) y, = bv,, - [bv,,] 1 < 1 Y2= _Y1 61 = Ibv,- [bvj<1 62= 1 - 61 The move-quantity (v, v,) can be obtained by one of the known methods. For example, the values of an evaluation function are obtained from an input signal and a single of the preceding frame being in the position shifted by an arbitrary shift vectorwith a fixed point on a TV screen as the position reference. The detection of a move- quantity vector is done in a procedure divided into N steps. At the Lth step, (L = 1, 2,-,N-1),(L+l)thshiftvectorgroupis determined on basis of the evaluation function values obtained with the Lth shift vector group. Finally, atthe Nth step, the move-quantity vector is provided bya vector determined onthe basis of the evaluation function values obtained with the Nth shiftvector group. (Japanese patent laid open publications 55162684,55-162685,55-162683).

Moreover, a method using a so-called gradient process detects any movequantity with high precision. According to this method, in detecting the move-quantity of a picture element ora block of picture elements of a frame, first,the move-quantity (this is a vectorquantity) of a picture element or a block of picture elements of the preceding frame which has the same position coordinates as those of the said picture element or block of picture elements is obtained by applying the gradient process. A picture element or a block of picture elements in a position deviated inversely by the obtained move-quantity is then cut out, and again its move-quantity is obtained by applying the gradient process. The process is repeated until the detected move-quantity becomes less than a predetermined quantity. Atthis stage, the move-quantity of the initial picture element orthe initial block of picture elements is given by the total sum of the detected move-quantities. (Material of the Institute of Electronics and Communication Engineers 3 GB 2 162 018 A 3 of Japan, IE 83-93,---Amove-quantity detection by the repeated gradient process method.", (in Japanese) 1984,2,2j.

Furthermore, the present applicant hasfiled patent applications, concerning a movement estimation system forvideo signals, in the United States of America (US serial No. 696,371), and in Britain (serial No. 8502144).

Fig. 6 shows the circuit configutation of a system according to the present invention. The system comprises a calculating circuit 1 which includes a frame memory 2 where the frame A of Fig. 4 is stored, and which calculates the value of a picture element (1) 1 -av,, j-avy. A calculating circuit 3 contains a frame memory 4 where the frame B of Fig. 4 is stored, and X(2) calculates the value of a picture element 1+bv,,j+bv,.

The system further comprises a frame memory 5 in which the interpolated frame 1 of Fig. 4 is stored, adders 6 and 9, and multipliers 7 and 8. The calculating circuit land the calculating circuit 3 will be explained in detail later. An input terminal "Video W' receives a picture signal, two input terminals "i" and "j" respectively receive coordinates (i,j) of a picture eiementto be interpolated, an inputterminal "a" receives a parameter a, which indicates the position of 90 an interpolated frame, and two inputterminals "vx" and "vy", respectively, receive the move-quantity values (vx,vy) of a given picture signal. The parts of the calculating circuit 1 and the calculating circuit 3 represented by dotted lines are indicated by Fig. 8 or Fig. 9. As explained earlierwith reference to equation (2), the value of the parameter a is uniquely deter mined by the relative position relationship between two consecutive frames of a given picture signal and an interpolated frame situated between them. 100 The circuitry of Fig. 6 operates as follows.

Two consecutive frames of a picture signal at the inputterminal "Video in" are entered into the frame memory 2 and the frame memory 4, respectively, one after another. The calculating circuit 1 and the calculating circuit 3 calculate the value of a picture element and the value of a picture element X11) bvxj+bvy, respectively, with their common input information, i.e. the coordinates (i, j) of a picture element to be interpolated, the move-quantity of the picture signal, and the value of the parameter a. The multiplier7 multiplies the output of the calculating circuit 1 by (1 - a), i.e. =b, and thereby produces the firstterm of the right-hand side of equation (3).

Likewise, the multiplier 8 multiplies the output of the calculating circuit 3 by a and thereby produces the second term of the right-hand side of equation (3).

These signals are added together by the adder 9, the output of which represents the value of a picture element yij. The value of yij thus obtained is stored in 120 the frame memory 5 at an address specified by the coordinates (i, j).

Therefore, the interpolated frame 1 is obtained by reading outthe frame memory 5 atthe appropriate timing.

It should be noted that in the frame rate conversion the read-out rate of the input original frame is different from that of the converted output frame. This fact gives rise to a technical problem in the case of Fig. 1, where the converted frame rate is greaterthan the original frame rate, rather than in the case of Fig. 2, where the converted frame rate is smaller than the original frame rate. Consider the frame rate conversion shown in Fig. 1. Fig. 7 shows the timing relationship between the original video signal which is stored in the frame memory 2 and the frame memory 4, and the converted video signal which is read outfrom the frame memory 5 with a new read-out rate.

In Fig. 7, (a) shows the read-in timing of the original video signal into the frame memory 2 and the frame memory 4, and (b) shows the timing for use in the reading-in or reading-out of the interpolated signal into or from the frame memory 5.

The calculating circuit 1 and the calculating circuit3, which are shown in Fig. 6, are constructed asfollows. The calculating circuit 1 performs the operation required by equation (4) and the calculating circuit3 performs the operation required by equation (5) and it will readily be seen that the circuit structure of the former is basically the same as that of the latter. Therefore, the following explanation is concerned with the circuit structure of the calculating circuit 1.

Fig. 8 shows an eximple of the structure of the calculating circuit 1. The circuit comprises multipliers 10, 15,20,21,22,23,26, and 27, adders 14,16,19,24, 25 and 28, fraction-part extracting circuits 13 and 8, the output of each of which isthe fractional part of its input, integer-part extracting circuits 12 and 17,the output of each of which is the integer part of its input.

As explained earlier with reference to equation (4) and equation (5), (see also Fig. 5), the coordinates of four picture elements to be read outfrom the f rame memory 2 are + 1), (Xi.+1j4, (Xi,+jj,+1) which enclose the coordinates of a picture element calcu lated for interpolation, i.e. (i -avj-av,). Theframe memory 2 is so designed that it provides the four picture elements upon receiving the coordinates of a picture elementfor interpolation, i.e. (i-avx,j-avy). A new coordinate i'is obtained at the output terminal of the integer-part extracting circuit 12 byway of the multiplier 10 and the adder 11. In the same way,j'is obtained at the output of the integer-part extracting circuit 17 byway of the multiplier 15 and the adder 16.

The value of avx is obtained by multiplier 10, the fractional part being denoted by (xl in Fig. 5. The value of av, is obtained by multiplier 15, the fractional part being denoted by P, in Fig. 5. Therefore, (xl and P, appear at the output terminal of fraction-part extract- ing circuit 13 and the output terminal of fraction-part extracting circuit 18, respectively. Adder 14 and adder 19 give the value Of ()Q and the value Of 02, respectively, using the following expressions defined earlier.

0C1 +a2 1,P1 P2 1.

In this way, with the four picture elements and the values of ocl, 0C2, P, , and P2the operation required by the equation (4) is carried out by means of operations 20-28.

The above structure of calculating circuit 1 may give a problem of processing time because it has a lot of operation stages. In orderto speed up the processing time, a simple calculating circuit shown in Fig. 9 is effective where instead of performing the interpola- tion process of the equation (4), one of the four picture 4 GB 2 162 018 A 4 elements used in it is chosen foran interpolated picture element In the circuit of Fig. 9, thecoordinates (i,j) of a picture elemeritto be chosen are determined with a minimum approximation errorby rounding off 5 (i-av., ,j-av,).

The frame rate conversion principle of the invention is shown above in detail. As stated earlier, atthe change of standard frame rate of a TV signal, both the frame rate conversion and the scanning line number conversion are required. When the numberof scanning lines perframe of a given picture signal is differentfrom that of a converted picture signal, the position of a picture element of an interpolated frame gets deviated in the longitudinal direction periodically from the position of a picture elementto be sampled of an original frame. See an exampleshown in Fig. 10, where the scanning line number ratio is given by 615.

In this case, if the distance between two consecutive scanning lines of an original frame is represented by 1, then the distance between two consecutive scanning lines of an interpolated frame is represented by 1.2, and the equation (4) should be modified as follows.

Y1 - = b W '),,v + a X(2 (8) J -., 1.2i-av. a+bv,.1.2i+bv, --- X1 i] a av,,1.2i-av, =(X2102Xi.j +Pl Xi'J'+11 + (X1102Xi'+,j'+ P1 Xi'+J'+11 where V= [i - avxl ff 1 is the symbol of Gauss) j'= [1.2i - avl ff 1 is the symbol of Gauss) (xl = 1 av,, - [avJ J< 1 (X2 1 -OCl 01 = 1 1.2j - av, [1.2j - avl J<1 02 = 1 -Pl In the same way, X(&2+)bv,1.2i+bv, is expressed by a similar equation. The move-quantity (v.,vy) used in the above equation is that of the picture elemeritto be interpolated as is so in the previous case.

The circuits shown in Fig. 8 and Fig. 9 are applicable in the above case with a minimum modification required by replacement ofi with 1.2j.

In general, if a given scanning line number ratio is kfl instead of 615 shown in the example, the equation (8) holds with replacement of 1.2 with kri.

As stated above, the invention makes it possible to have a converted picture signal of high quality with

Claims (4)

little picture blur and jerkiness even when an original picture signal is in quick motion, by means of detecting the move-quantity of the original signal and using it in the frame interpolation process. CLAIMS

1. A picture frame rate conversion system which converts a first picture signal having a firstframe rate into a second picture signal having a second frame ratewhich is differentfrom the firstframe rate, by generating an interpolation frame between two con- secutive frames of the first picture signal; wherein the interpolation frame is generated by using a first picture block in a first frame and a second picture block in a second frame of the first picture signal, the position of the selected second picture block in its frame being differentfrom the position the selected first picture block in its frame.

2. Asystern according to claim 1, wherein signals representing a picture element of the first picture block and a picture element of the second signal block are obtained by using four lattice point elements enclosing the respective picture block element.

3. A system according to claim 1 or claim 2, comprising a firstframe memory having a first calculating circuitfor providing a firstsignal for the first frame picture block; a second frame memory having a second calculating circuitfor providing a second signal Xi,b,.,j+b, forthe second frame picture block; means for providing an interpolation signal Yii according to thefirstsignal and the second signal; and a third frame memory for storing the thus interpolated frame andforproviding interpolated frames atthe converted rate; wherein i andj are the scanning line numberandthe picture cell number to provide the coordinates, a and b are determined according to the distance between the original frames and the interpolated frame where a-b= 1, and (v., v,) represents the moving amount of the picture.

4. A picture frame rate conversion system substantially as hereinbefore described with reference to Figs. 4to 10 of the accompanying drawings.

Printed in the United Kingdom for Her Majesty's Stationery Office. 8818935, 1186 18996. Published at the Patent Office, 25 Southampton Buildings, London WC2A lAY, from which copies may be obtained.