US5355169A - Method for processing a digital video signal having portions acquired with different acquisition characteristics - Google Patents

Method for processing a digital video signal having portions acquired with different acquisition characteristics Download PDF

Info

Publication number
US5355169A
US5355169A US08/031,109 US3110993A US5355169A US 5355169 A US5355169 A US 5355169A US 3110993 A US3110993 A US 3110993A US 5355169 A US5355169 A US 5355169A
Authority
US
United States
Prior art keywords
frames
input
picture
acquisition
series
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
US08/031,109
Other languages
English (en)
Inventor
John W. Richards
Martin R. Dorricott
Morgan W. A. David
Stephen M. Keating
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Europe BV United Kingdom Branch
Original Assignee
Sony United Kingdom Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony United Kingdom Ltd filed Critical Sony United Kingdom Ltd
Assigned to SONY UNITED KINGDOM LIMITED reassignment SONY UNITED KINGDOM LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: DAVID, MORGAN W. A., DORRICOTT, MARTIN R., KEATING, STEPHEN M., RICHARDS, JOHN W.
Assigned to SONY UNITED KINGDOM LIMITED reassignment SONY UNITED KINGDOM LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SONY BROADCAST & COMMUNICATIONS LIMITED
Application granted granted Critical
Publication of US5355169A publication Critical patent/US5355169A/en
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
    • H04N7/014—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes involving the use of motion vectors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection
    • H04N5/145—Movement estimation

Definitions

  • This invention relates to digital video signal processing and in particular with processing an input digital video signal representing a series of input frames to produce an output digital video signal representing a series of output frames.
  • the invention is more particularly concerned with the case where at least some of the input frames each have a first picture portion acquired with a first acquisition characteristic (such as, for example, 60 fields/second, 2:1 interlace) and a second picture portion acquired with a second acquisition characteristic (such as, for example, 30 frames/second, 1:1 progressive scan).
  • a first acquisition characteristic such as, for example, 60 fields/second, 2:1 interlace
  • a second picture portion acquired with a second acquisition characteristic (such as, for example, 30 frames/second, 1:1 progressive scan).
  • This invention is more particularly, but not exclusively, concerned with the case where the output signal simulates frames acquired with a different acquisition characteristic (such as, for example, 24 frames/second, 1:1 progressive scan), but is also applicable in the case where the output signal is to simulate frames wholly acquired with the acquisition characteristic of one of the picture portions of the input signal (such as, for example, 60 fields/second, 2:1 interlace).
  • a different acquisition characteristic such as, for example, 24 frames/second, 1:1 progressive scan
  • the output signal is to simulate frames wholly acquired with the acquisition characteristic of one of the picture portions of the input signal (such as, for example, 60 fields/second, 2:1 interlace).
  • Patent Application GB 2231228 A describes a method of converting a digital video signal in 60 fields/second, 2:1 interlace, format so as to simulate acquisition at 24 frames/second, 1:1 progressive scan format
  • progressive scan format frames are produced temporally aligned with the first, third, fourth, sixth, eighth and ninth of the ten input fields each either by inter-field interpolation of the respective input field with the preceding and succeeding fields, and/or by intra-field interpolation of the respective input field, depending upon the amount of motion detected in the input picture.
  • the first and Fourth progressive scan converted frames can be directly used as the first and third output frames.
  • the second and Fourth output frames are produced by motion compensated temporal interpolation half-way between the second and third progressive scan converted frames, and half-way between the fifth and sixth progressive scan converted frames, respectively.
  • Motion compensated temporal interpolation involves producing, for each pixel in the output frame, a motion vector indicative of the motion of the object which that pixel represents between ,the respective pair of progressive scan converted frames.
  • the value of each pixel in the output frame is then determined by averaging the values of the pixels located at positions in the respective pair of progressive scan converted frames offset by one-half of the motion vector one way and the other, respectively, from the position of the pixel in the output frame.
  • Patent Application GB 2249907 A describes a number of developments to the method described in GB 2231228 A, for example for converting 30 frames/second, 1:1 progressive scan format material by motion compensated temporal interpolation so as to simulate acquisition at 24 frames/second, 1:1 progressive scan format.
  • an object of one aspect of the invention is to convert an input video signal which is a composite of material acquired with a 60 fields/second, 2:1 interlace, video camera and material acquired from a computer graphics system in 30 frames/second, 1:1 progressive scan format (but included in the 60 fields/second, 2:1 interlace, signal with the fields of each pair temporally co-sited) to produce an output video signal which simulates acquisition at 24 frames/second, 1:1 progressive scan format, so that the output signal can be used, for example, with an electron beam recorder to produce a 24 frames/second film.
  • This example involves a change of frame rate.
  • An object of another aspect of the invention is to convert an input video signal crudely composited as above to produce an output signal which simulates acquisition of the whole picture in 60 fields/second, 2:1 interlace, format. This latter example does not involve a change of frame rate.
  • a first aspect of the present invention is more particularly concerned with providing a method of processing an input digital video signal representing a series of input frames to produce an output video signal representing a series of output frames, at least some of the input frames each having a first picture portion having a first acquisition characteristic and a second picture portion having a second acquisition characteristic.
  • the method of this first aspect of the present invention includes by the steps of distinguishing between the first and second picture portions in the input frames; and processing the first and second picture portions differently to produce the output frames so that the output frames simulate acquisition of the first and second picture portions with generally the same acquisition characteristic.
  • the distinguishing step may comprise detecting picture motion between fields of the same input frame; and the processing step may comprise producing an output frame: generally by intra-field interpolation of a respective input field in areas of relatively large detected motion; and generally by combination of respective input fields in areas of no or relatively small detected motion.
  • the processing step preferably includes combining, for each pixel, the corresponding intra-field interpolated pixel value with the corresponding field-combined pixel value in proportion to a degree of detected motion of that pixel.
  • this example of the method may further comprise the step of further processing the output frames using motion compensated temporal interpolation to simulate acquisition with a different acquisition characteristic (e.g. 24 Hz, 1:1).
  • first example of the invention involves decimation of the temporal sampling rate, and thus there is a loss of picture data.
  • the second picture portions are acquired as pairs of interlaced fields at the first frame rate (e.g.
  • the distinguishing step may include forming a first series of difference frames corresponding to the difference between the fields of each input frame; and the processing step may include: forming a series of progressive scan picture frames at least by intra-field interpolation of one field of each frame so that the interpolated picture frames are temporally co-sited with the first picture portions of respective ones of the input frames; producing a second series of difference frames by motion compensated temporal interpolation of the first series of difference frames so that the second difference frames are temporally co-sited with respect to the first picture portions of respective ones of the input frames; and matting the input frames and the interpolated picture frames using the second difference frames as a key signal.
  • the distinguishing step may include forming a first series of difference frames corresponding to the difference between the fields of each input frame; and the processing step comprises the steps of: forming a first series of intermediate frames by a motion compensated temporal interpolation process for converting the first picture areas to simulate acquisition with the third acquisition characteristic; forming a second series of intermediate frames by a motion compensated temporal interpolation process for converting the second picture areas to simulate acquisition with the third acquisition characteristic, each second intermediate frame being temporally co-sited with a respective one of the first intermediate frames; forming a second series of difference frames by motion compensated temporal interpolation for converting the first difference frames to the third acquisition characteristic, each second difference frame being temporally co-sited with a respective one of the first intermediate frames; and matting the first and second intermediate frames using the second difference frames as a key signal.
  • the processing step comprises the steps of: forming a first series of intermediate frames by a motion compensated temporal interpolation process for converting the first picture areas to simulate acquisition with the third acquisition characteristic
  • This second example of the invention and the modification thereto involve the generation of difference frames from the source video.
  • the source video may have been post-produced using a key signal, and in a third example of the invention this key signal is utilised in the acquisition conversion process.
  • this key signal is utilised in the acquisition conversion process. More specifically, in the third example of the method of the invention in which the first picture portions are acquired as progressive scan frames at a first frame rate (e.g. 30 Hz, 1:1), the second picture portions are acquired as pairs of interlaced fields at the first frame rate (e.g. 60 Hz, 2:1), the input signal is accompanied by a series of key frames indicating the first and second picture portions, and the output video signal simulates acquisition with a third different acquisition characteristic (e.g.
  • the distinguishing step may comprise detecting the key frames; and the processing step may include: forming a first series of intermediate frames by motion compensated temporal interpolation for converting the first picture areas to simulate acquisition with the third acquisition characteristic; forming a second series of intermediate frames by motion compensated temporal interpolation for converting the second picture areas to simulate acquisition with the third acquisition characteristic, each second intermediate frame being temporally co-sited with a respective one of the first intermediate frames; forming a second series of key frames by motion compensated temporal interpolation for converting the detected key frames to the third acquisition characteristic, each second key frame being temporally co-sited with a respective one of the first intermediate frames; and matting the first and second intermediate frames using the second key frames as a key signal.
  • the processing step includes: processing the input frames to simulate a first conversion from the first acquisition characteristic to the output acquisition characteristic using motion compensated temporal interpolation in which motion vectors are developed for pixels in the picture (for example as described in patent application GB 2231228 A or GB 2249907 A); and developing a measure of merit for each motion vector; and the distinguishing step is performed in dependence upon the measures of merit.
  • the processing step may further include: processing the input frames to simulate a second conversion from the second acquisition characteristic to the output acquisition characteristic using motion compensated temporal interpolation in which motion vectors are developed for pixels in the picture; and developing a measure of merit for each motion vector; and the distinguishing step may be performed in dependence upon a comparison of the measures of merit of the first and second conversions. The distinguishing step may also be performed in dependence upon a comparison of the motion vectors of the first and second conversions.
  • the present invention also provides apparatus adapted to perform the methods described above.
  • FIG. 1 illustrates a first example of a method of video signal processing according to the invention
  • FIG. 2 is a block diagram of an apparatus for performing the method of FIG. 1;
  • FIG. 3 is a block diagram showing more detail of a motion detector in the FIG. 2 apparatus
  • FIG. 4 shows an example of a characteristic of a non-linear motion estimate function of the FIG. 3 apparatus
  • FIG. 5 illustrates the temporal relationship of input and output frames in 30 Hz, 1:1 to 24 Hz, 1:1 conversion
  • FIG. 6 illustrates a second example of a method of video signal processing according to the invention
  • FIG. 7 shows sample pictures in the method of FIG. 6
  • FIG. 8 is a block diagram of an apparatus for performing the method of FIG. 6;
  • FIG. 9 shows sample pictures in a third example of a method of video signal processing according to the invention.
  • FIG. 10 is a block diagram of an apparatus for performing the method of FIG. 9;
  • FIG. 11 is a block diagram of an apparatus for performing a fourth example of a method according to the invention.
  • FIG. 12 illustrates further the fourth example of the invention.
  • FIG. 1 shows frames having only nine active lines, and it should be noted that actual high-definition video has of the order of one thousand active lines.
  • the temporal sampling rate of the input signal is decimated to the lowest common denominator of the acquisition rates of the two portions of the input signal, that is 30 Hz, 1:1 in this example.
  • the 60 Hz, 2:1, acquired rectangle would suffer from interlace smear, as shown in row B of FIG. 1.
  • the picture is formed by intra-field interpolation of one of the fields of the pair.
  • the picture is formed by a combination of intra-field interpolation of one of the fields of the pair and a combination of the two fields of the pair.
  • the whole of the combined frame is produced by combination of the two fields of the pair. Assuming that the square of FIG. 1 has a large degree of motion, then this process will produce output frames as shown in row C of FIG. 1, which are a better 30 Hz 1:1 representation of the picture than the frames of row B of FIG. 1.
  • Input fields if n on line 10 are supplied to a field store 12, an intra-field interpolator 14, a motion detector 16 and a field combiner 18.
  • the field store 12 produces a delay so that the fields if n-1 output therefrom on line 20 are one field period behind the input fields if n .
  • the delayed fields if n-1 are also supplied to the motion detector 16 and the field combiner.
  • Intra-field interpolated frames are supplied by the interpolator 14 to a proportional combiner 22, and field combined frames are supplied by the field combiner 18 to the proportional combiner 22.
  • the motion detector 16 For each pixel in a frame to be output, the motion detector 16 produces a motion estimate value P n ranging from, say, 0 to 255 indicative of the detected motion of the object which that pixel represents, and the proportional combiner 22 combines the values of the corresponding pixels produced by the intra-field interpolator 14 and the field combiner 18 in accordance with the value of the motion estimate value P n to produce the output pixel value, which is supplied to an output buffer 24, which then produces output frames OF n/2 simulating acquisition at 30 Hz, 1:1.
  • the intra-field interpolator 14 produces a vertically interpolated frame IF n/2 from every other input field if 2 , if 4 , if 6 , . . . .
  • the value of a pixel at location (x,y) in the interpolated frame IF n/2 is equal to the value of the pixel at the same location (x,y) in the corresponding even input field if n .
  • the value of a pixel at location (x,y) in the interpolated frame IF n/2 is the average of the values of the pixels at the locations (x,y-1) and (x,y+1) above and below that location in the corresponding even input field if n . That is to say:
  • the field combiner 18 produces a combined frame CF n/2 which is a direct combination of the respective pair of input fields if n-1 and if n , so that the even lines of the combined frame CF n/2 are taken from the even input field if n and the odd lines are taken from the preceding odd input field if n-1 . That is to say:
  • the proportional combiner 22 produces the value OP n/2 (x,y) of each pixel of the output frame OF n/2 by adding the proportion P n (x,y) of the value IF n/2 (x,y) of the corresponding pixel of the intra-field interpolated frame with the complementary proportion 256-P n (x,y) of the value CF n/2 (x,y) of the field combined frame. That is to say:
  • the motion detector 26 will now be described in further detail with reference to FIG. 3.
  • the operation of the motion detector is in some ways similar to the operation of the corresponding apparatus described in detail with reference to FIGS. 8 to 14 of patent application GB. 2231228 A.
  • the inter-field difference signal of two fields of the same frame is used to indicate motion, rather than the difference signal between the two fields to either side of the field under consideration, as in the case of GB 2231228 A.
  • an interpolator 26 is used on the odd field if n-1 to produce an interpolated even polarity field if n-1 ', and a filter 28 is used on the even input field if n to match the response of the interpolator 26 and produce a filtered even polarity field if n '. That is to say:
  • a difference modulus field df n is produced from the interpolated and filtered fields if n-1 ', if n ' by a subtracter 30 and a modulus operator 32 such that, for y even:
  • an average intensity corrector 36 produces a field ndf n of normalised difference values, for example in accordance with the formula, for y even:
  • i max is the maximum pixel value.
  • the normalised difference field ndf n is then vertically filtered with the preceding and succeeding normalised difference fields ndf n 2, ndf n 2 for example using a three-tap filter 38 with coefficients of one-quarter, one-half and one-quarter, to produce a filtered difference field fdf n . That is to say, for y even:
  • the first filtered difference field fdf n is then filtered by a spatial intra-field filter 40 in order to produce a difference frame DF n , for example such that:
  • the arrangement described with reference to FIGS. 1 to 4 converts material having portions acquired at 60 Hz, 2:1 and at 30 Hz, 1:1 so as to simulate acquisition as a whole at 30 Hz, 1:1.
  • the converted material may be directly used, or alternatively it may be further converted using motion compensated temporal interpolation so as to simulate acquisition in a different format, such as 24 Hz, 1:1 as described in detail in patent application GB 2249907 A, particularly with reference to FIG. 70.
  • row A represents a series of six 30 Hz, 1:1 input frames IF 1 to IF 6
  • row B represents a series of five 24 Hz, 1:1 output frames OF 1 to OF 5 .
  • the first output frame OF 1 may be directly derived from the first input frame IF 1 .
  • the second output frame OF 2 is temporally interpolated with motion compensation one quarter of the way between the second and third input frames IF 2 , IF 3 ;
  • the third output frame OF 3 is temporally interpolated with motion compensation half-way between the third and fourth input frames IF 3 , IF 4 ;
  • the fourth output frame OF 4 is temporally interpolated with motion compensation three-quarters of the way between the fourth and fifth input frames IF 4 , IF 5 .
  • the process repeats for the next five input frames IF 6 to IF 10 and the next four output frames OF 5 to OF 8 , and so on.
  • motion compensated temporal interpolation it is meant that for each pixel in the output frame, a motion vector is generated indicative of the motion of the object which that pixel represents between the corresponding two input frames to be interpolated.
  • the value of the output pixel is then derived from the values of the pixels in the two input frames offset from the location of the output pixel by amounts dependent on the motion vector and the temporal interpolation ratio.
  • the output pixel value OF 2 (x,y) at location (x,y) is given by:
  • M 2 (x,y) is the motion vector for a pixel at location (x,y) in output frame OF 2
  • IF 2 ⁇ and IF 3 ⁇ are the pixel values at the appropriate locations in input frames IF 2 , IF 3 .
  • row A represents a series of six frames A 1 to A 6 or twelve fields a 1 to a 12 of 60 Hz, 2:1 interlace material.
  • 60 Hz, 2:1 interlace some parts of the picture were acquired using a 60 Hz, 2:1 interlace video camera and thus the two fields of each frame are temporally offset, whereas other parts of the picture were acquired by 30 Hz, 1:1 computer graphics, and thus in these parts the two fields of each frame are temporally aligned.
  • the input fields a 1 to a 12 are vertically interpolated to produce a series of twelve full frames B 1 to B 12 as represented by row B, and each pair of vertically interpolated frames derived from the same input frame are subtracted to produce a series of six difference frames C 1 to C 6 , as represented by row C.
  • the difference frames may be produced by simple subtraction, but more preferably the difference values are also normalised by an arrangement similar to elements 30 to 36 described above with reference to FIG. 3.
  • the six difference frames are then 30 Hz, 1:1 to 24 Hz, 1:1 converted using motion compensated temporal interpolation to produce a series of four key frames D 1 to D 4 , as represented by row D.
  • the first key frame D 1 is interpolated three-quarters of the way between the first and second difference frames C 1 , C 2 ; the second key frame D 2 is directly derived from the third difference frame C 3 ; the third key frame D 3 is interpolated one-quarter of the way between the fourth and fifth difference frames C 1 , C 5 ; and the fourth key frame D 4 is interpolated half-way between the fifth and sixth difference frames C 5 , C 6 .
  • Row E of FIG. 6 also shows the six input frames A 1 to A 6 /twelve input fields a 1 to a 12 .
  • the two fields of each frame are directly combined to produce a series of six full frames F 1 to F 6 as represented by row F.
  • the 30 Hz, 1:1 full frames are then 30 Hz, 1:1 to 24 Hz, 1:1 converted using motion compensated temporal interpolation to produce a series of four 30 Hz, 1:1 to 24 Hz, 1:1 converted frames G 1 to G 4 as represented by row G.
  • the first converted frame G 1 is produced directly from the second 30 Hz frame F 2 .
  • the second converted frame G 2 is interpolated one-quarter of the way between the third and fourth 30 Hz frames F 3 , F 4 .
  • the third converted frame G 3 is interpolated half-way between the fourth and fifth 30 Hz frames F 1 , F 5 .
  • the fourth converted frame G 4 is interpolated three-quarters of the way between the fifth and sixth 30 Hz frames F 5 , F 6 .
  • Row H of FIG. 6 also shows the six input frames A 1 to A 6 /twelve input fields a 1 to a 12 .
  • certain of the fields are progressive scan converted using motion adaptive intra-field and inter-field interpolation (as described in detail in patent application GB 2231228 A) to produce a series of six full frames I 1 to I 6 , as represented by row I.
  • the first to sixth progressive scan converted frames I 1 to I 6 are produced from the third, fifth, sixth, eighth, tenth and eleventh input fields a 3 , a 5 , a 6 , a 8 , a 10 , a 11 , respectively, and optionally from the fields immediately preceding and succeeding those fields.
  • the six progressive scan converted frames I 1 to I 6 are then converted using motion compensated temporal interpolation to produce a series of four 60 Hz, 2:1, to 24 Hz, 1:1 converted frames J 1 to J 4 , as represented by row J.
  • the first and third 24 Hz, 1:1 frames J 1 , J 3 are directly produced from the first and fourth progressive scan converted frames I 1 , I 4 .
  • the second and fourth 24 Hz, 1:1 frames J 2 , J 4 are produced by motion compensated temporal interpolation half-way between the second and third progressive scan converted frames I 2 , I 3 , and half-way between the fifth and sixth progressive scan converted frames I 5 , I 6 , respectively.
  • the four key frames D 1 to D 4 of row D ape temporally aligned with respective ones of the four 30 Hz, 1:1 to 24 Hz, 1:1 converted frames G 1 to G 4 of row G and are also temporally aligned with respective ones of the four 60 Hz, 2:1 to 24 Hz, 1:1 converted frames J 1 to J 4 of row J.
  • the output picture signal is produced by selecting either the 30 Hz, 1:1 to 24 Hz, 1:1 converted signal (row G) or the 60 Hz, 2:1 to 24 Hz, 1:1 converted signal (row J) in dependence upon the value, for the pixel under consideration, of the 24 Hz, 1:1 key signal (row D). Specifically, if the key signal is less than a predetermined threshold, the corresponding pixel value of the 30 Hz, 1:1 to 24 Hz, 1:1 signal is selected, whereas if the key signal for that pixel is greater than the threshold, the corresponding pixel value of the 60 Hz, 2:1 to 24 Hz, 1:1 signal is selected.
  • FIG. 7 An example of how the above-described arrangement may operate is shown in FIG. 7.
  • Row A of FIG. 7 shows five fields a 3 to a 7 of the input 60 Hz, 2:1 video signal.
  • a circular object acquired at 60 Hz, 2:1 is moving to the left, whereas towards the bottom of the picture a rectangular object acquired at 30 Hz, 1:1 is moving to the right.
  • the background is static. (For simplicity, FIG. 7 does not attempt to distinguish between the odd and even lines of a frame. ) Because the circular object was acquired at 60 Hz, 2:1, its position appears to move progressively over the five fields a 3 to a 7 of row A of FIG. 7.
  • Row C of FIG. 7 shows the three corresponding difference frames C 2 to C 4 .
  • the only movement between the two fields of a frame is that of the moving circular object, and the difference frames contain the portions of the circular object which are covered or uncovered between fields of a pair.
  • Row D of FIG. 7 shows the corresponding key signal frames D 1 , D 2 .
  • FIG. 7 shows the two corresponding 60 Hz, 2:1 to 24 Hz, 1:1 converted frames J 1 , J 2 '
  • the circular object is correctly located, temporally interpolated between its locations in the third and fourth fields a 5 , a 6 of row A.
  • the rectangular object is in the wrong position, because its position should have been interpolated between the second and third frames A 3 , A 4 of row A, rather than being derived primarily from its position in the third and fourth fields a 5 , a 6 of row A.
  • Row G of FIG. 7 shows the two corresponding 30 Hz, 1:1 to 24 Hz, 1:1 converted frames G 1 , G 2 .
  • a source digital video tape recorder (DVTR) 44 can supply the input fields to a key signal processor 46, a frame store 48 and a 60 Hz, 2:1 to 24 Hz, 1:1 motion compensated temporal interpolator 50.
  • the key signal processor 46 includes a vertical interpolator 52 which produces the full frames B even for the even input fields a 2 , a 4 , a 6 , . . . , and a field store 54 and vertical interpolator 56 which produces the full frames B odd for the odd input fields a 1 , a 3 , a 5 , . . . .
  • the even and odd full frames B even , B odd are supplied to a subtractor 30 and modulus operator 32 and also to an adder 34, which supply an average intensity corrector 36, operating in the manner described above with reference to FIG. 3 (but on full frames rather than fields) to produce the difference frames C.
  • the differenced frames C can then be supplied to a 30 Hz, 1:1 to 24 Hz, 1:1 motion compensated temporal interpolator 57 as described in further detail with reference to FIG. 70 of patent application 2249907 A, which produces the key frames D which are recorded on a DVTR 56.
  • the frame store 48 operates to combine the input field pairs a 1 , a 2 ; a 3 , a 4 ; a 5 , a 6 ; . . . to produce the combined frames F, which can then be supplied to the interpolater 57 to produce the 30 Hz, 1:1 to 24 Hz, 1:1 converted frames G which are recorded on a DVTR 58.
  • the interpolator 50 is as described with reference to FIGS. 1 to 48 of patent application GB 2231228 A and produces from the input fields a the 60 Hz, 2:1 to 24 Hz, 1:1 converted frames J which are recorded on a DVTR 60.
  • the DVTR 56 can supply the recorded key frames D via a thresholder 62 to the control input of a multiplexer 64.
  • the thresholder 62 may simply pass the most significant bit of the key frame video signal so that the threshold is set at 128 for 8-bit video data, or alternatively may involve a more complicated thresholding arrangement.
  • the DVTRs 58, 60 can supply the 30 Hz, 1:1 to 24 Hz, 1:1 converted frames G and the 60 Hz, 2:1 to 24 Hz, 1:1 converted frames J in synchronism to the data inputs of the multiplexer 64, and thus the multiplexer 64 produces the matted output frames K which can be recorded on a DVTR 66.
  • the elements of the arrangement of FIG. 8 are controlled by a system controller 68 which supplies various control signals C 1 to C 10 .
  • input video having a composite acquisition of 30 Hz, 1:1 and 60 Hz, 2:1 is processed so as to simulate acquisition at 24 Hz, 1:1.
  • the output frame rate need not be different to the input frame rate.
  • a modified method could be used to convert video having a composite acquisition of 30 Hz 1:1 and 60 Hz, 2:1 so as to simulate acquisition wholly at 30 Hz, 1:1.
  • motion compensated temporal interpolation would be used to generate, for each pair of input fields, a 30 Hz, 1:1 full frame by combining fields, a 30 Hz, 1:1 progressive scan converted frame using motion adaptive intra-field/inter-field interpolation, and a 30 Hz difference frame, all three frames being temporally aligned by the motion compensated temporal interpolation processes.
  • a key signal is generated by forming a difference signal and then converting the key signal by motion compensated temporal interpolation to the required output format.
  • the key signal is then used to matte the two differently converted picture signals.
  • the input composite video signal may have been post-produced using a key signal, and in the third example of the invention the key signal used in post-producing the picture signal is used alongside the picture signal. If the format of the key signal is different from the required output format, the key signal is converted to the output format, but apart from that the key signal can be used directly as the input to the multiplexer 64 in an apparatus somewhat like that shown in FIG. 8.
  • row A shows a series of 60 Hz, 2:1 input picture frames A 2 to A 4 , or fields a 3 to a 7 , similar to row A of FIG. 7.
  • Row L shows the corresponding key signal fields 1 3 to 1 7 supplied with picture fields a 3 to a 7 .
  • the key signal is 60 Hz, 2:1 rather than 30 Hz, 1:1.
  • the areas shown blank denote the picture area acquired at 30 Hz, 1:1
  • the areas which are cross-hatched denote the picture area acquired at 60 Hz, 2:1.
  • the fields of the key signal of row L nearest to the output frame temporal sitings are progressive scan converted to produce a converted key signal as shown by row M of FIG. 9.
  • converted key signal frame M 1 is produced by progressive scan conversion based on input key signal field 1 3
  • converted key signal frame M 2 is produced by progressive scan conversion based on input key signal field 1 5
  • the input picture fields of row A are also 60 Hz, 2:1 to 24 Hz, 1:1 converted and 30 Hz, 1:1 to 24 Hz, 1:1 converted to produce the frames of rows J and G, respectively, in the same way as described above with reference to FIGS. 6 and 7.
  • the frames of row J can therefore suffer from incorrectly placed objects, and the frames of row G can suffer from loss of vertical resolution and double imaging.
  • the frames of rows J and G are then matted using the converted key signal of row M to produce the output frames as shown in row K.
  • FIG. 9 An apparatus for performing the FIG. 9 method will now be described with reference to FIG. 10 in which elements similar to those in FIG. 8 have been denoted by like reference numerals.
  • the source DVTR 44 can supply the input picture fields a to the frame store 48 which can produce the full 30 Hz, 1:1 picture frames F.
  • the frames can then be converted to 24 Hz, 1:1 by the motion compensated temporal interpolator 57, and the 30 Hz, 1:1 to 24 Hz, 1:1 converted picture frames G can be recorded on the DVTR 58.
  • the source DVTR 44 can also supply the input picture fields a to the interpolator 50 where they can be converted using motion compensated temporal interpolation from 60 Hz, 2:1 to 24 Hz, 1:1, and the converted picture frames J can be recorded on the DVTR 60.
  • the 60 Hz, 2:1 key signal fields are supplied by a DVTR 68 operating in synchronism with the DVTR 44 to a progressive scan converter 70 as described with reference to FIGS. 5 to 14 of patent application GB 2231228 A, which produces a progressive scan key frame M based on the first and third of every five input fields 1 and these converted key frames M are recorded by the DVTR 56.
  • the three DVTRs 56, 68, 60 then reproduce the converted key frames M and the converted picture frames G, J in synchronism, and the multiplexer 64 selects pixels from the 30 Hz, 1:1 to 24 Hz, 1:1 converted frames G or the 60 Hz, 2:1 to 24 Hz, 1:1 converted frames J in dependence on the converted key signal M to produce the output 24 Hz, 1:1 frees K which can be recorded on the DVTR 66.
  • the system controller 68 produces various control signals C 1 to C 9 to control the elements of the apparatus.
  • FIGS. 11 and 12 illustrate a further embodiment of the invention, which does not require a key signal to be supplied with the source video.
  • the source video signal from a DVTR 100 operating in slow-motion is supplied both to a 60 Hz, 2:1 to 24 Hz, 1:1 motion compensated temporal interpolator 102 substantially as described in GB 2231228A and to a 30 Hz, 2:1 to 24 Hz, 1:1 motion compensated temporal interpolator 104 substantially as described in GB 2249907 A.
  • the output of the interpolator 102 or of the interpolator 104 is selected by a multiplexer 106 in dependence upon a signal supplied by a figure of merit comparator 108, and the frames output at less than real-time rate from the multiplexer 106 are grouped in a silicon frame recorder 110 before being recorded by an output DVTR 112 operating in burst mode.
  • the 60 Hz, 2:1 to 24 Hz, 1:1 interpolator 102 comprises a progressive scan converter 114, which produces a progressive scan frame from each of the input fields by motion adaptive intra-field and/or inter-field interpolation.
  • the progressive scan frames are supplied to a time base corrector 116a, which in turn supplies the frames at appropriate timings to a direct block matcher 118a, a motion vector selector 120a and an interpolator 122a.
  • the direct block marcher 118a produces correlation surfaces, from which a motion vector estimator 124a deduces sets of motion vectors representing estimated motion of sample blocks in the picture and flags indicating how the motion vectors were deduced.
  • the sets of motion vectors and their flags are then processed by a motion vector reducer 126a to produce reduced sets of motion vectors, which are passed to the motion vector selector 120a.
  • the motion vector selector 120a assigns one of the motion vectors of the reduced set to each individual pixel in the sample block. In a first stage, motion vectors are assigned for each pixel in the progressive scan converted frames, and in a second stage, the motion vectors produced in the first stage are used to determine the motion vector for each pixel in the output frame which may be temporally sited part-way between a pair of the progressive scan converted frames. As described in detail with reference to FIGS. 36 to 48 of GB 2231228A. various tests are carried out to determine each output frame related motion vector and degrees of match are assessed.
  • a figure of merit FOMa is produced for each motion vector dependent upon the results of the tests carried out by the motion vector selector 120a, and the figure of merit values are supplied to the figure of merit comparator 108.
  • the interpolator 122a interpolates the value of each output pixel in an output frame temporally sited between a pair of the progressive scan converted frames from the values of the pixels in the progressive scan converted frames offset from the location of the pixel in the output frame by amounts dependent upon the respective motion vector and temporally siting of the output frame.
  • the 30 Hz, 1:1 to 24 Hz, 1:1 interpolator 104 is similar to the interpolator 102, except that instead of employing a motion adaptive progressive scan converter 114, progressive scan frames are produced by a field combiner 128 which interlaces the fields of each frame of the input video signal.
  • the interpolator 104 includes a time base corrector 116b, a direct block marcher 118b, motion vector estimator 124b, motion vector reducer 126b, motion vector selector 120b and interpolator 122b operating in the manner described in GB 2231228A and GB 2249907 A, but additionally, like the motion vector sector selector 120a, the motion vector selector 120b also produces for each motion vector a figure of merit FOMb which is supplied to a second input of the figure of merit comparator 108.
  • the figure of merit comparator 108 compares the figures of merit FOMa, FOMb, and if FOMa ⁇ FOMb it produces a value 1 to cause the multiplexer 106 to select the pixel value from the 60 Hz, 2:1 to 24 Hz, 1:1 interpolator 102, whereas if FOMa ⁇ FOMb it produces a value 0 to cause the multiplexer 106 to select the pixel value from the 30 Hz, 1:1 to 24 Hz, 1:1 interpolator 104.
  • the figure of merit comparator 108 may also use the two motion vectors Va, Vb associated with the figures of merit FOMa, FOMb is deciding which interpolator to select. For example, if both motion vectors are almost identical, it may be preferable to select the 60 Hz, 2:1 to 24 Hz, 2:1 comparator 102, even if the figures of merit suggest otherwise, because noise may have adversely affected the accuracy of the figure of merit FOMa.
  • the operation of the figure of merit comparator 108 may be described as:
  • Patent application GB 2231228A describes the use of a motion vector post-processor, between the motion vector selector and the interpolator, which compares each motion vector with the motion vectors of surrounding pixels in an attempt to determine any spurious incorrect motion vectors, which may then be modified in dependence upon the surrounding motion vectors.
  • Such motion vector post-processors may be provided in the arrangement of FIG. 11 between the motion vector selectors 120a, 120b and the interpolators 122a, 122b.
  • the figures of merit FOMa, FOMb may be modified by the motion vector post processors in dependence upon whether or not the respective motion vectors have been determined as likely to be incorrect and whether or not any action has been taken to correct them.
  • the motion vectors produced by the 60 Hz, 2:1 to 24 Hz, 1:1 interpolator for pixels representing that object will be O, V, O, V, . . . for successive frames produced by the progressive scan converter.
  • Such a stream of alternating motion vectors is a clear sign that the pixel was acquired at 30 Hz, 1:1. Therefore in a refinement to the embodiment of FIG. 11, the motion vector reducer 126a may be modified to detect such an alternating stream of motion vectors and either suppress them from being passed to the motion vector selector 120a, or alternatively assist the figure of merit selection.
  • Row A represents a series of five input frames A 1 to A 5 or ten fields a 1 to a 10 of 60 Hz, 2:1 interlaced material, part of which was acquired at 60 Hz, 2:1 interlace and part of which was acquired at 30 Hz, 1:1.
  • the input fields a 1 to a 10 are each progressive scan converted by the converter 114 of FIG. 11 to produce ten frames B 1 to B 10 .
  • a series of four 60 Hz, 2:1 to 24 Hz, 1:1 converted frames C 1 to C 3 is then produced.
  • the first and third frames C 1 , C 3 may be directly produced from the first and sixth progressive scan converted frames B 1 , B 6 .
  • the second and fourth frames C 2 , C 4 are produced by motion compensated temporal interpolation half-way between the third and fourth progressive scan converted frames B 3 , B 4 and half-way between the eighth and ninth progressive scan converted frames B 8 , B 9 , respectively, and figure of merit frames D 2 , D 4 are also produced associated with the motion vectors used in producing the frames C 2 , C 4 .
  • frames C 1 and C 3 need not be produced by motion compensated temporal interpolation, the interpolator 102 is still employed to produce frames of motion vectors associated with the frames C 1 , C 3 and respective frames of figures of merit D 1 , D 3 .
  • Row I of FIG. 12 also represents the series of five input frames A 1 to A 5 or ten input frames a 1 to a 10 . Pairs of the fields are used by the field combiner 128 of FIG. 11 to produce a series of five field combined frames H 1 to H 5 . A series of four 30 Hz, 1:1 to 24 Hz, 1:1 converted frames G 1 to G 4 is then produced. The first converted frame G 1 may be directly produced from the first field combined frame H 1 , whereas the second to fourth converted frames G 2 , G 3 , G 4 are produced by motion compensated temporal interpolation one-quarter, one-half and three-quarters, respectively, of the way between field combined frames H 2 and H 3 , H 3 and H 4 , and H 4 and H 5 , respectively.
  • Figures of merit frames F 2 to F 4 are produced for each of the converted frames G 2 to G 4 , and a figure of merit frame is also produced for the first frame G 1 even though the first frame G 1 , need not be produced by motion compensated temporal interpolation.
  • the figure of merit in the frame D i and the figure of merit in the corresponding frame F i are compared by the figure of merit comparator 108, and in dependence on the result (as described above ) the value of the corresponding pixel in the corresponding frame C i or G i is selected as the value of the corresponding pixel in the corresponding output frame E i by the multiplexer 106.
  • a series of four output frames E 1 to E 4 is produced.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Television Systems (AREA)
US08/031,109 1992-03-18 1993-03-12 Method for processing a digital video signal having portions acquired with different acquisition characteristics Expired - Lifetime US5355169A (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB9205921 1992-03-18
GB9205921A GB2265275B (en) 1992-03-18 1992-03-18 Digital video signal processing

Publications (1)

Publication Number Publication Date
US5355169A true US5355169A (en) 1994-10-11

Family

ID=10712400

Family Applications (1)

Application Number Title Priority Date Filing Date
US08/031,109 Expired - Lifetime US5355169A (en) 1992-03-18 1993-03-12 Method for processing a digital video signal having portions acquired with different acquisition characteristics

Country Status (5)

Country Link
US (1) US5355169A (enrdf_load_stackoverflow)
EP (1) EP0561561B1 (enrdf_load_stackoverflow)
JP (1) JPH0646384A (enrdf_load_stackoverflow)
DE (1) DE69320223T2 (enrdf_load_stackoverflow)
GB (1) GB2265275B (enrdf_load_stackoverflow)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6542195B1 (en) * 1998-03-31 2003-04-01 Victor Company Of Japan, Limited Image pick-up apparatus for outputting image frames by thinning out the same
US20030219238A1 (en) * 2002-04-05 2003-11-27 Akira Yamaguchi Frame conversion apparatus and frame conversion method
US20050078215A1 (en) * 1997-10-10 2005-04-14 Swartz Peter D. Interlaced video field motion detection
US20050105797A1 (en) * 1998-10-29 2005-05-19 Pixar Color management system
US20050243216A1 (en) * 2004-04-30 2005-11-03 Sven Salzer Block mode adaptive motion compensation
US20060114334A1 (en) * 2004-09-21 2006-06-01 Yoshinori Watanabe Image pickup apparatus with function of rate conversion processing and control method therefor
US20070046785A1 (en) * 2005-08-31 2007-03-01 Kabushiki Kaisha Toshiba Imaging device and method for capturing image

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2281835B (en) * 1993-09-08 1998-04-22 Sony Uk Ltd Method and apparatus for synchronising video signals
EP0648047B1 (en) * 1993-10-11 2005-08-24 THOMSON multimedia Method and apparatus for forming a video signal using motion estimation and signal paths with different interpolation processing
DE4419891B4 (de) * 1994-06-07 2006-06-29 Micronas Gmbh Verfahren zur progressiven Konversion von Fernsehbildern mit Zeilensprung
DE102006001079B4 (de) * 2006-01-09 2020-08-06 Carl Zeiss Ag HMD-Vorrichtung und Anzeigeverfahren dafür
GB2552526A (en) 2016-07-28 2018-01-31 Siemens Ag Electrochemical method of ammonia generation

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3457369A (en) * 1965-02-04 1969-07-22 Marconi Co Ltd Television field-repetition frequency conversion using variable delay
US4862266A (en) * 1987-06-09 1989-08-29 Sony Corporation Television standards converters
GB2231228A (en) * 1989-04-27 1990-11-07 Sony Corp Video signal to photographic film conversion
JPH0385976A (ja) * 1989-08-30 1991-04-11 Sanyo Electric Co Ltd テレビジョン方式変換装置
GB2240232A (en) * 1990-01-17 1991-07-24 Avesco Plc Converting field rate of telecine signal
WO1992009172A2 (en) * 1990-11-16 1992-05-29 Rank Cintel Limited Video systems
EP0514012A2 (en) * 1991-04-15 1992-11-19 Vistek Electronics Limited Method and apparatus for the standard conversion of an image signal
US5187575A (en) * 1989-12-29 1993-02-16 Massachusetts Institute Of Technology Source adaptive television system

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3457369A (en) * 1965-02-04 1969-07-22 Marconi Co Ltd Television field-repetition frequency conversion using variable delay
US4862266A (en) * 1987-06-09 1989-08-29 Sony Corporation Television standards converters
GB2231228A (en) * 1989-04-27 1990-11-07 Sony Corp Video signal to photographic film conversion
JPH0385976A (ja) * 1989-08-30 1991-04-11 Sanyo Electric Co Ltd テレビジョン方式変換装置
US5187575A (en) * 1989-12-29 1993-02-16 Massachusetts Institute Of Technology Source adaptive television system
GB2240232A (en) * 1990-01-17 1991-07-24 Avesco Plc Converting field rate of telecine signal
WO1992009172A2 (en) * 1990-11-16 1992-05-29 Rank Cintel Limited Video systems
EP0514012A2 (en) * 1991-04-15 1992-11-19 Vistek Electronics Limited Method and apparatus for the standard conversion of an image signal

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8120710B2 (en) 1997-10-10 2012-02-21 Tamiras Per Pte. Ltd., Llc Interlaced video field motion detection
US20050078215A1 (en) * 1997-10-10 2005-04-14 Swartz Peter D. Interlaced video field motion detection
US20090161021A1 (en) * 1997-10-10 2009-06-25 Genesis Microchip Inc. Interlaced video field motion detection
US7522221B2 (en) * 1997-10-10 2009-04-21 Genesis Microchip Inc. Interlaced video field motion detection
US6542195B1 (en) * 1998-03-31 2003-04-01 Victor Company Of Japan, Limited Image pick-up apparatus for outputting image frames by thinning out the same
US7155054B2 (en) * 1998-10-29 2006-12-26 Pixar Color management system
US20050105797A1 (en) * 1998-10-29 2005-05-19 Pixar Color management system
US7421027B2 (en) * 2002-04-05 2008-09-02 Sony Corporation Frame conversion apparatus and frame conversion method
US20030219238A1 (en) * 2002-04-05 2003-11-27 Akira Yamaguchi Frame conversion apparatus and frame conversion method
US7440032B2 (en) * 2004-04-30 2008-10-21 Matsushita Electric Industrial Co., Ltd. Block mode adaptive motion compensation
US20050243216A1 (en) * 2004-04-30 2005-11-03 Sven Salzer Block mode adaptive motion compensation
US20060114334A1 (en) * 2004-09-21 2006-06-01 Yoshinori Watanabe Image pickup apparatus with function of rate conversion processing and control method therefor
US7860321B2 (en) * 2004-09-21 2010-12-28 Canon Kabushiki Kaisha Image pickup apparatus with function of rate conversion processing and control method therefor
US20070046785A1 (en) * 2005-08-31 2007-03-01 Kabushiki Kaisha Toshiba Imaging device and method for capturing image
US7593037B2 (en) * 2005-08-31 2009-09-22 Kabushiki Kaisha Toshiba Imaging device and method for capturing image

Also Published As

Publication number Publication date
GB2265275B (en) 1996-01-03
EP0561561A3 (enrdf_load_stackoverflow) 1994-01-26
GB9205921D0 (en) 1992-04-29
EP0561561B1 (en) 1998-08-12
DE69320223T2 (de) 1999-03-04
JPH0646384A (ja) 1994-02-18
GB2265275A (en) 1993-09-22
EP0561561A2 (en) 1993-09-22
DE69320223D1 (de) 1998-09-17

Similar Documents

Publication Publication Date Title
US5005077A (en) Motion dependent video signal processing
EP0395275B1 (en) Motion dependent video signal processing
US4862267A (en) Motion compensated interpolation of digital television images
US4992869A (en) Motion dependent video signal processing
EP0549681B1 (en) Video image processing
US5005078A (en) Motion dependent video signal processing
US5012337A (en) Motion dependent video signal processing
US5610662A (en) Method and apparatus for reducing conversion artifacts
US5027205A (en) Motion dependent video signal processing
JPS63313987A (ja) ディジタルテレビジョン画像の動きベクトル数の減少方法
JPS63313980A (ja) テレビジョン画像の動きベクトル評価方法
JPS63313984A (ja) テレビジョン画像の動きベクトル処理装置
EP0395276B1 (en) Video signal to photographic film conversion
US5355169A (en) Method for processing a digital video signal having portions acquired with different acquisition characteristics
US4998168A (en) Motion dependent video signal processing
US5012336A (en) Motion dependent video signal processing
EP1039746B1 (en) Line interpolation method and apparatus
JPS63313983A (ja) テレビジョン画像の動きベクトル処理装置
EP0395272B1 (en) Motion dependent video signal processing
KR0141702B1 (ko) 움직임 백터 처리방법 및 그 장치
GB2205713A (en) Motion compensated interpolation of digital television images
JPH0462234B2 (enrdf_load_stackoverflow)
JPH06326980A (ja) 動き補償映像信号処理方式
JP4470324B2 (ja) 画像信号変換装置および方法
JPH05252486A (ja) 映像信号の走査変換装置

Legal Events

Date Code Title Description
STPP Information on status: patent application and granting procedure in general

Free format text: APPLICATION UNDERGOING PREEXAM PROCESSING

AS Assignment

Owner name: SONY UNITED KINGDOM LIMITED, ENGLAND

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:RICHARDS, JOHN W.;DORRICOTT, MARTIN R.;DAVID, MORGAN W. A.;AND OTHERS;REEL/FRAME:006563/0092

Effective date: 19930322

AS Assignment

Owner name: SONY UNITED KINGDOM LIMITED

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SONY BROADCAST & COMMUNICATIONS LIMITED;REEL/FRAME:006604/0687

Effective date: 19930615

FPAY Fee payment

Year of fee payment: 4

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 8

REMI Maintenance fee reminder mailed
FPAY Fee payment

Year of fee payment: 12