Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
  • G11B27/102—Programmed access in sequence to addressed parts of tracks of operating record carriers
  • G11B27/105—Programmed access in sequence to addressed parts of tracks of operating record carriers of operating discs
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B2220/00—Record carriers by type
  • G11B2220/20—Disc-shaped record carriers
  • G11B2220/25—Disc-shaped record carriers characterised in that the disc is based on a specific recording technology
  • G11B2220/2537—Optical discs
  • G11B2220/2562—DVDs [digital versatile discs]; Digital video discs; MMCDs; HDCDs

Definitions

• the invention relates to a method and to an apparatus for controlling the insertion of additional fields or frames into a first format picture sequence having e.g. 24 progressive frames per second in order to construct therefrom a second format picture sequence having e.g. 25 frames per second.
• the major TV systems in the world use interlaced scanning and either 50Hz field frequency (e.g. in Europe and China for PAL and SECAM) or 60Hz or nearly 60Hz field frequency (e.g. in USA and Japan for NTSC) , denoted 50i and ⁇ Oi, respectively.
• 50Hz field frequency e.g. in Europe and China for PAL and SECAM
• 60Hz or nearly 60Hz field frequency e.g. in USA and Japan for NTSC
• Interlaced fields ILF are derived from original film frames ORGFF. From a first original film frame OFRl three output fields OFl to OF3 are generated, and from a third original film frame 0FR3 three output fields OF ⁇ to 0F8 are generated. From a second original film frame OFR2 two output fields OF4 and OF5 are generated, and from a fourth original film frame OFR4 two output fields OF9 and OF10 are generated, and so on.
• distribution media do have a single- format video and audio track which are playable worldwide rather than the current situation where at least a 50Hz and a 60Hz version exist of each packaged media title, e.g. DVD. Because many sources consist of 24 fps (frames per second) film, this 24p format is preferably the desired format for such single-format video tracks, which format therefore needs to be adapted at play-back time for displaying correctly on display devices, both, in the 50Hz and in the 60Hz countries .
• a problem to be solved by the invention is to provide a field or frame insertion scheme for conversion from 24p format to 25 fps format in an improved manner thereby minimising motion judder artefacts.
• This problem is solved by the method disclosed in claim 1.
• An apparatus that utilises this method is disclosed in claim 2.
• the characteristics of a current movie scene such as global motion, brightness/intensity level and scene change locations are evaluated in order to apply duplicated or repeated frames/fields at subjectively non-annoying locations.
• the invention uses relatively easily available information about the source material to be converted from 24p to 25 fps for adaptively inserting repeated fields/ frames at non-equidistant locations where the resulting insertion artefacts are minimum.
• the invention can be used for all frame rate conversion problems where there is a small difference between source frame rate and destination frame rate. If these frame rates differ a lot, such as in 24 fps to 30 fps conversion, there is hardly any freedom left for shifting in time fields or frames to be repeated.
• the invention facilitates computationally inexpensive conversion from 24 fps to 25 fps format picture sequences (example values) with minimised motion judder.
• the inventive method is suited for controlling the insertion of additional fields or frames into a first format picture sequence in order to construct therefrom a second format picture sequence the frame frequency of which is constant and is greater than that of the first format picture sequence, the method including the steps:
• the inventive apparatus is suited for controlling the insertion of additional fields or frames into a first format picture sequence in order to construct therefrom a second format picture sequence the frame frequency of which is constant and is greater than that of the first format picture sequence, said apparatus including means that are adapted for determining locations of fields or frames in said first format picture sequence at which locations the insertion of a corresponding additional field or frame causes a minimum visible motion judder in said second format picture sequence, and for inserting in said first format picture sequence a field or a frame at some of said locations at non-regular field or frame insertion distances such that in total the average distance between any adjacent frames corresponds to that of said second format picture sequence, and for presenting said first format picture sequence together with said non-regularly inserted fields and/or frames in the format of said second format picture sequence.
• Advantageous additional embodiments of the invention are disc osed in the respective dependent claims.
• FIG. 1 Simplified block diagram of an inventive disc player
• Fig. 2 Application of 3:2 pull-down on a 24p source picture sequence to provide a 60i picture sequence
• Fig. 5 Time line for regular frame repetition according to Fig. 3;
• Fig . 6 Example motion judder tolerance values of a video sequence
• Fig. 77 Example irregular temporal locations for field or frame repetition and the resulting varying presentation delay;
• Fig. 8 Frame or field repetition distance expressed as a function of video delay and motion judder tolerance;
• Fig. 9 The frame or field repetition distance function of Fig. 8 whereby the maximum and minimum video delays depend on the required degree of lip-sync;
• Fig. 10 24 fps format frames including a repeated frame without motion compensation;
• Fig. 11 25 fps format frame output related to Fig. 10;
• Fig. 12 24 fps format frames including a repeated frame with motion compensation;
• Fig. 13 25 fps format frame output related to Fig. 12.
• a disk drive including a pick-up and an error cor- rection stage PEC reads a 24p format encoded video and audio signal from a disc D.
• the output signal passes through a track buffer and de-multiplexer stage TBM to a video decoder VDEC and an audio decoder ADEC, respectively.
• a controller CTRL can control PEC, TBM, VDEC and ADEC.
• the interface IF may check automatically which mode or modes the TV receiver or a display can process and present.
• the replay mode information is derived automatically from feature data
• the replay mode information is input by the user interface UI upon displaying a corresponding request for a user.
• the controller CTRL, or the video decoder VDEC itself determines from characteristics of the decoded video signal at which temporal locations a field or a frame is to be repeated by the video decoder. In some, embodiments of the invention these temporal locations are also controlled by the audio signal or signals coming from audio decoder ADEC as explained below.
• the invention can also be used in other types of devices, e.g. a digital settop box or a digital TV receiver, in which case the front-end including the disk drive and the track buffer is replaced by a tuner for digital signals.
• Fig. 3 shows a regular pattern of repeated frames wherein one frame is repeated every 24 frames, i.e. at t n , t n +l, t n +2, t n +3, etc. seconds, for achieving a known 24p to 25 fps conversion.
• Fig. 4 shows a regular pattern of repeated fields wherein one field is repeated every 24 fields, i.e. at t n , t n +0.5, t n +l, t n +1.5, t n +2, etc. seconds, for achieving a known 24p to 25 fps conversion.
• This kind of processing is applicable if the display device has an interlaced output.
• the number of locations on the time axis where judder occurs are doubled, but the intensity of each 'judder instance' is halved as compared to the frame repeat.
• Top fields are derived from the first, third, fifth, etc. line of the indicated frame of the source sequence and bottom fields are derived from the second, fourth, sixth, etc. line of the indicated frame of the source sequence.
• Fig. 5 shows a time line for regular frame repetition according to Fig. 3, with markers at the temporal locations t n , t n +l, t n +2, t n +3, etc. seconds where frame repetition occurs .
• control information For carrying out the inventive adaptive insertion of repeated fields or frames at non-equidistant (or irregular) locations corresponding control information is required.
• Content information and picture signal characteristics about the source material become available as soon as the picture sequence is compressed by a scheme such as MPEG-2 Video, MPEG- 4 Video or MPEG-4 Video part 10, which supposedly will be used not only for current generation broadcast and packaged media such as DVD but also for future media such as disks based on blue laser technology.
• Such picture signal characteristics can be transferred from the encoder via a disk or via broadcast to the decoder as MPEG user data or private data. Alternatively, the video decoder can collect or calculate and provide such information.
• the set of motion vectors MV for each frame is collected and processed such that it can be determined whether a current frame has large visibly moving areas, since such areas suffer most from motion judder when duplicating frames or fields.
• the average absolute vector length AvgMVi can be calculated for a frame as an indication for a panning motion: v g MV l with * i' denoting the frame number, 'VX' and 'VY' being the number of motion vectors in x (horizontal) and y (vertical) direction of the image. Therefore, VX and VY are typically obtained by dividing the image size in the respective direction by the block size for motion estimation. If motion vectors within one frame point to different reference frames at different temporal distance to the current frame , a normalising factor RDistx,y for this distance is required in addition:
• a motion segmentation of each image is calculated, i.e. one or more clusters of adjacent blocks having motion vectors with similar length and direction are determined, in order to detect multiple large-enough moving areas with different motion directions.
• the average motion vector can be calculated for example by: nClusters AvgMV c ⁇ ClusterSize c " ' nClusters ClusterSize c -i , ( 2a) wherein AvgMVc is the average motion vector length for the identified cluster 'c'.
• this approach eliminates the effect of motion vectors for randomly moving small objects within an image that are not member of any identified block cluster motion and that do not contribute significantly to motion judder visibility.
• the processing may take into account as weighting factors for vgMVc whether the moving areas are strongly textured or have sharp edges, as this also increases visibility of motion judder.
• Information about texture strength can be derived most conveniently from a statistical analysis of transmitted or received or replayed AC transform coefficients for the prediction error. In principle, texture strength should be determined from analysing an original image block, however, in many cases such strongly textured blocks after encoding using motion compensated prediction will also have more prediction error energy in their AC coefficients than less textured blocks.
• MJT is proportional to 1/AvgMV
• MJT is proportional to 1/ (texture strength
• MJT is proportional to l/(edge strength).
• Fig. 6 shows example motion judder tolerance values MJT(t) over a source sequence.
• the current size of the motion judder tolerance value influences the distribution, as depicted in Fig. 7a, of inserted repeated frames or fields into the resulting 25 fps sequence, i.e. the frame or field repetition distance FRD.
• Fig. 7a the distribution, as depicted in Fig. 7a, of inserted repeated frames or fields into the resulting 25 fps sequence, i.e. the frame or field repetition distance FRD.
• Early or delayed insertion of repeated frames causes a negative or positive delay of the audio track relative to the video track as indicated in Fig. 7b, i.e. a varying presentation delay for video.
• a maximum tolerable video delay relative to audio in both directions is considered when applying the mapping from motion judder tolerance MJT to frame or field repetition distance FRD.
• FRD f (VD, MJT) , (4) with the following general characteristics:
• FRD is proportional to 1/MJT
• the optimum field or frame repetition distance is shown as FRD 0 p-
• the maximum allowable video delay is shown as VDmax.
• scene change information generated by a video encoder can be used to insert one or more repeated fields or frames at such locations, the number of repetitions depending on the current degree of video delay.
• repeated fields or frames can be inserted after a fade-to- black sequence, a fade-to-white sequence or a fade to any colour. All such singular locations have a very high MJT value .
• repeated frames could be used at such locations even if at other picture content fields only would be repeated in order to reduce motion judder intensity at individual locations.
• repeated frames and repeated fields may co-exist in a converted picture sequence.
• VD m i nLipSync and VD maxLipSync if speech or short sound peaks (which are caused by special events like a clapping door) are detected and a slowly moving or static scene is detected;
• a detection of speech can be derived for example in case of the mostly-used multi-channel audio by evaluating the centre channel relative to left and right channels, as speech in movies is mostly coded into the centre channel. If the centre channel shows a bursty energy distribution over time that is significantly different from the energy distribution in the left and right channels, then the likelihood of speech being present is high.
• MJT(i+k)) MJT(i+k)) , (5) wherein ' i' denotes the current frame number and 'k' denotes a running number referencing the adjacent frames.
• a general characteristic of each such function is that FRD increases if MJT (i) is smaller than the surrounding MJT values and decreases if MJT(i) is larger than the surrounding MJT values.
• Related picture signal characteristics can be transferred as MPEG user data or private data from the encoder via a disk or via broadcast signal to the decoder.
• motion compensated interpolation of frames rather than repetition of frames can be applied without computational expense.
• Such motion compensated interpolation can make use of the transmitted motion vectors for the current frame.
• these motion vectors are not suitable for motion compensated frame interpolation since they are optimised for optimum prediction gain rather than indicating the true motion of a scene.
• a highly accurate frame can be interpolated between the current and the previous frame. Panning means that all motion vectors within a frame are identical or nearly identical in length and orientation. Hence an interpolated frame can be generated by translating the previous frame by half the distance indicated by the average motion vector for the current frame.
• the previous frame is the reference frame for the motion compensated prediction of the current frame and that the interpolated frame is equidistantly positioned between the previous and current frame. If the prediction frame is not the previous frame, adequate scaling of the average motion vector is to be applied.
• a zoom is characterised by zero motion vectors in the zoom centre and increasing length of centre- (opposite) -directed motion vectors around this zoom centre, the motion vector length increasing in relation to the distance from the zoom centre .
• this kind of motion compensated interpolation yields an improved motion judder behaviour compared to repeating a frame, as is illustrated in Fig. 10 to 13.
• Fig. 10 in 24 fps format and Fig. 11 after 25 fps format conversion show frames (indicated as vertical bars) with a motion trajectory for a vertically moving object and one instance of frame repetition, which results in a 'freeze frame'.
• Fig. 12 shows insertion of a motion interpolated frame which, when presented at the increased 25 fps target frame rate as depicted in Fig. 13, leads to a 'slowly moving frame' rather than a 'freeze frame'.
• the above-disclosed controls for frame and/or field repetition and interpolation for frame rate conversion can be applied, both, at the encoder and at the decoder side of an MPEG-2 (or similar) compression system since most side information is available at both sides, possibly except reliable scene change indication.
• the locations for fields or frames to be repeated or interpolated can be conveyed in the (MPEG-2 or otherwise) compressed 24 fps video signal.
• Flags to indicate temporal order of fields (top_field_first) and repetition of the first field for display (repeat_first_field) exist already in the MPEG-2 syntax. If it is required to signal the conversion pattern both for 24 fps to 30 fps and 24 fps to 25 fps conversion for the same video signal, one of the two series of flags may be conveyed in a suitable user data field for each picture.
• the invention can be applied in an optical disc player or in an optical disc recorder, or in a harddisk recorder, e.g. an HDD recorder or a PC, or in a settop box, or in a TV receiver.
• a harddisk recorder e.g. an HDD recorder or a PC

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Television Signal Processing For Recording (AREA)
• Television Systems (AREA)
• Signal Processing For Digital Recording And Reproducing (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (1)

Family

ID=34626528

Family Applications (1)

Country Status (7)

Cited By (2)

Families Citing this family (10)

Citations (2)

Family Cites Families (9)

• 2004
  • 2004-01-21 EP EP04090021A patent/EP1558033A1/en not_active Withdrawn
  • 2004-11-04 JP JP2006549890A patent/JP4766567B2/ja not_active Expired - Fee Related
  • 2004-11-04 DE DE602004018984T patent/DE602004018984D1/de not_active Expired - Lifetime
  • 2004-11-04 US US10/586,116 patent/US20080232784A1/en not_active Abandoned
  • 2004-11-04 EP EP04797609A patent/EP1707002B1/en not_active Expired - Lifetime
  • 2004-11-04 CN CN200480040820A patent/CN100581235C/zh not_active Expired - Fee Related
  • 2004-11-04 BR BRPI0418350-9A patent/BRPI0418350B1/pt not_active IP Right Cessation
  • 2004-11-04 WO PCT/EP2004/012483 patent/WO2005071956A1/en not_active Ceased

Patent Citations (2)

Also Published As

Similar Documents

Legal Events